Question

我正在使用MRJob在亚马逊的EMR上运行迭代的hadoop程序。

当我没有使用“--pool-emr-job-flows”选项时，一切正常（但很慢）。当我使用此选项时，

Traceback (most recent call last):
  File "ic_bfs_eval.py", line 297, in <module>
    res = main()
  File "ic_bfs_eval.py", line 262, in main
    frac, mr_rounds = bfs(db_name, T, samples, total_steps_cap)
  File "ic_bfs_eval.py", line 183, in bfs
    runner.run()
  File "/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg/mrjob/runner.py", line 620, in __exit__
    self.cleanup()
  File "/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg/mrjob/emr.py", line 987, in cleanup
    super(EMRJobRunner, self).cleanup(mode=mode)
  File "/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg/mrjob/runner.py", line 566, in cleanup
    self._cleanup_job()
  File "/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg/mrjob/emr.py", line 1061, in _cleanup_job
    self._opts['ec2_key_pair_file'])
  File "/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg/mrjob/ssh.py", line 209, in ssh_terminate_single_job
    num_jobs_match = HADOOP_JOB_LIST_NUM_RE.match(job_list_lines[0])
IndexError: list index out of range

我正在初始化一个MRJob：

mrJob2 = MRBFSSampleIter(args=["-c", "~/mrjob.conf",
                                       "-r", "emr",
                                       "--no-output",
                                       "--output-dir", tmp_dir_out,
                                       "--pool-emr-job-flows", tmp_dir_in])

关于为什么会发生这种情况的任何想法？

Answer 1

当我设置ssh密钥对时，这对我来说就消失了。我认为它仍然是一个错误，因为ssh应该是可选的。但最简单的解决方法是设置密钥对，如http://mrjob.readthedocs.org/en/latest/guides/emr-quickstart.html#configuring-ssh-credentials

所述

当使用--pool-emr-job-flows时，MRJob无法在EMR上为新作业加注星标

1 个答案: