Amazon EMR + mrjob:bootstrap错误,“bootstrap action 1返回非零返回码”

时间:2015-07-01 16:48:39

标签: amazon-ec2 emr bootstrapping mrjob

我正在尝试使用ec2实例在Amazon的EMR上运行mrjob。它一直在工作,直到我意识到我正在使用python包(mechanize,BeautifulSoup,boto)。所以,我添加到我的mrjob.conf文件中,但现在我不断收到此错误:

No handlers could be found for logger "mrjob.emr"
Traceback (most recent call last):
  File "run_job.py", line 102, in <module>
    run()
  File "run_job.py", line 76, in run
    runner.run()
  File "C:\Program Files (x86)\Python278\lib\site-packages\mrjob\runner.py", line 464, in run
    self._run()
  File "C:\Program Files (x86)\Python278\lib\site-packages\mrjob\emr.py", line 821, in _run
    self._wait_for_job_to_complete()
  File "C:\Program Files (x86)\Python278\lib\site-packages\mrjob\emr.py", line 1689, in _wait_for_jo
b_to_complete
    raise Exception(msg)
Exception: Job on job flow j-CZDG75Z1X58 failed with status FAILED: On the master instance (i-0ef8bb
cc), bootstrap action 1 returned a non-zero return code

我在这里结束了我的智慧,我一直试图解决这个问题已经有一段时间了,我似乎无法得到它。这是conf文件:

runners:
  emr:
    bootstrap_mrjob: True
    ami_version: 2.4.11

    bootstrap:
        - sudo apt-get install -y python-pip
        - sudo pip install mechanize
        - sudo pip install bs4
        - sudo pip install mrjob
        - sudo pip install boto

    aws_access_key_id: xxxx
    aws_region: xxxx
    aws_secret_access_key: xxxx
    num_ec2_core_instances: 1
    ec2_core_instance_type: m1.small
    iam_instance_profile: xxxx

1 个答案:

答案 0 :(得分:0)

终于搞清楚了!

在mrjob.conf中,我将bootstrap:及其所有元素更改为

\documentclass{article}

\begin{document}

<<table1, echo=FALSE, message=F, warning=F, results="asis">>=
library(stargazer)
lm1 <- lm(mpg ~ wt, data = mtcars)
lm2 <- lm(mpg ~ hp, data = mtcars)
lm3 <- lm(mpg ~ cyl, data = mtcars)
lm4 <- lm(mpg ~ disp, data = mtcars)

stargazer(lm1, lm2, lm3, lm4, title = "Basic OLS Models", omit.stat = c("f",    "ser"))
@
\end{document}

事实证明,mjrob几乎总是自动将自己的包安装到ec2实例。所以我根本不需要在引导程序中使用它。

现在它像魅力一样运行! :)