Question

我尝试在Amazon EMR群集（AMI版本3.2.0 / Hadoop版本2.4.0）上安装sqoop 2（版本1.99.3）。当我启动sqoop服务器时，我在localhost.log中看到了这个错误：

Sep 10, 2014 4:55:56 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.apache.sqoop.server.ServerInitializer
java.lang.RuntimeException: Failure in server initialization
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
    at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:36)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
    at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
    at org.apache.catalina.core.StandardService.start(StandardService.java:525)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.sqoop.common.SqoopException: MAPREDUCE_0002:Failure on submission engine initialization
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:115)
    at org.apache.sqoop.framework.JobManager.initialize(JobManager.java:215)
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:53)
    ... 25 more
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:113)
    ... 27 more

根据installation instructions，我在这里做了什么。请注意，使用EMR，$ HADOOP_HOME是/ home / hadoop。

我从Apache下载了sqoop-1.99.3-bin-hadoop200.tar.gz并将其解压缩到$ HADOOP_HOME / sqoop。
将以下内容添加到catalina.properties中的common.loader属性中： /home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/的hadoop /共享/ hadoop的/纱线/ *。罐
在sqoop.properties中：
- 我用/ home / hadoop / sqoop / log
- 我将/ BASEDIR @替换为/ home / hadoop / sqoop
- 对于属性org.apache.sqoop.submission.engine.mapreduce.configuration.directory，我将/ etc / hadoop / conf /替换为/ home / hadoop / conf /
然后我启动了服务器：bin / sqoop.sh server start

就我从错误中可以看出，问题的根源是这一行：

Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

并且此属性在mapred-site.xml中设置：

<property><name>mapreduce.framework.name</name><value>yarn</value></property>

这就是我想要的。

我觉得我在sqoop的安装/配置中缺少一些参数设置，并且我尝试过其他一些事情（例如尝试AMI 3.0.4 / Hadoop 2.2.0）但是无法启动sqoop服务器。

FYI。我已经在Kyle Mulka's blog上阅读了这篇文章，但它引用了不同版本的Hadoop和sqoop，并没有提供对我的配置的深入了解。我已经阅读了本网站上的其他一些页面，但还没有找到一个引用我使用的Hadoop和sqoop版本的页面。我已经看到这种配置与Cloudera一起运行（sqoop2和带有纱线的Hadoop 2），虽然还没有能够弄清楚这将如何转化为EMR安装。

Answer 1

感谢对this post的反馈，我改变了catalina.properties中的common.loader属性，而不是之前的内容：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/yarn/*.jar

到此：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/hdfs/*.jar,/home/hadoop/share/hadoop/hdfs/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/mapreduce/lib/*.jar,/home/hadoop/share/hadoop/tools/lib/*.jar,/home/hadoop/share/hadoop/yarn/*.jar,/home/hadoop/share/hadoop/yarn/lib/*.jar

之后，sqoop服务器成功启动。

在带有yarn的Amazon EMR上运行sqoop2服务器时出错

1 个答案: