将数据从Mysql导入hdfs时出错

时间：2014-03-06 13:30:12

标签： hadoop2 sqoop2

我有hadoop 2.2.0群集，其上安装了3个节点和sqoop 2。当我启动sqoop服务器时，我遇到以下异常：

2014-03-06 18:24:57,105 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-03-06 18:24:57,105 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-03-06 18:24:57,106 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-03-06 18:24:57,115 INFO  mapreduce.MapreduceSubmissionEngine [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:74)] Initializing Map-reduce Submission Engine
2014-03-06 18:24:57,275 INFO  mapreduce.MapreduceSubmissionEngine [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:94)] Found hadoop configuration file core-site.xml
2014-03-06 18:24:57,275 INFO  mapreduce.MapreduceSubmissionEngine [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:94)] Found hadoop configuration file hdfs-site.xml
2014-03-06 18:24:57,275 INFO  mapreduce.MapreduceSubmissionEngine [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:94)] Found hadoop configuration file mapred-site.xml
2014-03-06 18:24:57,276 INFO  mapreduce.MapreduceSubmissionEngine [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:94)] Found hadoop configuration file yarn-site.xml
2014-03-06 18:24:57,823 ERROR server.ServerInitializer [org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:56)] Server startup failure
org.apache.sqoop.common.SqoopException: MAPREDUCE_0002:Failure on submission engine initialization
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:106)
    at org.apache.sqoop.framework.FrameworkManager.initialize(FrameworkManager.java:200)
    at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:53)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
    at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
    at org.apache.catalina.core.StandardService.start(StandardService.java:525)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:104)
    ... 26 more
2014-03-06 18:24:57,827 INFO  server.ServerInitializer [org.apache.sqoop.server.ServerInitializer.contextDestroyed(ServerInitializer.java:39)] Shutting down Sqoop server

原因是什么？

1 个答案:

答案 0 :(得分：0)

这看起来非常类似于我在Hadoop 2.4.0上尝试启动sqoop 2服务器时遇到的问题。问题是我没有正确设置catalina.properties中的common.loader属性。我最初设置的属性如下：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/yarn/*.jar

但后来改为：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/hdfs/*.jar,/home/hadoop/share/hadoop/hdfs/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/mapreduce/lib/*.jar,/home/hadoop/share/hadoop/tools/lib/*.jar,/home/hadoop/share/hadoop/yarn/*.jar,/home/hadoop/share/hadoop/yarn/lib/*.jar

为我解决了这个问题。 （请注意，我正在使用Amazon EMR;在此环境中，$ HADOOP_HOME设置为/ home / hadoop。）

我遇到的问题的原始描述是here。

相关问题
最新问题