通过Spark运行时出现Sqoop错误

时间：2018-03-09 05:23:46

标签： apache-spark cloudera sqoop

当我通过sqoop命令运行此代码时，它可以正常工作

sqoop import --connect "jdbc:sqlserver://myhost:port;databaseName=DBNAME" \
 --username MYUSER -P \
 --compress --compression-codec snappy \
 --as-parquetfile \
 --table MYTABLE \
 --warehouse-dir /user/myuser/test1/ \
 --m 1

然后我创建如下的spark scala代码。但是当我使用spark-submit执行项目时，它无法正常工作

val sqoop_options: SqoopOptions = new SqoopOptions()
sqoop_options.setConnectString("jdbc:sqlserver://myhost:port;databaseName=DBNAME")
sqoop_options.setTableName("MYTABLE");
sqoop_options.setUsername("MYUSER");
sqoop_options.setPassword("password");
sqoop_options.setNumMappers(1);
sqoop_options.setTargetDir("/user/myuser/test1/");
sqoop_options.setFileLayout(FileLayout.ParquetFile);
sqoop_options.setCompressionCodec("org.apache.hadoop.io.compress.SnappyCodec")
val importTool = new ImportTool
val sqoop = new Sqoop(importTool, conf, sqoop_options);
val retCode = ToolRunner.run(sqoop, null);

它返回驱动程序未找到错误，即使我在同一个集群上运行它。 我已经在/ var / lib / sqoop目录上放了适当的库，这就是sqoop命令运行良好的原因。但是，当我通过spark-submit运行它时，是否会引用另一个库路径？

详细错误日志：

 /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/spark/conf/spark-env.sh: line 75: spark.driver.extraClassPath=.:/etc/hbase/conf:/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar://opt/cloudera/parcels/CDH/lib/hbase/hbase-server.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/guava-12.0.1.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/zookeeper.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-spark.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/hive-hbase-handler.jar: No such file or directory
/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/spark/conf/spark-env.sh: line 77: spark.executor.extraClassPath=.:/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar://opt/cloudera/parcels/CDH/lib/hbase/hbase-server.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/guava-12.0.1.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/zookeeper.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-spark.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/hive-hbase-handler.jar: No such file or directory
2018-03-09 13:59:37,332 INFO  [main] security.UserGroupInformation: Login successful for user myuser using keytab file myuser.keytab
2018-03-09 13:59:37,371 INFO  [main] sqoop.Sqoop: Running Sqoop version: 1.4.6
2018-03-09 13:59:37,426 WARN  [main] sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2018-03-09 13:59:37,478 INFO  [main] manager.SqlManager: Using default fetchSize of 1000
2018-03-09 13:59:37,479 INFO  [main] tool.CodeGenTool: Beginning code generation
2018-03-09 13:59:37,479 INFO  [main] tool.CodeGenTool: Will generate java class as codegen_MYTABLE
Exception in thread "main" java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:856)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
        at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
        at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
        at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
        at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
        at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
        at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at com.test.spark.sqoop.SqoopExample$.importSQLToHDFS(SqoopExample.scala:56)
        at com.test.spark.sqoop.SqoopExample$.main(SqoopExample.scala:18)
        at com.test.spark.sqoop.SqoopExample.main(SqoopExample.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

现在我的错误是：

spark-submit --files kafka-jaas.conf,ampuser.keytab --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka-jaas.conf" --driver-java-options "-Djava.security.auth.login.config=kafka-jaas.conf" --conf spark.driver.extraClassPath=/var/lib/sqoop/sqljdbc4.jar:/opt/cloudera/parcels/CDH/lib/sqoop/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/* --class com.danamon.spark.sqoop.SqoopExample --deploy-mode client --master yarn kafka-streaming-0.0.1-SNAPSHOT-jar-with-dependencies.jar


18/03/13 20:54:51 INFO security.UserGroupInformation: Login successful for user ampuser using keytab file ampuser.keytab
18/03/13 20:54:51 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/03/13 20:54:51 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
18/03/13 20:54:51 INFO manager.SqlManager: Using default fetchSize of 1000
18/03/13 20:54:51 INFO tool.CodeGenTool: Beginning code generation
18/03/13 20:54:51 INFO tool.CodeGenTool: Will generate java class as codegen_BD_AC_ACCT_PREFERENCES
18/03/13 20:54:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [BD_AC_ACCT_PREFERENCES] AS t WHERE 1=0
18/03/13 20:54:52 INFO orm.CompilationManager: $HADOOP_MAPRED_HOME is not set
Note: /tmp/sqoop-ampuser/compile/95e3ef854d67b50d8ef72955151dc846/codegen_BD_AC_ACCT_PREFERENCES.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/03/13 20:54:54 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ampuser/compile/95e3ef854d67b50d8ef72955151dc846/codegen_BD_AC_ACCT_PREFERENCES.jar
18/03/13 20:54:54 INFO mapreduce.ImportJobBase: Beginning import of BD_AC_ACCT_PREFERENCES
18/03/13 20:54:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
        at org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:190)
        at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:94)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
        at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
        at org.apache.sqoop.manager.SQLServerManager.importTable(SQLServerManager.java:163)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at com.danamon.spark.sqoop.SqoopExample$.importSQLToHDFS(SqoopExample.scala:57)
        at com.danamon.spark.sqoop.SqoopExample$.main(SqoopExample.scala:18)
        at com.danamon.spark.sqoop.SqoopExample.main(SqoopExample.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.kitesdk.data.mapreduce.DatasetKeyOutputFormat
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 22 more

是否由我的Cloudera安装导致配置不正确？或者HADOOP_HOME，MAPRED_HOME等未正确设置？ 我应该为此创建新问题吗？

1 个答案:

答案 0 :(得分：0)

您需要将HADOOP_MAPRED_HOME设置为〜/ .bashrc中的$HADOOP_HOME 

sudo nano ~/.bashrc

然后添加此行

export HADOOP_MAPRED_HOME = $HADOOP_HOME

保存文件，然后运行此命令

source ~/.bashrc
相关问题
最新问题