色相工作流程中的Sqoop失败

时间:2019-02-04 12:27:20

标签: hive sqoop hue cloudera-quickstart-vm oozie-workflow

在命令shell中运行以下sqoop导入时,效果很好。

import --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username retail_dba --password cloudera -m 1 --table categories --hive-database retail_stage --hive-table categories --fields-terminated-by "|" --hive-import

但是同一条语句在Hue工作流中运行时会失败,并显示以下错误

>>> Invoking Sqoop command line now >>>

2019-02-04 11:46:18,411 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2019-02-04 11:46:18,609 [main] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.6-cdh5.13.0
2019-02-04 11:46:18,664 [main] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
2019-02-04 11:46:18,696 [main] WARN  org.apache.sqoop.ConnFactory  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2019-02-04 11:46:18,936 [main] INFO  org.apache.sqoop.manager.MySQLManager  - Preparing to use a MySQL streaming resultset.
2019-02-04 11:46:18,951 [main] INFO  org.apache.sqoop.tool.CodeGenTool  - Beginning code generation
2019-02-04 11:46:20,510 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
2019-02-04 11:46:20,555 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
2019-02-04 11:46:20,565 [main] INFO  org.apache.sqoop.orm.CompilationManager  - HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
2019-02-04 11:46:25,907 [main] ERROR org.apache.sqoop.tool.ImportTool  - Import failed: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:222)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:187)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:170)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)


<<< Invocation of Sqoop command completed <<<

No child hadoop job is executed.
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

请记住,当在Hue工作流程中运行list-databases命令时,效果很好。

Cloudera Quickstart VM(泊坞窗映像)详细信息

版本:Cloudera Express 5.13.0 (#55 built by jenkins on 20171002-1719 git: bd657e597e6743c458ee2c9aabe808b7c972981c)

Java VM名称:Java HotSpot(TM) 64-Bit Server VM

Java VM供应商:Oracle Corporation

Java版本:1.7.0_67

实际上,在oozie中作为sqoop操作放置的任何命令都会失败。

以下是在docker映像中如何启动cloudera quickstart vm

  1. 启动cloudera快速入门容器: docker run --hostname=quickstart.cloudera --privileged=true -t -i -v /Users/Yunus/Documents/ClouderaShare:/src --publish-all=true -p 8888:8888 -p 8020:8020 -p 8032:8032 -p 7180:7180 -p 80:80 -p 50070:50070 -p 11000:11000 -p 21050:21050 -p 8088:8088 -p 8042:8042 cloudera-5-13 /usr/bin/docker-quickstart
  2. 启动cloudera管理器:home/cloudera/cloudera-manager --express
  3. 修复时钟偏移问题:/etc/init.d/ntpd start
  4. Mysql连接器已位于usr / share / java /目录中:sudo -u hdfs hadoop fs -put usr/share/java/mysql-connector-java-5.1.34-bin.jar /user/oozie/share/lib/lib_20171023234839/sqoop

0 个答案:

没有答案