在命令shell中运行以下sqoop导入时,效果很好。
import --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username retail_dba --password cloudera -m 1 --table categories --hive-database retail_stage --hive-table categories --fields-terminated-by "|" --hive-import
但是同一条语句在Hue工作流中运行时会失败,并显示以下错误
>>> Invoking Sqoop command line now >>>
2019-02-04 11:46:18,411 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2019-02-04 11:46:18,609 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.13.0
2019-02-04 11:46:18,664 [main] WARN org.apache.sqoop.tool.BaseSqoopTool - Setting your password on the command-line is insecure. Consider using -P instead.
2019-02-04 11:46:18,696 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2019-02-04 11:46:18,936 [main] INFO org.apache.sqoop.manager.MySQLManager - Preparing to use a MySQL streaming resultset.
2019-02-04 11:46:18,951 [main] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code generation
2019-02-04 11:46:20,510 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
2019-02-04 11:46:20,555 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
2019-02-04 11:46:20,565 [main] INFO org.apache.sqoop.orm.CompilationManager - HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
2019-02-04 11:46:25,907 [main] ERROR org.apache.sqoop.tool.ImportTool - Import failed: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:222)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:187)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:170)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
<<< Invocation of Sqoop command completed <<<
No child hadoop job is executed.
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
请记住,当在Hue工作流程中运行list-databases
命令时,效果很好。
版本:Cloudera Express 5.13.0 (#55 built by jenkins on 20171002-1719 git: bd657e597e6743c458ee2c9aabe808b7c972981c)
Java VM名称:Java HotSpot(TM) 64-Bit Server VM
Java VM供应商:Oracle Corporation
Java版本:1.7.0_67
实际上,在oozie中作为sqoop操作放置的任何命令都会失败。
以下是在docker映像中如何启动cloudera quickstart vm
docker run --hostname=quickstart.cloudera --privileged=true -t -i -v /Users/Yunus/Documents/ClouderaShare:/src --publish-all=true -p 8888:8888 -p 8020:8020 -p 8032:8032 -p 7180:7180 -p 80:80 -p 50070:50070 -p 11000:11000 -p 21050:21050 -p 8088:8088 -p 8042:8042 cloudera-5-13 /usr/bin/docker-quickstart
home/cloudera/cloudera-manager --express
/etc/init.d/ntpd start
sudo -u hdfs hadoop fs -put usr/share/java/mysql-connector-java-5.1.34-bin.jar /user/oozie/share/lib/lib_20171023234839/sqoop