我已经使用sqoop将mySql数据库表导入了hdfs。该数据库可能每天都会更新,因此我可能需要在hdfs中反映这些更新。为此,我尝试将sqoop用作:
sqoop import --connect jdbc:mysql://127.0.0.1/crawl_data_stats --username root --password password --table scraped_metadata -m 1 --as-parquetfile --target-dir /databases/crawl_data_stats/scraped_metadata --check-column timestamp --incremental lastmodified --last-value '2019-05-16 13:06:31' --driver com.mysql.jdbc.Driver --merge-key id
这会产生错误:
ERROR tool.ImportTool: Import failed: java.io.IOException: Could not load jar /tmp/sqoop-root/compile/4bdd1f516ac0e2901cd52ff27ebd49c1/codegen_scraped_metadata.jar into JVM. (Could not find class scraped_metadata.)
at org.apache.sqoop.util.ClassLoaderStack.addJarFile(ClassLoaderStack.java:92)
at com.cloudera.sqoop.util.ClassLoaderStack.addJarFile(ClassLoaderStack.java:36)
at org.apache.sqoop.tool.ImportTool.loadJars(ImportTool.java:121)
at org.apache.sqoop.tool.ImportTool.lastModifiedMerge(ImportTool.java:463)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:529)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: scraped_metadata
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.sqoop.util.ClassLoaderStack.addJarFile(ClassLoaderStack.java:88)
... 11 more
I have just started using sqoop and don't know how to resolve this issue.
I came across the similar issue at
https://stackoverflow.com/questions/56088078/error-tool-importtool-import-failed-java-io-ioexception-could-not-load-jar-t.
However, there are no suggestions
Thanks for help