Question

在我的本地方框中我安装了mysql并在本地安装了sqoop以连接配置单元来提取数据

1）sqoop list-databases --connect jdbc:mysql://localhost/db --username db1

返回

16/05/13 21:49:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/05/13 21:49:50 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

INFORMATION_SCHEMA 搜索测试

2）我想从hive表获取数据到mysql表，所以我试过

sqoop export --connect jdbc:mysql://localhost/db --username db1 --table search \
--driver com.mysql.jdbc.Driver \
--export-dir /projects/Tool_DB/search/dt=20160403/region=0 \
--input-fields-terminated-by '\001'

显示错误

16/05/13 21:48:24 INFO mapreduce.Job: Running job: job_1459561405699_1417866
16/05/13 21:48:43 INFO mapreduce.Job: Job job_1459561405699_1417866 running in uber mode : false
16/05/13 21:48:43 INFO mapreduce.Job:  map 0% reduce 0%
16/05/13 21:48:52 INFO mapreduce.Job: Task Id : attempt_1459561405699_1417866_m_000001_0, Status : FAILED
Error: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:647)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:350)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2408)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2445)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2230)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

3）我也尝试了

sqoop export --connect jdbc:mysql://localhost/db --username db1 --table search \
--driver com.mysql.jdbc.Driver \
--hcatalog-database db 
--hcatalog-table search

并显示错误

16/05/13 21:53:08 INFO mapreduce.ExportJobBase: Configuring HCatalog for export job
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatOutputFormat
    at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:420)
    at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:912)
    at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
    at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatOutputFormat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

有任何解决此错误的建议或提示吗？

Answer 1

您遇到此问题，因为未设置HCAT_HOME路径。

请设置路径并尝试。

export HCAT_HOME= < hive-hcatalog path in local>

例如：

export HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog

线程＆＃34; main＆＃34;中的例外情况java.lang.NoClassDefFoundError：org / apache / hive / hcatalog / mapreduce / HCatOutputFormat

1 个答案: