CentOS上的sqoop-export

时间:2013-01-29 15:20:14

标签: mysql sqoop

我想使用来自Hive =>的sqoop-export; MySQL的 我执行了以下命令:

sqoop-export --connect jdbc:mysql://localhost/mydb --table tbl_1 --export-dir /user/hive/warehouse/tbl_1 --username test --password test123

当我在本地计算机上执行命令时,它可以工作。但是当我在服务器上尝试它时却没有。

本地机器:Ubuntu 12.10 with MySQL 5.5.28-0ubuntu0.12.10.2

服务器:带有MySQL 5.0.95-log的CentOS 5.6

Hadoop / Hive / Sqoop版本相同。

我将hadoop / hive / sqoop从我的电脑复制到服务器,并在服务器上运行良好。

正确运行创建表。

但为什么sqoop错误?

日志在下面。

 sqoop-export --connect jdbc:mysql://localhost/honeybee --table tbl_1 --export-dir /user/hive/warehouse/tbl_1 --username test --password test123
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
13/01/30 07:57:05 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/01/30 07:57:05 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/01/30 07:57:05 INFO tool.CodeGenTool: Beginning code generation
13/01/30 07:57:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_1` AS t LIMIT 1
13/01/30 07:57:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_1` AS t LIMIT 1
13/01/30 07:57:05 INFO orm.CompilationManager: HADOOP_HOME is /home/hadoop-1.1.1/libexec/..
Note: /tmp/sqoop-root/compile/26876fd02da2baebf0e2db4e7ddb6ddd/tbl_1.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/01/30 07:57:07 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/26876fd02da2baebf0e2db4e7ddb6ddd/tbl_1.jar
13/01/30 07:57:07 INFO mapreduce.ExportJobBase: Beginning export of tbl_1
13/01/30 07:57:08 INFO input.FileInputFormat: Total input paths to process : 1
13/01/30 07:57:08 INFO input.FileInputFormat: Total input paths to process : 1
13/01/30 07:57:08 INFO mapred.JobClient: Running job: job_201301300430_0033
13/01/30 07:57:09 INFO mapred.JobClient:  map 0% reduce 0%
13/01/30 07:57:16 INFO mapred.JobClient: Task Id : attempt_201301300430_0033_m_000000_0, Status : FAILED
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:350)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2408)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2445)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2230)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
    at java.sql.DriverManager.getConnection(DriverManager.java:579)
    at java.sql.DriverManager.getConnection(DriverManager.java:221)
    at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:184)
    at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
    at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
    ... 8 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at java.net.Socket.connect(Socket.java:528)
    at java.net.Socket.<init>(Socket.java:425)
    at java.net.Socket.<init>(Socket.java:241)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
    ... 26 more

13/01/30 07:57:21 INFO mapred.JobClient: Task Id : attempt_201301300430_0033_m_000000_1, Status : FAILED
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:350)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2408)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2445)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2230)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
    at java.sql.DriverManager.getConnection(DriverManager.java:579)
    at java.sql.DriverManager.getConnection(DriverManager.java:221)
    at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:184)
    at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
    at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
    ... 8 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at java.net.Socket.connect(Socket.java:528)
    at java.net.Socket.<init>(Socket.java:425)
    at java.net.Socket.<init>(Socket.java:241)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
    ... 26 more

13/01/30 07:57:27 INFO mapred.JobClient: Task Id : attempt_201301300430_0033_m_000000_2, Status : FAILED
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:350)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2408)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2445)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2230)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
    at java.sql.DriverManager.getConnection(DriverManager.java:579)
    at java.sql.DriverManager.getConnection(DriverManager.java:221)
    at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:184)
    at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:76)
    at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
    at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
    ... 8 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at java.net.Socket.connect(Socket.java:528)
    at java.net.Socket.<init>(Socket.java:425)
    at java.net.Socket.<init>(Socket.java:241)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
    ... 26 more

13/01/30 07:57:35 INFO mapred.JobClient: Job complete: job_201301300430_0033
13/01/30 07:57:35 INFO mapred.JobClient: Counters: 7
13/01/30 07:57:35 INFO mapred.JobClient:   Job Counters 
13/01/30 07:57:35 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=22142
13/01/30 07:57:35 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/01/30 07:57:35 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/01/30 07:57:35 INFO mapred.JobClient:     Rack-local map tasks=4
13/01/30 07:57:35 INFO mapred.JobClient:     Launched map tasks=4
13/01/30 07:57:35 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
13/01/30 07:57:35 INFO mapred.JobClient:     Failed map tasks=1
13/01/30 07:57:35 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 28.2556 seconds (0 bytes/sec)
13/01/30 07:57:35 INFO mapreduce.ExportJobBase: Exported 0 records.
13/01/30 07:57:35 ERROR tool.ExportTool: Error during export: Export job failed!

1 个答案:

答案 0 :(得分:1)

我解决了这个问题。

问题是MySQL远程连接。

我用过

sqoop-export --connect jdbc:mysql://localhost/mydb --table tbl_1 --export-dir /user/hive/warehouse/tbl_1 --username test --password test123

但这是错误的。

我应该使用主机名而不是localhost。

我通过这个问题了解了有关sqoop的一件事。

Sqoop运行MapReduce。

因此,Data节点必须能够访问服务器上的MySQL。 (我在名称节点上安装了MySQL。)

另外,/ etc / my.cnf

我评论'bind-address = 127.0.0.1'&amp;跳过联网

然后它运作良好。