使用Sqoop / socket创建HSQL到Hadoop时出错

时间：2019-05-08 19:01:22

标签： hadoop sqoop hsqldb

我正在尝试使用Sqoop版本1.4.6.2.6.5.0-292将数据从HyperSQL数据库表导入Hadoop DFS。我将尝试总结我已采取的步骤：

  1. 首先，我下载并解压缩了HSQL 1.8.0.10版本（与Sqoop中使用的版本相同）。我添加了一个server.properties文件，并像这样配置数据库文件夹：server.database.0=file:test/test

  2. 数据库服务如下启动：

  

nohup java -cp /data/programs/ETU/hsqldb-1.8.0/hsqldb/lib/hsqldb.jar org.hsqldb.Server＆

  1. 创建了一个名为RT_SALES的表并将其填充数据后，我能够使用sqoop list-tables --connect "jdbc:hsqldb:hsql://localhost" --connection-manager "org.apache.sqoop.manager.HsqldbManager" --username "SA"断言来自sqoop的成功连接

结果显示了我希望的一张表RT_SALES。但是，当尝试使用

实际将数据导入HDFS时 
sqoop import --num-mappers 1 --connect "jdbc:hsqldb:hsql://localhost" --connection-manager "org.apache.sqoop.manager.HsqldbManager" --username "SA" --table "RT_SALES" --target-dir "/some-folder"

我收到以下错误：

19/05/08 20:16:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.5.0-292
19/05/08 20:16:13 INFO manager.SqlManager: Using default fetchSize of 1000
19/05/08 20:16:13 INFO tool.CodeGenTool: Beginning code generation
19/05/08 20:16:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
19/05/08 20:16:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
19/05/08 20:16:14 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.6.5.0-292/hadoop-mapreduce
Note: /tmp/sqoop-user1/compile/50b3c7f6e0257e6f477a23139bedcb39/RT_SALES.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/05/08 20:16:16 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-user1/compile/50b3c7f6e0257e6f477a23139bedcb39/RT_SALES.jar
19/05/08 20:16:16 INFO mapreduce.ImportJobBase: Beginning import of RT_SALES
19/05/08 20:16:17 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
19/05/08 20:16:18 INFO client.AHSProxy: Connecting to Application History server at server1/100.00.000.00:10200
19/05/08 20:16:18 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 70501 for user1 on ha-hdfs:server1
19/05/08 20:16:18 INFO security.TokenCache: Got dt for hdfs://server1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:server1, Ident: (HDFS_DELEGATION_TOKEN token 70501 for user1)
19/05/08 20:16:18 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
19/05/08 20:16:18 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
19/05/08 20:16:21 INFO db.DBInputFormat: Using read commited transaction isolation
19/05/08 20:16:21 INFO mapreduce.JobSubmitter: number of splits:1
19/05/08 20:16:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1557311883605_0041
19/05/08 20:16:21 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:server1, Ident: (HDFS_DELEGATION_TOKEN token 70501 for user1)
19/05/08 20:16:22 INFO impl.TimelineClientImpl: Timeline service address: http://server1:8188/ws/v1/timeline/
19/05/08 20:16:23 INFO impl.YarnClientImpl: Submitted application application_1557311883605_0041
19/05/08 20:16:23 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1557311883605_0041/
19/05/08 20:16:23 INFO mapreduce.Job: Running job: job_1557311883605_0041
19/05/08 20:16:31 INFO mapreduce.Job: Job job_1557311883605_0041 running in uber mode : false
19/05/08 20:16:31 INFO mapreduce.Job:  map 0% reduce 0%
19/05/08 20:16:37 INFO mapreduce.Job: Task Id : attempt_1557311883605_0041_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: socket creation error
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.RuntimeException: java.sql.SQLException: socket creation error
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
        ... 9 more
Caused by: java.sql.SQLException: socket creation error
        at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
        at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
        at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
        at org.hsqldb.jdbcDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
        ... 10 more

从其他帖子来看，该消息表明HSQLDB没有运行。但是，我能够使用集成的HSQL客户端进行连接，并且如上所述，sqoop至少可以查询表-因此该服务必须正在运行。

我已经尝试过（没有成功）：

  • 为hsqldb定义其他端口（用9002代替9001）
  • 为用户SA创建密码
  • 设置数据库名称（用“ test”代替“”）

更新：将服务器属性silent设置为false后，我收到以下日志：

[Server@49476842]: [Thread[main,5,main]]: checkRunning(false) entered
[Server@49476842]: [Thread[main,5,main]]: checkRunning(false) exited
[Server@49476842]: Startup sequence initiated from main() method
[Server@49476842]: Loaded properties from [/data/programs/ETU/hsqldb-1.8.0/hsqldb/server.properties]
[Server@49476842]: [Thread[main,5,main]]: start() entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: run() entered
[Server@49476842]: Initiating startup sequence...
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.tls=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.port=9002
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.trace=true
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.database.0=file:test/test
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.restart_on_shutdown=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.no_system_exit=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.silent=false
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.default_page=index.html
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.address=0.0.0.0
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.dbname.0=test
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: server.root=.
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openServerSocket() entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: Got server socket: ServerSocket[addr=0.0.0.0/0.0.0.0,localport=9002]
[Server@49476842]: Server socket opened successfully in 5 ms.
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openServerSocket() exiting
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openDatabases() entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: Opening database: [file:test/test]
[Server@49476842]: Database [index=0, id=0, db=file:test/test, alias=test] opened sucessfully in 118 ms.
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: openDatabases() exiting
[Server@49476842]: Startup sequence completed in 124 ms.
[Server@49476842]: 2019-05-10 12:59:46.675 HSQLDB server 1.8.0 is online
[Server@49476842]: To close normally, connect and execute SHUTDOWN SQL
[Server@49476842]: From command line, use [Ctrl]+[C] to abort abruptly
[Server@49476842]: [Thread[main,5,main]]: start() exiting
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection(Socket[addr=/127.0.0.1,port=38030,localport=9002]) entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection() exited
[Server@49476842]: [Thread[HSQLDB Connection @5e60432f,5,HSQLDB Connections @49476842]]: 0:trying to connect user SA
[Server@49476842]: 0:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT null CONNECTION_READONLY null
[Server@49476842]: 0:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT false CONNECTION_READONLY null
[Server@49476842]: 0:SQLCLI:SQLPREPARE SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
[Server@49476842]: 0:SQLCLI:SQLEXECUTE:1
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLFREESTMT:1
[Server@49476842]: 0:SQLCLI:SQLPREPARE SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
[Server@49476842]: 0:SQLCLI:SQLEXECUTE:2
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLFREESTMT:2
[Server@49476842]: 0:SELECT * FROM INFORMATION_SCHEMA.SYSTEM_PRIMARYKEYS WHERE 1=1 AND TABLE_NAME = 'RT_SALES'
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SELECT * FROM INFORMATION_SCHEMA.SYSTEM_PRIMARYKEYS WHERE 1=1 AND TABLE_NAME = 'RT_SALES'
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLPREPARE SELECT t.* FROM "RT_SALES" AS t WHERE 1=0
[Server@49476842]: 0:SQLCLI:SQLEXECUTE:3
[Server@49476842]: 0:SQLCLI:SQLENDTRAN:COMMIT
[Server@49476842]: 0:SQLCLI:SQLFREESTMT:3
[Server@49476842]: 0:SQLCLI:SQLDISCONNECT
[Server@49476842]: [Thread[HSQLDB Connection @5e60432f,5,HSQLDB Connections @49476842]]: 0:disconnected SA
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection(Socket[addr=/127.0.0.1,port=38190,localport=9002]) entered
[Server@49476842]: [Thread[HSQLDB Server @49476842,5,main]]: handleConnection() exited
[Server@49476842]: [Thread[HSQLDB Connection @7450cadd,5,HSQLDB Connections @49476842]]: 1:trying to connect user SA
[Server@49476842]: 1:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT false CONNECTION_READONLY null
[Server@49476842]: 1:call "org.hsqldb.Library.getDatabaseProductName"()
[Server@49476842]: 1:HSQLCLI:SETSESSIONATTR:AUTOCOMMIT null CONNECTION_READONLY null
[Server@49476842]: 1:SQLCLI:SQLDISCONNECT
[Server@49476842]: [Thread[HSQLDB Connection @7450cadd,5,HSQLDB Connections @49476842]]: 1:disconnected SA

0 个答案:

没有答案
相关问题
最新问题