我正在尝试导入一个Teradata表,该表具有一个复合键作为主键。因此设置--num mapper
1并在没有--split-by
参数的情况下运行它。但是,我遇到了以下问题。请帮忙。
Error: java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.00.00.01] [Error 3707] [SQLState 42000] Syntax error, expected something like a name or a Unicode delimited identifier between the word 'Record' and the 'Start' keyword.
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:102)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:298)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:179)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:120)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:111)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:372)
at com.teradata.jdbc.jdbc_4.TDStatement.prepareRequest(TDStatement.java:541)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.<init>(TDPreparedStatement.java:96)
at com.teradata.jdbc.jdk6.JDK6_SQL_PreparedStatement.<init>(JDK6_SQL_PreparedStatement.java:21)
at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.constructPreparedStatement(JDK6_SQL_Connection.java:76)
at com.teradata.jdbc.jdbc_4.TDSession.prepareStatement(TDSession.java:1491)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:101)
答案 0 :(得分:2)
无论--split-by
或--boundary-query
选项如何,您都必须使用--num-mappers
或-m
选项。
拆分列不一定等于PK。你可以有复杂的PK和一些int Split列。但是使用复合键中的一列作为--split-by
选项。
--split-by
~~&gt;用于拆分工作单位的表格列
如果以上选项(仅--boundry-query
)无效,请尝试--split-by
+ --split-by
:
sqoop import \
--connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" \
--username=retail_dba \
--password=cloudera \
--table departments \
--target-dir /user/cloudera/departments \
-m 2 \
--boundary-query "select 2, 8 from departments limit 1" \
--split-by department_id
默认情况下,sqoop将使用查询select min(<split-by>), max(<split-by>) from <table name>
来查找创建拆分的边界。在某些情况下,此查询不是最佳查询,因此您可以使用--boundary-query
参数指定任意返回两个数字列的任意查询。
答案 1 :(得分:0)
尝试使用 -M1 在sqoop命令中而不是 - num mapper 1。