Question

我正在处理sqoop增量导入命令。但我最终收到错误信息，我无法理解问题出在哪里。下面是我的MySQL表数据

+----+-----------+
| ID | NAME      |
+----+-----------+
|  1 | Sidhartha |
|  2 | Sunny     |
|  3 | Saketh    |
|  4 | Bobby     |
|  5 | Yash      |
|  6 | Nimmi     |
+----+-----------+

包含4条记录的Hive表：DAY是分区列

importedtable.id    importedtable.name  importedtable.day
1   Sidhartha   1
2   Sunny   1
3   Saketh  1
4   Bobby   1

我的Sqoop命令：

sqoop import --connect jdbc:mysql://127.0.0.1/mydb --table MYTAB --driver com.mysql.jdbc.Driver --username root --password cloudera --hive-import --hive-table importedtable --incremental append --check-column id --last-value $(hive -e "select max(id) from importedtable") --target-dir '/home/incdata';

错误讯息：

17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: The
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: method
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: class
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: org.apache.commons.logging.impl.SLF4JLogFactory#release()
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: was
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: invoked.
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: Please
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: see
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: http://www.slf4j.org/codes.html#release
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: for
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: an
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: explanation.
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: --target-dir
17/03/08 12:15:14 ERROR tool.BaseSqoopTool: Unrecognized argument: /home/incdata

任何人都可以告诉我在做sqoop命令时的错误是什么。

Answer 1

问题在于将hive查询作为--last-value参数的值传递，

--last-value $(hive -e "select max(id) from importedtable")

这会将日志消息以及结果发送到--last-value。

对查询使用-S（ - silent）标志

--last-value $(hive -S -e "select max(id) from importedtable")

sqoop incremental import命令出错

1 个答案: