Question

我有下表：MYSQL中的mergetab。

id | name      | city        | rectime             |
---+-----------+-------------+---------------------+
 1 | Sidhartha | Hyderabad   | 2017-04-18 15:31:22 |
 2 | Saketh    | Bengaluru   | 2017-04-18 15:32:37 |
 3 | Sunny     | Mumbai      | 2017-04-18 15:32:57 |
 4 | Bobby     | Delhi       | 2017-04-18 15:33:15 |

我在其中插入了一条记录，如下所示：

id | name      | city        | rectime             |
---+-----------+-------------+---------------------+
 1 | Sidhartha | Hyderabad   | 2017-04-18 15:31:22 |
 2 | Saketh    | Bengaluru   | 2017-04-18 15:32:37 |
 3 | Sunny     | Mumbai      | 2017-04-18 15:32:57 |
 4 | Bobby     | Delhi       | 2017-04-18 15:33:15 |
 5 | Madhavi   | Dharmavaram | 2017-04-18 16:57:09 |

我尝试使用以下命令执行sqoop增量导入：

sqoop import --connect jdbc:mysql://127.0.0.1/mydb --table mergetab --username root --password cloudera --hive-import --hive-table mergetab --incremental-append --check-column id --last-value $(hive -S -e "select max(id) from mergetab");

我得到以下错误，我无法理解我的sqoop命令有什么问题。错误：

17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: The
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: method
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: class
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: org.apache.commons.logging.impl.SLF4JLogFactory#release()
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: was
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: invoked.
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: Please
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: see
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: http://www.slf4j.org/codes.html#release
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: for
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: an

我在Cloudera VM中练习这个。谁能告诉我我在这里犯的错误是什么？

Answer 1

您的代码行--last-value $(hive -S -e "select max(id) from mergetab")的问题是，即使在静默模式下，配置单元也会始终打印标题/日志记录信息，因此，last-value将始终获得如下值 -

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.1.0/libexec/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
5

您需要编写一些扭曲的shell脚本来提取最后一行，或者您可以使用直线showHeader和outputformat，如下所示，只是获取直接赋值给变量的值

--last value $(beeline --showHeader=false --outputformat=tsv2 -e "your query")

Sqoop增量导入失败

1 个答案: