Question

当我从终端运行sqoop export命令时，它运行正常。但是，如果我从oozie工作流程中运行相同的命令，则会抛出以下错误。

ror: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:122)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)

Caused by: java.lang.RuntimeException: Can't parse input data: '2018-05-14,967,893,74,8863.330000000005,7617.07,1246.26'
at adjust_jazz_compare.__loadFromFields(adjust_jazz_compare.java:512)
at adjust_jazz_compare.parse(adjust_jazz_compare.java:430)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
... 10 more Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:854)
at adjust_jazz_compare.__loadFromFields(adjust_jazz_compare.java:482)
... 12 more

下面是我正在使用的命令

export --connect <jdbc> --username <user> --password <pass> --table <table> --export-dir <dir> --input-fields-terminated-by ',' --input-lines-terminated-by '\n'

配置单元中的表格属性

hive.inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=,, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{totalSize=3973, numRows=70, rawDataSize=3903, COLUMN_STATS_ACCURATE=true, numFiles=1, transient_lastDdlTime=1530647041}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

Answer 1

其中包含的行 "'2018-05-14,967,893,74,8863.330000000005,7617.07,1246.26'"无法在sqoop中进行解析。

从Hive终端检查此行的格式正确。可能是某些特殊字符或空格，因为该数据无法解析。

请共享配置单元中存在的表的架构以及您的配置单元查询。

Answer 2

可以显示如何创建表格吗？也许您忘记了Hive中的字段分隔符？

FIELDS TERMINATED BY ','

示例创建表代码：

CREATE TABLE IF NOT EXISTS employee (
    eid int, 
    name String,
    salary String, 
    destination String)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

Sqoop导出失败。无法解析输入数据：'<data>'

2 个答案: