Question

该查询与mysql完美配合，但在与Sqoop一起使用时，将数据从MySql导入Hive时失败。

select cmpdl.*,ct.value from (SELECT cmp.domain_id,
       cmp.model,
       cmp.name,
       cmp.status,
       br.name,
       cat.category,
       dl.start_date,
       dl.end_date,
       cmp.created_at,
       cmp.updated_at
FROM domain cmp
JOIN brands br ON cmp.brand_id=br.brands_id
JOIN delxxx dl ON cmp.del_id=dl.del_id
JOIN
  (SELECT c.domain_id,
          group_concat(b.category_name) AS category
   FROM domain c
   JOIN categories b ON find_in_set(b.category_id,xxxxx)
   GROUP BY c.domain_id) cat ON cmp.domain_id=cat.domain_id
WHERE cmp.domain_type='XXXXXX'
  AND cmp.category='XXXXXXX'
  AND cmp.model != 'XXX') cmpdl join context_targeting ct on cmpdl.domain_id=ct.domain_id and ct.context_targeting_group='xxxxxxx' AND $CONDITIONS;

错误记录

Error: java.io.IOException: SQLException in nextKeyValue
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'cmp.domain_id' in 'on clause'
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:389)
    at com.mysql.jdbc.Util.getInstance(Util.java:372)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2535)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1911)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2034)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
    ... 12 more

我无法弄清楚别名如何超出范围并且未找到列的确切问题。

Answer 1

我的sqoop 拆分属性存在问题

其中我提到了cmp.domain_id来代替cmpdl.domain_id，因为Sqoop Map-Reduce工作失败了

Sqoop导入失败，并在'on子句'中显示未知列

1 个答案: