Question

我正在尝试从包含二进制形式的数据类型UUID的字段的Mongo集合中加载数据（例如BinData（3，“/ qHWF5hGQU + w6unYcTQxWw ==”））。该作业失败

org.apache.pig.backend.executionengine.ExecException: ERROR 2108: \
  Could not determine data type of field: 1423ed53-5064-0000-784b-7bf2e2dd837b".

我构建了mongo-hadoop 1.1版（来自Master分支）。 https://github.com/mongodb/mongo-hadoop。它有效，除非有UUID。下面是我的脚本和错误。有什么想法吗？

register '/pig/lib/mongo-java-driver-2.9.3.jar';
register '/pig/lib/mongo-hadoop-core_cdh4.3.0-1.1.0.jar';
register '/pig/lib/mongo-hadoop-pig_cdh4.3.0-1.1.0.jar';
a = LOAD 'mongodb://localhost/TestDb.SocialUser'
      USING com.mongodb.hadoop.pig.MongoLoader();
store a INTO 'a';

2013-07-10 15:03:35,630 [Thread-6] INFO  org.apache.hadoop.mapred.LocalJobRunner - Map task executor complete.
2013-07-10 15:03:35,632 [Thread-6] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local402930066_0001
java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 2108: Could not determine data type of field: 1423ed53-5064-0000-784b-7bf2e2dd837b
  at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2108: \
    Could not determine data type of field: 1423ed53-5064-0000-784b-7bf2e2dd837b
  at org.apache.pig.impl.util.StorageUtil.putField(StorageUtil.java:208)
  at org.apache.pig.impl.util.StorageUtil.putField(StorageUtil.java:166)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat$PigLineRecordWriter.write(PigTextOutputFormat.java:68)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat$PigLineRecordWriter.write(PigTextOutputFormat.java:44)
  at org.apache.pig.builtin.PigStorage.putNext(PigStorage.java:296)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
  at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:558)
  at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
  at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:264)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
  at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:266)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:724)
2013-07-10 15:03:39,235 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.

Answer 1

MongoLoader有一个方法convertBSONtoPigType，用于将记录阅读器返回的类型转换为与pig兼容的类型。如果类型不是可识别的类型 - 即包括java.util.Date，则该方法默认输出对象并破坏猪。

如果向mongo加载器添加一个模式，该模式为UUID提供一个char数组类型的char数组，例如

加载'/mongodb://mongoserver/db.collection'使用MongoLoader（myguid：chararray）底层java代码调用.toString（）对象（在本例中为java.util.UUID）并输出一个正常的UUID。

你也可以切实改变convertBSONtoPigType方法来做同样的事情。

public static Object convertBSONtoPigType(final Object o) throws ExecException {
    if (o == null) {
        return null;
    } else if (o instanceof Number || o instanceof String) {
        return o;
    } else if (o instanceof Date) {
        return ((Date) o).getTime();
    } else if (o instanceof ObjectId) {
        return o.toString();
    } else if (o instanceof UUID) {
        return o.toString();
    }
    else if (o instanceof BasicBSONList) {
        BasicBSONList bl = (BasicBSONList) o;
        Tuple t = tupleFactory.newTuple(bl.size());
        for (int i = 0; i < bl.size(); i++) {
            t.set(i, convertBSONtoPigType(bl.get(i)));
        }
        return t;
    } else if (o instanceof Map) {
        //TODO make this more efficient for lazy objects?
        Map<String, Object> fieldsMap = (Map<String, Object>) o;
        HashMap<String, Object> pigMap = new HashMap<String, Object>(fieldsMap.size());
        for (Map.Entry<String, Object> field : fieldsMap.entrySet()) {
            pigMap.put(field.getKey(), convertBSONtoPigType(field.getValue()));
        }
        return pigMap;
    } else {
        return o;
    }

}

令我困惑的是MongoLoader不支持具有未知架构的UUID的原因。原因是，UUID / BinData是Mongo的一部分并被广泛使用。

也许这是他们可以解决的问题。

无论如何 - 希望这有帮助。

此致

Answer 2

这是因为UUID不是Pig中的内置类型。这篇文章给出了解决方案。

https://groups.google.com/forum/#!topic/mongodb-user/jAijeRtOG0o

Pig MongoLoader异常使用UUID加载数据

2 个答案: