Question

我试图在mapper和reducer之间传递一个复杂的可写，更具体地说是ObjectWritables的ArrayWritable。

public class ObjectArrayWritable extends ArrayWritable {
    public ObjectArrayWritable() {
        super(ObjectWritable.class);
    }
}

我的映射器：Mapper<LongWritable, Text, Text, ObjectArrayWritable>

我的减速机：Reducer<Text,ObjectArrayWritable,Text,ObjectArrayWritable>

reducer在可写接口的readfields()方法上崩溃并抛出以下异常：

java.lang.NullPointerException
    at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:183)
    at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
    at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)

当我尝试实现自己的自定义可写并调试它时，我注意到readfields(DataInput data)中DataInput的缓冲区被收到null ..

请注意，我的错误类似于： hadoop + Writable interface + readFields throws an exception in reducer但清洁对象没有帮助..

有人可以帮忙吗？

Answer 1

我不认为你正在处理ObjectArrayWritable中没有任何内容的情况。我不能确定，因为没有代码

复位类型的reducer中的ReadFields（）中的null缓冲区

1 个答案: