Question

编辑：问题已解决 - 我有一个非常愚蠢的错误。

我有一个MapReduce管道，包含map，reduce，map和reduce。我使用SequenceFileOutputFormat作为第一个reduce，使用SequenceFileInputFormat作为第二个map。我看过它的用法，看起来我正在使用它。我正在使用的类型是IntWritable和IntPairArrayWritable（一个使用来自mahout的IntPairWritable的自定义ArrayWritable子类）。问题是当在第二个映射中读取IntPairArrayWritable时，当我尝试将单个IntPairWritables取出时，我得到一个ClassCastException。我不确定这是由于我如何使用ArrayWritable类的错误，或者我使用SequenceFile {Input，Output}格式是否有问题。我在这里和其他地方看了很多例子，它看起来像我正在做他们两个正确，但我仍然得到一个错误。有什么帮助吗？

细节：

这是我的第一个减速器类：

public static class WalkIdReducer extends MapReduceBase implements
        Reducer<IntWritable, IntPairWritable, IntWritable, IntPairArrayWritable> {

    @Override
    public void reduce(IntWritable walk_id, Iterator<IntPairWritable> values,
            OutputCollector<IntWritable, IntPairArrayWritable> output,
            Reporter reporter) throws IOException {
        ArrayList<IntPairWritable> value_array = new ArrayList<IntPairWritable>();
        while (values.hasNext()) {
            value_array.add(values.next());
        }
        output.collect(walk_id, IntPairArrayWritable.fromArrayList(value_array));
    }
}

第二个映射器类：

public static class NodePairMapper extends MapReduceBase implements
        Mapper<IntWritable, IntPairArrayWritable, IntPairWritable, Text> {

    @Override
    public void map(IntWritable key, IntPairArrayWritable value,
            OutputCollector<IntPairWritable, Text> output,
            Reporter reporter) throws IOException {
        // The following line gives a ClassCastException;
        // See IntPairArrayWritable.toArrayList(), below
        ArrayList<IntPairWritable> values = value.toArrayList();
        // other unimportant stuff
    }
}

第一个MapReduce的作业配置的相关部分：

    conf.setReducerClass(WalkIdReducer.class);
    conf.setOutputKeyClass(IntWritable.class);
    conf.setOutputValueClass(IntPairArrayWritable.class);
    conf.setOutputFormat(SequenceFileOutputFormat.class);

第二个MapReduce：

    conf.setInputFormat(SequenceFileInputFormat.class);
    conf.setMapperClass(NodePairMapper.class);

最后，我的ArrayWritable子类：

public static class IntPairArrayWritable extends ArrayWritable
{
    // These two methods are what people say is all you need for
    // creating an ArrayWritable subclass
    public IntPairArrayWritable() {
        super(IntPairArrayWritable.class);
    }

    public IntPairArrayWritable(IntPairWritable[] values) {
        super(IntPairArrayWritable.class, values);
    }

    // Some convenience methods, so I can use ArrayLists in
    // other parts of the code
    public static IntPairArrayWritable fromArrayList(
            ArrayList<IntPairWritable> array) {
        IntPairArrayWritable writable = new IntPairArrayWritable();
        IntPairWritable[] values = new IntPairWritable[array.size()];
        for (int i=0; i<array.size(); i++) {
            values[i] = array.get(i);
        }
        writable.set(values);
        return writable;
    }

    public ArrayList<IntPairWritable> toArrayList() {
        ArrayList<IntPairWritable> array = new ArrayList<IntPairWritable>();
        for (Writable pair : this.get()) {
            // This line is what kills it.  I get a ClassCastException here.
            IntPairWritable int_pair = (IntPairWritable) pair;
            array.add(int_pair);
        }
        return array;
    }
}

我得到的具体错误如下：

java.lang.ClassCastException: WalkAnalyzer$IntPairArrayWritable cannot be cast to org.apache.mahout.common.IntPairWritable
at WalkAnalyzer$IntPairArrayWritable.toArrayList(WalkAnalyzer.java:231)
at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:84)
at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:77)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)

我很困惑为什么来自ArrayWritable的get（）方法的内容是WalkAnalyzer$IntPairArrayWritable的实例 - 我期望get（）返回{{{}}所包含的元素数组1}}，如API中所述。

修改

我发现了问题。我是如何编写IntPairArrayWritable的构造函数的。当我打电话给IntPairArrayWritable时，我打电话给super(IntPairArrayWritable.class);。代码应该看起来像这样：

super(IntPairWritable.class);

我认为对ArrayWritable子类使用一个不太明显混淆的名称是个好主意，因此错误更容易被发现。

Answer 1

检查IntPairWritable的import语句。看起来你在Mapper中选择了错误的包名，因此即使它的名字也是IntPairWritable，它也会转换为差异类。

Hadoop ArrayWritable给了我一个ClassCastException

1 个答案: