Question

我有以下用于MR工作的课程，但是当我运行工作时，工作失败，并提出以下异常建议。

    public class MongoKey implements WritableComparable<MongoKey> {
    ...
    private Text name;
private Text place;

public MongoKey() {

    this.name = new Text();
    this.place = new Text();

}

public MongoKey(Text name, Text place) {

    this.name = name;
    this.place = place;

}

public void readFields(DataInput in) throws IOException {

    name.readFields(in);
    place.readFields(in);

}

public void write(DataOutput out) throws IOException {

    name.write(out);
    place.write(out);

}
    public int compareTo(MongoKey o) {

            MongoKey other = (MongoKey)o;
            int cmp = name.compareTo(other.name);
            if(cmp != 0){
                return cmp;
            }

            return place.compareTo(other.place);
        }
    }

public class MongoValue implements Writable {
...
public void readFields(DataInput in) throws IOException {

        profession.readFields(in);
    }

    public void write(DataOutput out) throws IOException {

        profession.write(out);
    }
}

public class MongoReducer extends Reducer<MongoKey, MongoValue, MongoKey, BSONWritable> {

...
context.write(key, new BSONWritable(output)); // line 41
}

public class MongoHadoopJobRunner extends Configured implements Tool {

    public int run(String[] args) throws Exception {

        if (args.length != 2) {
            System.out.println("usage: [input] [output]");
            System.exit(-1);
        }

        Configuration conf = getConf();
        for (String arg : args)
            System.out.println(arg);
        GenericOptionsParser parser = new GenericOptionsParser(conf, args);

        conf.set("mongo.output.uri", "mongodb://localhost/demo.logs_aggregate");

        MongoConfigUtil.setOutputURI(conf, "mongodb://localhost/demo.logs_aggregate");
        MongoConfigUtil.setOutputFormat(conf, MongoOutputFormat.class);

        final Job job = new Job(conf, "mongo_hadoop");
        job.setOutputFormatClass(MongoOutputFormat.class);
        // Job job = new Job();

        job.setJarByClass(MongoHadoopJobRunner.class);

        // job.setJobName("mongo_hadoop");
        job.setNumReduceTasks(1);

        job.setMapperClass(MongoMapper.class);
        job.setReducerClass(MongoReducer.class);

        job.setMapOutputKeyClass(MongoKey.class);
        job.setMapOutputValueClass(MongoValue.class);

        job.setOutputKeyClass(MongoKey.class);
        job.setOutputValueClass(BSONWritable.class);

        job.setInputFormatClass(MongoInputFormat.class);
        for (String arg2 : parser.getRemainingArgs()) {
            System.out.println("remaining: " + arg2);
        }
        Path inPath = new Path(parser.getRemainingArgs()[0]);
        MongoInputFormat.addInputPath(job, inPath);

        job.waitForCompletion(true);

        return 0;
    }

    public static void main(String[] pArgs) throws Exception {
        Configuration conf = new Configuration();
        for (String arg : pArgs) {
            System.out.println(arg);
        }
        GenericOptionsParser parser = new GenericOptionsParser(conf, pArgs);
        for (String arg2 : parser.getRemainingArgs()) {
            System.out.println("ree" + arg2);
        }
        System.exit(ToolRunner.run(conf, new MongoHadoopJobRunner(), parser
                .getRemainingArgs()));
    }

}

出现以下异常

java.lang.Exception: java.lang.IllegalArgumentException: can't serialize class com.name.custom.MongoKey
...
...
at com.mongodb.hadoop.output.MongoRecordWriter.write(MongoRecordWriter.java:93)
     at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
     at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
     at com.name.custom.MongoReducer.reduce(MongoReducer.java:41)
     at com.name.custom.MongoReducer.reduce(MongoReducer.java:11)

似乎代码不应该有任何问题，但为什么它无法序列化字段我完全无能为力。

非常感谢

Answer 1

正如我从MongoRecordWriter source code看到的那样，它不支持任意WritableComparable个对象作为键。您可以使用其中一个类作为密钥：BSONWritable，BSONObject，Text，UTF8，简单的包装器，如IntWritable。另外我认为你可以使用Serializable对象作为关键。所以我建议你两个解决方法：

制作MongoKey serializable（implements Serializable，实施writeObject，readObject方法。
使用其中一个受支持的类作为键，例如，您可以使用Text作为键：Text key = new Text(name.toString() + "\t" + place.toString());

Answer 2

此：

java.lang.Exception: java.lang.IllegalArgumentException: can't serialize class com.name.custom.MongoKey

引发了异常，因为MongoKey没有实现java.io.Serializable。

将Serializable添加到您的类声明

WritableComparable对象不可序列化

2 个答案: