Question

我有一个MapReduce作业，它读取avro数据，然后输出avro数据。但是，当我在作业成功时检查输出文件时，它们没有.avro扩展名，我可以使用简单的文本编辑器查看它们。

我将我的驱动程序配置为输出avro所以我不确定问题是什么，并且非常感谢任何帮助。

这是我的Driver类：

public class Driver extends Configured implements Tool{

public static void main(String[] args) throws Exception {
    int res =
            ToolRunner.run(new Configuration(), new Driver(), args);
    System.exit(res);
}

@Override
public int run(String[] args) throws Exception {
    Job job = new Job(getConf());
    job.setJarByClass(Driver.class);
    job.setJobName("nearestpatient");


    AvroJob.setOutputKeySchema(job, Pair.getPairSchema(Schema.create(Schema.Type.LONG), Schema.create(Schema.Type.STRING)));
    job.setOutputValueClass(NullWritable.class);

    job.setMapperClass(PatientMapper.class);
    job.setReducerClass(PatientReducer.class);

    job.setInputFormatClass(AvroKeyInputFormat.class);
    AvroJob.setInputKeySchema(job, PatientAvro.getClassSchema());

    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(LongWritable.class);


    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);

    return 0;
}
}

这是我的Reducer类：

public class PatientReducer extends Reducer<LongWritable, LongWritable, AvroWrapper<Pair<Long, String>>, NullWritable> {

    @Override
    public void reduce(LongWritable providerKey, Iterable<LongWritable> patients, Context context) throws IOException, InterruptedException {

        String outputList = "[";
   `enter code here` List<Long> patientList = new ArrayList<>();
    for (LongWritable patientKey : patients) {
        outputList += new LongWritable(patientKey.get()) + ", ";
    }
    outputList = outputList.substring(0, outputList.length() - 2);
    outputList += "]";
    context.write(new AvroWrapper<Pair<Long, String>>(new Pair<Long, String> (providerKey.get(), outputList)), NullWritable.get());
}
}

Answer 1

在run（）方法中，您需要添加以下内容

job.setOutputFormatClass(AvroKeyValueOutputFormat.class);

Answer 2

In your code replace line

FileOutputFormat.setOutputPath(job, new Path(args[1]));

with

job.setOutputFormatClass(AvroKeyOutputFormat.class);
AvroKeyOutputFormat.setOutputPath(job, new Path(args[1]));

MapReduce Avro输出正在创建文本文件

2 个答案: