Question

我正在尝试使用2个输入文件和2个映射器在Java中运行MapReduce程序。

以下是代码：

public class CounterMapper {
    public static class MyMap1 extends
            Mapper<LongWritable, Text, Text, LongWritable> {
        public void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {
            String[] line = value.toString().split("\t");

            int age = Integer.parseInt(line[26]);

            context.write(new Text(line[7]), new LongWritable(age));
        }
    }

    public static class MyMap2 extends Mapper {
        public void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {
            String[] line = value.toString().split("\t");
            int age = Integer.parseInt(line[26]);
            context.write(new Text(line[7]), new LongWritable(age));
        }
    }

    public static class MyRed extends
            Reducer<Text, LongWritable, Text, LongWritable> {
        String line = null;

        public void reduce(Text key, Iterable<LongWritable> values,
                Context context) throws IOException, InterruptedException {
            for (LongWritable value : values) {
                line = value.toString();
            }
            context.write(key, new LongWritable());
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        @SuppressWarnings("deprecation")
        Job job = new Job(conf, "ProjectQuestion2");
        job.setJarByClass(CounterMapper.class);
        FileOutputFormat.setOutputPath(job, new Path(args[2]));

        job.setNumReduceTasks(1);
        job.setMapperClass(MyMap1.class);
        job.setMapperClass(MyMap2.class);
        job.setReducerClass(MyRed.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(LongWritable.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(LongWritable.class);

        MultipleInputs.addInputPath(job, new Path(args[0]),
                TextInputFormat.class, MyMap1.class);
        MultipleInputs.addInputPath(job, new Path(args[1]),
                TextInputFormat.class, MyMap2.class);

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

在运行作业后我得到以下错误：

INFO mapreduce.Job：任务ID：attempt_1486434709675_0016_m_000000_2，状态：FAILED错误：java.io.IOException：输入密钥中的不匹配 map：期待org.apache.hadoop.io.Text，收到 org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask $ MapOutputBuffer.collect（MapTask.java:1073）在 org.apache.hadoop.mapred.MapTask $ NewOutputCollector.write（MapTask.java:715）在 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write（TaskInputOutputContextImpl.java:89）在 org.apache.hadoop.mapreduce.lib.map.WrappedMapper $ Context.write（WrappedMapper.java:112）在org.apache.hadoop.mapreduce.Mapper.map（Mapper.java:124）at org.apache.hadoop.mapreduce.Mapper.run（Mapper.java:145）at at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run（DelegatingMapper.java:55）在org.apache.hadoop.mapred.MapTask.runNewMapper（MapTask.java:787）在org.apache.hadoop.mapred.MapTask.run（MapTask.java:341）at org.apache.hadoop.mapred.YarnChild $ 2.run（YarnChild.java:164）at at java.security.AccessController.doPrivileged（Native Method）at javax.security.auth.Subject.doAs（Subject.java:415）at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1693）在org.apache.hadoop.mapred.YarnChild.main（YarnChild.java:158）

感谢任何意见......谢谢。

Answer 1

它可能使用基类map()中的Mapper方法而不是你的方法。因为这是一个身份映射器（通过），它会与你看到的错误相匹配。

我会做一些事情：

在MyMap2中将Mapper更改为Mapper<LongWritable, Text, Text, LongWritable>。
通过向map()注释添加Mapper注释，确保您的@Override方法覆盖基础Job job = new Job(conf, "ProjectQuestion2");类。

你也可以（改进）：

将Job job = Job.getInstance(conf, "ProjectQuestion2");更改为job.setMapOutputKeyClass()以删除处理警告。
job.setMapOutputValueClass()和$('#dp').datepicker({ onSelect: function(dateText) { var today = new Date(new Date().getFullYear(), new Date().getMonth(), new Date().getDate()).getTime(); var selected = new Date(dateText).getTime(); if (today > selected) alert('prior to today'); else if (today < selected) alert('after today'); else alert('today'); } });设置两次，您可以删除一对。

Hadoop错误：java.io.IOException：键入map中的键不匹配：expected org.apache.hadoop.io.Text，收到org.apache.hadoop.io.LongWritable

1 个答案: