在mapreduce的map函数中使用了context.write两次

时间:2017-08-23 07:38:52

标签: java hadoop mapreduce

在简单的地图功能中,有两个输入文件。第一个文件行以M开头,第二个文件行以N开头。映射函数为:

public void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {
            int m = 4, p = 4;
            String line = value.toString();
            // (M, i, j, Mij);
            String[] indicesAndValue = line.split(",");
            Text outputKey = new Text();
            Text outputValue = new Text();
            if (indicesAndValue[0].equals("M")) {
             //First file
                for (int k = 1; k <= p; k++) {
                    outputKey.set(indicesAndValue[1] + "," + k);
                    // outputKey.set(i,k);
                    outputValue.set(indicesAndValue[0] + ","
                            + indicesAndValue[2] + "," + indicesAndValue[3]);
                    // outputValue.set(M,j,Mij);
                    context.write(outputKey, outputValue);
                }
            } else {
             //Second file
                // (N, j, k, Njk);
                for (int k = 1; k <= m; k++) {
                    outputKey.set(k + "," + indicesAndValue[2]);
                    outputValue.set("N," + indicesAndValue[1] + ","
                            + indicesAndValue[3]);
                    context.write(outputKey, outputValue);
                }
            }
        }

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { int m = 4, p = 4; String line = value.toString(); // (M, i, j, Mij); String[] indicesAndValue = line.split(","); Text outputKey = new Text(); Text outputValue = new Text(); if (indicesAndValue[0].equals("M")) { //First file for (int k = 1; k <= p; k++) { outputKey.set(indicesAndValue[1] + "," + k); // outputKey.set(i,k); outputValue.set(indicesAndValue[0] + "," + indicesAndValue[2] + "," + indicesAndValue[3]); // outputValue.set(M,j,Mij); context.write(outputKey, outputValue); } } else { //Second file // (N, j, k, Njk); for (int k = 1; k <= m; k++) { outputKey.set(k + "," + indicesAndValue[2]); outputValue.set("N," + indicesAndValue[1] + "," + indicesAndValue[3]); context.write(outputKey, outputValue); } } } 现在,我的两个输入文件是相同的。所以,我想只使用一个输入文件,但我想使用context.write两次。分割线后,将其映射两次。当我更改下面的代码时,reduce函数不会产生任何输出! 改变了地图功能:

public void map(LongWritable key, Text value, Context context)
                throws IOException, InterruptedException {
            Configuration conf = context.getConfiguration();
            int m = 4, p = 4;
            String line = value.toString();
            // (M, i, j, Mij);
            String[] indicesAndValue = line.split(",");
            Text outputKey = new Text();
            Text outputValue = new Text();
            //if (indicesAndValue[0].equals("M")) {
                for (int k = 1; k <= p; k++) {
                    outputKey.set(indicesAndValue[1] + "," + k);
                    // outputKey.set(i,k);
                    outputValue.set(indicesAndValue[0] + ","
                            + indicesAndValue[2] + "," + indicesAndValue[3]);
                    // outputValue.set(M,j,Mij);
                    context.write(outputKey, outputValue);
                    outputKey.set(k + "," + indicesAndValue[2]);
                    outputValue.set("N," + indicesAndValue[1] + ","
                            + indicesAndValue[3]);
                    context.write(outputKey, outputValue);
                }
        }
减少功能: public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { Configuration conf = context.getConfiguration(); int m = 4, p = 4; String line = value.toString(); // (M, i, j, Mij); String[] indicesAndValue = line.split(","); Text outputKey = new Text(); Text outputValue = new Text(); //if (indicesAndValue[0].equals("M")) { for (int k = 1; k <= p; k++) { outputKey.set(indicesAndValue[1] + "," + k); // outputKey.set(i,k); outputValue.set(indicesAndValue[0] + "," + indicesAndValue[2] + "," + indicesAndValue[3]); // outputValue.set(M,j,Mij); context.write(outputKey, outputValue); outputKey.set(k + "," + indicesAndValue[2]); outputValue.set("N," + indicesAndValue[1] + "," + indicesAndValue[3]); context.write(outputKey, outputValue); } }

0 个答案:

没有答案