我的输入是很多文本文件。我希望我的map-reduce程序将所有文件名和相关句子与文件名一起写入一个输出文件中，我想从映射器中发出文件名（键）和相关句子（值） 。 reducer将收集键和所有值，并在输出中写入文件名及其相关句子。
这是我的mapper和reducer的代码：
public class WordCount {
public static class Map extends MapReduceBase implements Mapper<LongWritable,
Text, Text, Text> {
public void map(LongWritable key, Text value, OutputCollector<Text,Text>
output, Reporter reporter) throws IOException {
String filename = new String();
FileSplit filesplit = (FileSplit)reporter.getInputSplit();
filename=filesplit.getPath().getName();
output.collect(new Text(filename), value);
}
}
public static class Reduce extends MapReduceBase implements Reducer<Text, Text,
Text, Text> {
public void reduce(Text key, Iterable<Text> values, OutputCollector<Text,
Text> output, Reporter reporter) throws IOException {
StringBuilder builder = new StringBuilder();
for(Text value : values) {
String str = value.toString();
builder.append(str);
}
String valueToWrite=builder.toString();
output.collect(key, new Text(valueToWrite));
}
@Override
public void reduce(Text arg0, Iterator<Text> arg1,
OutputCollector<Text, Text> arg2, Reporter arg3)
throws IOException {
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setJarByClass(WordCount.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setNumReduceTasks(1);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
}
}
输出如下：
14/03/21 00:38:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
14/03/21 00:38:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same.
14/03/21 00:38:27 WARN mapred.JobClient: No job jar file set. User classes may not
be found. See JobConf(Class) or JobConf#setJar(String).
14/03/21 00:38:27 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/21 00:38:27 INFO mapred.FileInputFormat: Total input paths to process : 2
14/03/21 00:38:27 INFO mapred.JobClient: Running job: job_local_0001
14/03/21 00:38:27 INFO util.ProcessTree: setsid exited with exit code 0
14/03/21 00:38:27 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4911b910
14/03/21 00:38:27 INFO mapred.MapTask: numReduceTasks: 1
14/03/21 00:38:27 INFO mapred.MapTask: io.sort.mb = 100
14/03/21 00:38:27 INFO mapred.MapTask: data buffer = 79691776/99614720
14/03/21 00:38:27 INFO mapred.MapTask: record buffer = 262144/327680
14/03/21 00:38:27 INFO mapred.MapTask: Starting flush of map output
14/03/21 00:38:27 INFO mapred.MapTask: Finished spill 0
14/03/21 00:38:27 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And
is in the process of commiting
14/03/21 00:38:28 INFO mapred.JobClient: map 0% reduce 0%
14/03/21 00:38:30 INFO mapred.LocalJobRunner:
file:/root/Desktop/wordcount/sample.txt:0+5371
14/03/21 00:38:30 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
14/03/21 00:38:30 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1f8166e5
14/03/21 00:38:30 INFO mapred.MapTask: numReduceTasks: 1
14/03/21 00:38:30 INFO mapred.MapTask: io.sort.mb = 100
14/03/21 00:38:30 INFO mapred.MapTask: data buffer = 79691776/99614720
14/03/21 00:38:30 INFO mapred.MapTask: record buffer = 262144/327680
14/03/21 00:38:30 INFO mapred.MapTask: Starting flush of map output
14/03/21 00:38:30 INFO mapred.MapTask: Finished spill 0
14/03/21 00:38:30 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And
is in the process of commiting
14/03/21 00:38:31 INFO mapred.JobClient: map 100% reduce 0%
14/03/21 00:38:33 INFO mapred.LocalJobRunner:
file:/root/Desktop/wordcount/sample.txt~:0+587
14/03/21 00:38:33 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
14/03/21 00:38:33 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3963b3e
14/03/21 00:38:33 INFO mapred.LocalJobRunner:
14/03/21 00:38:33 INFO mapred.Merger: Merging 2 sorted segments
14/03/21 00:38:33 INFO mapred.Merger: Down to the last merge-pass, with 2 segments
left of total size: 7549 bytes
14/03/21 00:38:33 INFO mapred.LocalJobRunner:
14/03/21 00:38:33 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And
is in the process of commiting
14/03/21 00:38:33 INFO mapred.LocalJobRunner:
14/03/21 00:38:33 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to
commit now
14/03/21 00:38:33 INFO mapred.FileOutputCommitter: Saved output of task
'attempt_local_0001_r_000000_0' to file:/root/Desktop/wordcount/output
14/03/21 00:38:36 INFO mapred.LocalJobRunner: reduce > reduce
14/03/21 00:38:36 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
14/03/21 00:38:37 INFO mapred.JobClient: map 100% reduce 100%
14/03/21 00:38:37 INFO mapred.JobClient: Job complete: job_local_0001
14/03/21 00:38:37 INFO mapred.JobClient: Counters: 21
14/03/21 00:38:37 INFO mapred.JobClient: File Input Format Counters
14/03/21 00:38:37 INFO mapred.JobClient: Bytes Read=5958
14/03/21 00:38:37 INFO mapred.JobClient: File Output Format Counters
14/03/21 00:38:37 INFO mapred.JobClient: Bytes Written=8
14/03/21 00:38:37 INFO mapred.JobClient: FileSystemCounters
14/03/21 00:38:37 INFO mapred.JobClient: FILE_BYTES_READ=26020
14/03/21 00:38:37 INFO mapred.JobClient: FILE_BYTES_WRITTEN=117337
14/03/21 00:38:37 INFO mapred.JobClient: Map-Reduce Framework
14/03/21 00:38:37 INFO mapred.JobClient: Map output materialized bytes=7557
14/03/21 00:38:37 INFO mapred.JobClient: Map input records=122
14/03/21 00:38:37 INFO mapred.JobClient: Reduce shuffle bytes=0
14/03/21 00:38:37 INFO mapred.JobClient: Spilled Records=244
14/03/21 00:38:37 INFO mapred.JobClient: Map output bytes=7301
14/03/21 00:38:37 INFO mapred.JobClient: Total committed heap usage
(bytes)=954925056
14/03/21 00:38:37 INFO mapred.JobClient: CPU time spent (ms)=0
14/03/21 00:38:37 INFO mapred.JobClient: Map input bytes=5958
14/03/21 00:38:37 INFO mapred.JobClient: SPLIT_RAW_BYTES=185
14/03/21 00:38:37 INFO mapred.JobClient: Combine input records=0
14/03/21 00:38:37 INFO mapred.JobClient: Reduce input records=0
14/03/21 00:38:37 INFO mapred.JobClient: Reduce input groups=2
14/03/21 00:38:37 INFO mapred.JobClient: Combine output records=0
14/03/21 00:38:37 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
14/03/21 00:38:37 INFO mapred.JobClient: Reduce output records=0
14/03/21 00:38:37 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
14/03/21 00:38:37 INFO mapred.JobClient: Map output records=122
当我使用相同的inputformat（keyvaluetextinputformat.class）配置运行上面的mapper和reducer时，它不会在输出中写入任何内容。
为实现目标，我应该改变什么？
答案 0 :(得分：0)
KeyValueTextInputFormat不是您的案例的正确输入格式。如果要使用此输入格式，输入中的每一行应包含一个键值对，默认情况下由用户指定的分隔符或制表符分隔。但在您的情况下，输入是＆＃34;文件集＆＃34;并且您希望输出作业为＆＃34;文件名，文件内容＆＃34;。
实现此目的的一种方法是使用TextInputFormat作为输入格式。我已在下面进行了测试，但它确实有效。
在地图功能中获取文件的文件名和内容
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException
{
String filename = new String();
FileSplit filesplit = (FileSplit)context.getInputSplit();
filename=filesplit.getPath().getName();
context.write(new Text(filename), new Text(value));
}
在reduce函数中，我们构建所有值的字符串，这些值将是文件的内容
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException
{
StringBuilder builder= new StringBuilder();
for (Text value : values)
{
String str = value.toString();
builder.append(str);
}
String valueToWrite= builder.toString();
context.write(key, new Text(valueToWrite));
}
}
最后在作业驱动程序类中，将inputformat设置为我们的自定义格式，并将reducers中没有设置为1
job.setInputFormatClass(TextInputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(myMapper.class);
job.setReducerClass(myReducer.class);
job.setNumReduceTasks(1);