Question

在我的工作输出中，我为输出中打印的每条预期...Job: map 100% reduce 0%行打印了...Job: map n% reduce -%行。除此之外，工作按预期运行。

见下面的第6,22和28行（以星号为前缀）。任何想法为什么会发生，或者它意味着什么？

14/03/12 14:51:56 INFO mapred.LocalJobRunner:
14/03/12 14:51:56 INFO mapred.MapTask: Starting flush of map output
14/03/12 14:51:56 INFO mapred.MapTask: Spilling map output
14/03/12 14:51:56 INFO mapred.MapTask: bufstart = 0; bufend = 51192402; bufvoid = 104857600
14/03/12 14:51:56 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 18693168(74772672); length = 7521229/6553600
** 14/03/12 14:51:57 INFO mapreduce.Job:  map 25% reduce 0%
14/03/12 14:51:59 INFO mapred.LocalJobRunner: map > sort
14/03/12 14:51:59 INFO mapred.MapTask: Finished spill 0
14/03/12 14:51:59 INFO mapred.Task: Task:attempt_local1547766427_0001_m_000007_0 is done. And is in the process of committing
14/03/12 14:51:59 INFO mapred.LocalJobRunner: map
14/03/12 14:51:59 INFO mapred.Task: Task 'attempt_local1547766427_0001_m_000007_0' done.
14/03/12 14:51:59 INFO mapred.LocalJobRunner: Finishing task: attempt_local1547766427_0001_m_000007_0
14/03/12 14:51:59 INFO mapred.LocalJobRunner: Starting task: attempt_local1547766427_0001_m_000008_0
14/03/12 14:51:59 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/03/12 14:51:59 INFO mapred.MapTask: Processing split: hdfs://<removed>.lzo:0+21976289
14/03/12 14:51:59 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/03/12 14:51:59 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/03/12 14:51:59 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/03/12 14:51:59 INFO mapred.MapTask: soft limit at 83886080
14/03/12 14:51:59 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/03/12 14:51:59 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
** 14/03/12 14:52:00 INFO mapreduce.Job:  map 100% reduce 0%
14/03/12 14:52:02 INFO mapred.LocalJobRunner:
14/03/12 14:52:02 INFO mapred.MapTask: Starting flush of map output
14/03/12 14:52:02 INFO mapred.MapTask: Spilling map output
14/03/12 14:52:02 INFO mapred.MapTask: bufstart = 0; bufend = 52931779; bufvoid = 104857600
14/03/12 14:52:02 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 18670736(74682944); length = 7543661/6553600
** 14/03/12 14:52:03 INFO mapreduce.Job:  map 29% reduce 0%

修改

我仍然不知道为什么会发生这种情况，但我错误地以本地模式运行。以群集模式运行作业（就是它所调用的），显示预期的输出。

Answer 1

看来你没有减速器。你在jobTracker localhost上验证了吗？

Answer 2

您似乎正在使用旧的API; 通过查看＆＃34; org.apache.hadoop.mapred＆＃34;，试试＆＃34; org.apache.hadoop.mapreduce＆＃34;代替。

在Hadoop Java作业中奇怪的“映射100％减少0％@输出

2 个答案: