在我的map-reduce作业中,当我将reducer的数量设置为0时,程序在不到5分钟的时间内成功完成。但是,当我将reducer的数量设置为某个而不是0(1-40之间)甚至没有指定reducer类时,它会在许多失败的任务中进行得非常缓慢,并突然失败并显示以下错误消息。由于我没有指定reducer类,唯一可能导致此错误的是对映射器的输出进行排序。映射器产生大约27,000个不同的密钥。这里有什么问题?日志文件中没有任何东西可以帮助。我唯一的猜想是大量的密钥导致了这个问题。我该如何解决这个问题?
13/10/28 13:06:40 INFO mapred.JobClient: map 70% reduce 18%
13/10/28 13:07:13 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000052_2, Status : FAILED
Task attempt_201310101346_0661_m_000052_2 failed to report status for 600 seconds. Killing!
13/10/28 13:07:19 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000059_2, Status : FAILED
Task attempt_201310101346_0661_m_000059_2 failed to report status for 600 seconds. Killing!
13/10/28 13:07:23 INFO mapred.JobClient: map 70% reduce 19%
13/10/28 13:07:24 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000060_2, Status : FAILED
Task attempt_201310101346_0661_m_000060_2 failed to report status for 600 seconds. Killing!
13/10/28 13:07:26 INFO mapred.JobClient: map 70% reduce 20%
13/10/28 13:07:47 INFO mapred.JobClient: map 71% reduce 20%
13/10/28 13:07:54 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000061_2, Status : FAILED
Task attempt_201310101346_0661_m_000061_2 failed to report status for 600 seconds. Killing!
13/10/28 13:08:22 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000062_2, Status : FAILED
Task attempt_201310101346_0661_m_000062_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:03 INFO mapred.JobClient: map 72% reduce 20%
13/10/28 13:09:15 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000063_2, Status : FAILED
Task attempt_201310101346_0661_m_000063_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:18 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000066_2, Status : FAILED
Task attempt_201310101346_0661_m_000066_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:20 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000068_2, Status : FAILED
Task attempt_201310101346_0661_m_000068_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:55 INFO mapred.JobClient: map 72% reduce 21%
13/10/28 13:10:30 INFO mapred.JobClient: map 72% reduce 22%
13/10/28 13:10:33 INFO mapred.JobClient: map 73% reduce 22%
13/10/28 13:12:21 INFO mapred.JobClient: map 74% reduce 22%
13/10/28 13:13:40 INFO mapred.JobClient: map 74% reduce 23%
13/10/28 13:14:09 INFO mapred.JobClient: Job complete: job_201310101346_0661
13/10/28 13:14:09 INFO mapred.JobClient: Counters: 31
13/10/28 13:14:09 INFO mapred.JobClient: File System Counters
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of bytes read=0
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of bytes written=542433323
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of read operations=0
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of bytes read=316452240
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of bytes written=0
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of read operations=100
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient: Job Counters
13/10/28 13:14:09 INFO mapred.JobClient: Failed map tasks=1
13/10/28 13:14:09 INFO mapred.JobClient: Launched map tasks=169
13/10/28 13:14:09 INFO mapred.JobClient: Launched reduce tasks=1
13/10/28 13:14:09 INFO mapred.JobClient: Data-local map tasks=23
13/10/28 13:14:09 INFO mapred.JobClient: Rack-local map tasks=146
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=72419457
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=1194624
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 13:14:09 INFO mapred.JobClient: Map-Reduce Framework
13/10/28 13:14:09 INFO mapred.JobClient: Map input records=6894035
13/10/28 13:14:09 INFO mapred.JobClient: Map output records=6422195
13/10/28 13:14:09 INFO mapred.JobClient: Map output bytes=2300749768
13/10/28 13:14:09 INFO mapred.JobClient: Input split bytes=6200
13/10/28 13:14:09 INFO mapred.JobClient: Combine input records=6422195
13/10/28 13:14:09 INFO mapred.JobClient: Combine output records=801608
13/10/28 13:14:09 INFO mapred.JobClient: Spilled Records=801608
13/10/28 13:14:09 INFO mapred.JobClient: CPU time spent (ms)=48077980
13/10/28 13:14:09 INFO mapred.JobClient: Physical memory (bytes) snapshot=13246660608
13/10/28 13:14:09 INFO mapred.JobClient: Virtual memory (bytes) snapshot=20080971776
13/10/28 13:14:09 INFO mapred.JobClient: Total committed heap usage (bytes)=10181214208
13/10/28 13:14:09 INFO mapred.JobClient: org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/10/28 13:14:09 INFO mapred.JobClient: BYTES_READ=316416597
13/10/28 13:14:09 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1373)
at Mondrian8.main(Mondrian8.java:547)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)