在map-reduce作业中,Shuffle / Sorting失败

时间:2013-10-29 16:47:56

标签: hadoop parallel-processing mapreduce

在我的map-reduce作业中,当我将reducer的数量设置为0时,程序在不到5分钟的时间内成功完成。但是,当我将reducer的数量设置为某个而不是0(1-40之间)甚至没有指定reducer类时,它会在许多失败的任务中进行得非常缓慢,并突然失败并显示以下错误消息。由于我没有指定reducer类,唯一可能导致此错误的是对映射器的输出进行排序。映射器产生大约27,000个不同的密钥。这里有什么问题?日志文件中没有任何东西可以帮助。我唯一的猜想是大量的密钥导致了这个问题。我该如何解决这个问题?

  13/10/28 13:06:40 INFO mapred.JobClient:  map 70% reduce 18%
  13/10/28 13:07:13 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000052_2, Status : FAILED
  Task attempt_201310101346_0661_m_000052_2 failed to report status for 600 seconds. Killing!
  13/10/28 13:07:19 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000059_2, Status : FAILED
  Task attempt_201310101346_0661_m_000059_2 failed to report status for 600 seconds. Killing!
  13/10/28 13:07:23 INFO mapred.JobClient:  map 70% reduce 19%
  13/10/28 13:07:24 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000060_2, Status : FAILED
 Task attempt_201310101346_0661_m_000060_2 failed to report status for 600 seconds. Killing!
 13/10/28 13:07:26 INFO mapred.JobClient:  map 70% reduce 20%
 13/10/28 13:07:47 INFO mapred.JobClient:  map 71% reduce 20%
 13/10/28 13:07:54 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000061_2, Status : FAILED
 Task attempt_201310101346_0661_m_000061_2 failed to report status for 600 seconds. Killing!
 13/10/28 13:08:22 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000062_2, Status : FAILED
 Task attempt_201310101346_0661_m_000062_2 failed to report status for 600 seconds. Killing!
 13/10/28 13:09:03 INFO mapred.JobClient:  map 72% reduce 20%
 13/10/28 13:09:15 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000063_2, Status : FAILED
 Task attempt_201310101346_0661_m_000063_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:18 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000066_2, Status : FAILED
 Task attempt_201310101346_0661_m_000066_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:20 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000068_2,     Status : FAILED
Task attempt_201310101346_0661_m_000068_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:55 INFO mapred.JobClient:  map 72% reduce 21%
13/10/28 13:10:30 INFO mapred.JobClient:  map 72% reduce 22%
13/10/28 13:10:33 INFO mapred.JobClient:  map 73% reduce 22%
13/10/28 13:12:21 INFO mapred.JobClient:  map 74% reduce 22%
13/10/28 13:13:40 INFO mapred.JobClient:  map 74% reduce 23%
13/10/28 13:14:09 INFO mapred.JobClient: Job complete: job_201310101346_0661
13/10/28 13:14:09 INFO mapred.JobClient: Counters: 31
13/10/28 13:14:09 INFO mapred.JobClient:   File System Counters
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of bytes read=0
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of bytes written=542433323
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of read operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of bytes read=316452240
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of bytes written=0
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of read operations=100
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient:   Job Counters 
13/10/28 13:14:09 INFO mapred.JobClient:     Failed map tasks=1
13/10/28 13:14:09 INFO mapred.JobClient:     Launched map tasks=169
13/10/28 13:14:09 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 13:14:09 INFO mapred.JobClient:     Data-local map tasks=23
13/10/28 13:14:09 INFO mapred.JobClient:     Rack-local map tasks=146
13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=72419457
13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=1194624
13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
 13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
 13/10/28 13:14:09 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 13:14:09 INFO mapred.JobClient:     Map input records=6894035
13/10/28 13:14:09 INFO mapred.JobClient:     Map output records=6422195
 13/10/28 13:14:09 INFO mapred.JobClient:     Map output bytes=2300749768
13/10/28 13:14:09 INFO mapred.JobClient:     Input split bytes=6200
13/10/28 13:14:09 INFO mapred.JobClient:     Combine input records=6422195
13/10/28 13:14:09 INFO mapred.JobClient:     Combine output records=801608
13/10/28 13:14:09 INFO mapred.JobClient:     Spilled Records=801608
13/10/28 13:14:09 INFO mapred.JobClient:     CPU time spent (ms)=48077980
13/10/28 13:14:09 INFO mapred.JobClient:     Physical memory (bytes) snapshot=13246660608
13/10/28 13:14:09 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=20080971776
 13/10/28 13:14:09 INFO mapred.JobClient:     Total committed heap usage (bytes)=10181214208
 13/10/28 13:14:09 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
 13/10/28 13:14:09 INFO mapred.JobClient:     BYTES_READ=316416597
 13/10/28 13:14:09 INFO mapred.JobClient: Job Failed: NA
 Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1373)
    at Mondrian8.main(Mondrian8.java:547)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

0 个答案:

没有答案