我已经设置了一个Amazon EMR作业流，其中包含1个按需核心节点和4个带有出价的任务节点。当我仅在核心节点上运行任务时，每个步骤在1小时内完成。当我很幸运并拥有1个核心+4个任务节点时，步骤通常会在10分钟内完成。
我的问题是，当任务节点被亚马逊带走时，剩下的任务尝试开始痛苦，他们可以痛苦7-10个小时。
从下面的日志中可以看出，从08:56（0％） - 09:01（43％）一切正常，但随后任务尝试开始失败。现在根据我的计算基于以下事实：仅在1个核心节点上运行步骤将花费不到一个小时，我预计步骤将在不到1小时内完成43％-100％。然而，它至少还有5个多小时的痛苦：09：01-14：30。这对我来说并不正常（不要谈论浪费的时间和金钱）。我怎么能解决这个问题？是什么导致这种情况？
2014-05-21 08:55:39,317 INFO com.amazon.ws.emr.hadoop.fs.EmrFileSystem (main): Opening 's3://test/log-parser/code/hadoop-script.sh' for reading
2014-05-21 08:55:45,555 INFO com.amazon.ws.emr.hadoop.fs.EmrFileSystem (main): Opening 's3://test/log-parser/code/hadoop-upload-script.sh' for reading
2014-05-21 08:55:52,990 INFO com.amazon.ws.emr.hadoop.fs.EmrFileSystem (main): Opening 's3://test/log-parser/code/log-parser.jar' for reading
2014-05-21 08:55:59,840 INFO com.innovid.logParser.LogParserMapReduce (main): LogParserMapReduce: waitingForCompletion
2014-05-21 08:56:00,190 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at /1.1.1.1:9022
2014-05-21 08:56:05,397 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input paths to process : 31
2014-05-21 08:56:05,426 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2014-05-21 08:56:05,434 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev c7d54fffe5a853c437ee23413ba71fc6af23c91d]
2014-05-21 08:56:05,669 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:135
2014-05-21 08:56:05,697 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): user.name is deprecated. Instead, use mapreduce.job.user.name
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2014-05-21 08:56:05,698 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): keep.failed.task.files is deprecated. Instead, use mapreduce.task.files.preserve.failedtasks
2014-05-21 08:56:05,699 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.job.name is deprecated. Instead, use mapreduce.job.name
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-05-21 08:56:05,700 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
2014-05-21 08:56:05,701 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2014-05-21 08:56:05,701 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
2014-05-21 08:56:06,123 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1400094353_0186
2014-05-21 08:56:06,779 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1400094353_0186 to ResourceManager at /10.42.102.163:9022
2014-05-21 08:56:06,871 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://1.1.1.1:9046/proxy/application_1400094353_0186/
2014-05-21 08:56:06,872 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1400094353_0186
2014-05-21 08:56:15,441 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1400094353_0186 running in uber mode : false
2014-05-21 08:56:15,443 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2014-05-21 08:57:10,871 INFO org.apache.hadoop.mapreduce.Job (main): map 1% reduce 0%
2014-05-21 08:57:19,928 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2014-05-21 08:57:25,971 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2014-05-21 08:57:36,033 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2014-05-21 08:57:52,141 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2014-05-21 08:58:00,189 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2014-05-21 08:58:07,234 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2014-05-21 08:58:14,285 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2014-05-21 08:58:20,321 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2014-05-21 08:58:28,369 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2014-05-21 08:58:35,421 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2014-05-21 08:58:44,481 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2014-05-21 08:58:51,530 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2014-05-21 08:58:59,583 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2014-05-21 08:59:06,625 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2014-05-21 08:59:12,697 INFO org.apache.hadoop.mapreduce.Job (main): map 16% reduce 0%
2014-05-21 08:59:20,766 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2014-05-21 08:59:26,804 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2014-05-21 08:59:33,868 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2014-05-21 08:59:39,907 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2014-05-21 08:59:46,959 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2014-05-21 08:59:54,003 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2014-05-21 09:00:01,052 INFO org.apache.hadoop.mapreduce.Job (main): map 23% reduce 0%
2014-05-21 09:00:08,108 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2014-05-21 09:00:16,196 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2014-05-21 09:00:22,241 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2014-05-21 09:00:29,288 INFO org.apache.hadoop.mapreduce.Job (main): map 27% reduce 0%
2014-05-21 09:00:36,328 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2014-05-21 09:00:43,410 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2014-05-21 09:01:22,288 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2014-05-21 09:01:26,318 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 0%
2014-05-21 09:01:29,334 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2014-05-21 09:01:32,351 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 0%
2014-05-21 09:01:35,368 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 0%
2014-05-21 09:01:38,384 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 0%
2014-05-21 09:01:40,395 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 0%
2014-05-21 09:01:41,401 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 0%
2014-05-21 09:01:43,416 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 0%
2014-05-21 09:01:45,428 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 0%
2014-05-21 09:01:47,439 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2014-05-21 09:01:49,451 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 0%
2014-05-21 09:01:51,474 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 0%
2014-05-21 09:01:54,491 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 0%
2014-05-21 09:06:44,031 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 0%
2014-05-21 09:07:44,353 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 09:22:04,475 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000106_0, Status : FAILED
2014-05-21 09:22:05,499 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 09:32:10,244 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 09:37:25,611 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000079_0, Status : FAILED
2014-05-21 09:37:25,613 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000094_0, Status : FAILED
2014-05-21 09:37:25,614 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000107_0, Status : FAILED
2014-05-21 09:37:25,615 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000088_0, Status : FAILED
2014-05-21 09:37:26,620 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 09:48:50,521 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 09:52:45,491 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000081_0, Status : FAILED
2014-05-21 09:52:46,495 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:05:30,717 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:08:06,373 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000090_0, Status : FAILED
2014-05-21 10:08:07,377 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:18:50,064 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:23:28,146 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000110_0, Status : FAILED
2014-05-21 10:23:29,150 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:35:29,790 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:38:48,507 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000104_0, Status : FAILED
2014-05-21 10:38:49,511 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 10:52:10,333 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 10:54:09,754 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000108_0, Status : FAILED
2014-05-21 10:54:10,758 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:05:30,115 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:09:29,958 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000095_0, Status : FAILED
2014-05-21 11:09:30,961 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:22:10,617 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:24:51,159 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000084_0, Status : FAILED
2014-05-21 11:24:51,160 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000116_0, Status : FAILED
2014-05-21 11:24:52,163 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:35:29,377 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:40:12,354 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000078_0, Status : FAILED
2014-05-21 11:40:13,358 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 11:52:09,812 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 11:55:32,516 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000113_0, Status : FAILED
2014-05-21 11:55:33,520 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:08:50,313 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:10:53,733 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000093_0, Status : FAILED
2014-05-21 12:10:54,739 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:22:10,023 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:26:14,825 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000097_0, Status : FAILED
2014-05-21 12:26:15,828 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:38:50,340 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:41:35,872 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000114_0, Status : FAILED
2014-05-21 12:41:36,876 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:41:45,906 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000049_0, Status : FAILED
2014-05-21 12:41:55,938 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000053_0, Status : FAILED
2014-05-21 12:41:56,942 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 12:42:05,974 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000041_0, Status : FAILED
2014-05-21 12:42:06,977 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 12:42:16,009 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000062_0, Status : FAILED
2014-05-21 12:42:17,013 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 12:42:26,043 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000045_0, Status : FAILED
2014-05-21 12:42:36,077 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000050_0, Status : FAILED
2014-05-21 12:42:37,080 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 12:52:09,942 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 12:55:29,568 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 12:56:56,841 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000109_0, Status : FAILED
2014-05-21 12:56:56,842 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000099_0, Status : FAILED
2014-05-21 12:56:57,845 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 12:57:06,876 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000044_0, Status : FAILED
2014-05-21 12:57:07,879 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 12:57:16,909 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000072_0, Status : FAILED
2014-05-21 12:57:17,912 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 12:57:25,974 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000057_0, Status : FAILED
2014-05-21 12:57:26,977 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 12:57:36,006 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000056_0, Status : FAILED
2014-05-21 12:57:46,038 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000069_0, Status : FAILED
2014-05-21 12:57:47,041 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 12:57:56,069 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000076_0, Status : FAILED
2014-05-21 12:57:57,072 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 0%
2014-05-21 12:58:06,102 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000042_0, Status : FAILED
2014-05-21 12:58:07,105 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 0%
2014-05-21 12:58:16,134 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000071_0, Status : FAILED
2014-05-21 12:58:26,166 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000051_0, Status : FAILED
2014-05-21 12:58:27,171 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 0%
2014-05-21 12:58:36,207 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000039_0, Status : FAILED
2014-05-21 12:58:37,210 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 0%
2014-05-21 12:58:46,240 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000060_0, Status : FAILED
2014-05-21 12:58:47,243 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 0%
2014-05-21 12:58:56,271 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000061_0, Status : FAILED
2014-05-21 13:08:50,122 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 13:12:09,752 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:12:17,779 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000091_0, Status : FAILED
2014-05-21 13:12:17,780 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000083_0, Status : FAILED
2014-05-21 13:12:17,781 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000115_0, Status : FAILED
2014-05-21 13:12:18,784 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 13:12:27,813 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000070_0, Status : FAILED
2014-05-21 13:12:28,816 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 13:25:30,288 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:27:37,695 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000085_0, Status : FAILED
2014-05-21 13:27:38,698 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 13:27:47,729 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000058_0, Status : FAILED
2014-05-21 13:27:57,762 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000063_0, Status : FAILED
2014-05-21 13:27:58,765 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 13:38:50,829 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:42:58,603 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000086_0, Status : FAILED
2014-05-21 13:42:59,606 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 13:43:08,636 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000077_0, Status : FAILED
2014-05-21 13:55:30,006 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 13:58:19,532 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000098_0, Status : FAILED
2014-05-21 13:58:20,535 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 14:08:49,521 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 14:13:40,429 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000101_0, Status : FAILED
2014-05-21 14:13:41,433 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 14:13:50,462 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000059_0, Status : FAILED
2014-05-21 14:25:29,656 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 14:29:01,327 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000103_0, Status : FAILED
2014-05-21 14:29:02,330 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2014-05-21 14:29:11,361 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000067_0, Status : FAILED
2014-05-21 14:29:21,394 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000075_0, Status : FAILED
2014-05-21 14:29:22,397 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 0%
2014-05-21 14:29:31,427 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000046_0, Status : FAILED
2014-05-21 14:29:32,430 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 14:29:41,458 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000047_0, Status : FAILED
2014-05-21 14:29:42,461 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 14:29:51,491 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000054_0, Status : FAILED
2014-05-21 14:30:01,550 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000068_0, Status : FAILED
2014-05-21 14:30:02,554 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 14:30:11,591 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000048_0, Status : FAILED
2014-05-21 14:30:12,594 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 0%
2014-05-21 14:30:21,626 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000038_0, Status : FAILED
2014-05-21 14:30:22,632 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 0%
2014-05-21 14:30:31,660 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000052_0, Status : FAILED
2014-05-21 14:30:41,691 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000043_0, Status : FAILED
2014-05-21 14:30:42,694 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 0%
2014-05-21 14:30:51,724 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000065_0, Status : FAILED
2014-05-21 14:30:52,727 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 0%
2014-05-21 14:42:09,877 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2014-05-21 14:44:22,301 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000082_0, Status : FAILED
2014-05-21 14:44:22,302 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000080_0, Status : FAILED
2014-05-21 14:44:22,303 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000102_0, Status : FAILED
2014-05-21 14:44:22,304 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000096_0, Status : FAILED
2014-05-21 14:44:23,307 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2014-05-21 14:44:32,338 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000055_0, Status : FAILED
2014-05-21 14:44:33,341 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2014-05-21 14:44:42,371 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000073_0, Status : FAILED
2014-05-21 14:44:52,405 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000074_0, Status : FAILED
2014-05-21 14:44:53,408 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2014-05-21 14:45:02,441 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000064_0, Status : FAILED
2014-05-21 14:45:03,446 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 0%
2014-05-21 14:45:12,480 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000066_0, Status : FAILED
2014-05-21 14:45:13,486 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 0%
2014-05-21 14:45:22,520 INFO org.apache.hadoop.mapreduce.Job (main): Task Id : attempt_1400503094353_0186_m_000040_0, Status : FAILED