为什么减少率为16％？

时间：2012-03-21 11:41:57

标签： hadoop mapreduce cloudera

我有一个map reduce工作，我试图在一个相对较小的数据集上运行。我一直遇到一个问题，即减少工作一直停留在16％。我的任务跟踪器的日志读取：

2012-03-21 17:09:23,829 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:26,865 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:32,902 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:38,938 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:41,973 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:48,010 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:51,045 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:09:57,086 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:03,120 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:06,154 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:12,198 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:18,234 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:21,271 INFO org.apache.hadoop.mapred.TaskTracker:> attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:27,310 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:33,342 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:36,374 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:42,403 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:48,435 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:51,462 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:10:57,495 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:11:03,523 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:11:06,545 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:11:12,578 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)
2012-03-21 17:11:18,607 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201203211704_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.16 MB/s)

1 个答案:

答案 0 :(得分：3)

我打开了一个FileSystem对象，并没有在访问文件的映射器中关闭它。在map（）定义的末尾添加fs.close（）可以解决问题。

相关问题
最新问题