由于hadoop中的合并操作,Mapper Task挂在那里

时间:2013-03-20 07:52:44

标签: hadoop

我的hadoop作业的mapper任务总是挂在那里,即使它的进度是100%,它的状态仍在运行。我在mapper任务日志中看到了很多合并操作。我已经将参数io.sort.mb和io.sort.factor更改为更大的数字。但它没有解决我的问题。我有大约2000个mapper任务,mapper任务的总输出大小是200G,这意味着每个mapper任务的输出大小大约是100M,这对我来说应该不是很大。所以我不明白为什么mapper任务会做很多合并操作

关于这个问题的任何想法?感谢

这是我的日志(BTW,大小数字的含义是什么?):

[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44335 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45076 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44057 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45369 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45388 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45816 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44765 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45761 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45379 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44456 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45581 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45939 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45418 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45241 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44809 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44736 bytes
[INFO] Merger - Merging 2 sorted segments

0 个答案:

没有答案