mapreduce hadoop中没有减速器

Question

我只有一个键从mapper发送到reducer并且我没有设置减速器为10.因此一个减速器将在该键上运行，剩下的9个减速器将执行什么？

Answer 1

The other 9 reducers will run through their lifecycle as normal, they just wont have any key/values to process once they run, so they will stop quickly. Thus you will waste resources while they needlessly run.

You'll generally (most output formats do this) also find you end up with a part file for a reduce that ran but didnt write anything. The part file won't contain any actual data just file metadata, for example gzip headers.

Answer 2

其他9个减速器将一直运行，直到它们的时段结束。

他们没有任何K / V对可以快速处理和停止。

您可以使用自定义分区程序将地图输出均匀地分配给所有Reducer; - 至少为第一级;并且最后阶段通过一个减速器结合 - 从而减少了大部分减速阶段的计算负荷。