Question

我在群集中运行了Map Reduce作业。我在HDInsight上创建了一个有2个名字节点和4个数据节点的作业。

我没有设置地图任务的数量和减少任务。执行mapreduce作业后，我得到如下结果。

我注意到启动的reduce任务的数量是1.Dose，这意味着我的工作只在一个节点中执行？如何查看此作业中使用了多少个节点？

File System Counters
                FILE: Number of bytes read=2209390166
                FILE: Number of bytes written=3314494070
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=1084887535
                WASB: Number of bytes written=1205106549
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Launched map tasks=2
                Launched reduce tasks=1
                Rack-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=148221
                Total time spent by all reduces in occupied slots (ms)=302038
                Total time spent by all map tasks (ms)=148221
                Total time spent by all reduce tasks (ms)=151019
                Total vcore-seconds taken by all map tasks=148221
                Total vcore-seconds taken by all reduce tasks=151019
                Total megabyte-seconds taken by all map tasks=113833728
                Total megabyte-seconds taken by all reduce tasks=231965184
        Map-Reduce Framework
                Map input records=3820642
                Map output records=3820642
                Map output bytes=1092815223
                Map output materialized bytes=1104695065
                Input split bytes=286
                Combine input records=0
                Combine output records=0
                Reduce input groups=200
                Reduce shuffle bytes=1104695065
                Reduce input records=3820642
                Reduce output records=3820642
                Spilled Records=11461926
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=16229
                CPU time spent (ms)=225140
                Physical memory (bytes) snapshot=1377296384
                Virtual memory (bytes) snapshot=5068787712
                Total committed heap usage (bytes)=1175453696
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1084887106
        File Output Format Counters
                Bytes Written=68771556

Answer 1

是的你设置减少数是1，输出减少是1你可以看到它的地图输入记录= 3820642 映射输出记录= 3820642 映射输出字节= 1092815223 映射输出物化字节= 1104695065 输入拆分字节= 286 合并输入记录= 0 合并输出记录= 0 减少输入组= 200 减少随机字节= 1104695065 减少输入记录= 3820642 减少输出记录= 3820642

如何查看我的mapreduce作业是否在多个节点上执行？

1 个答案: