运行1TB teragen时为什么没有减速机？

Question

我正在使用以下命令运行hadoop的terasort基准：

jar /Users/karan.verma/Documents/backups/h/hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar  teragen -Dmapreduce.job.maps=100 1t random-data

并为100个地图任务打印了以下日志：

18/03/27 13:06:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/27 13:06:04 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
18/03/27 13:06:05 INFO terasort.TeraSort: Generating -727379968 using 100
18/03/27 13:06:05 INFO mapreduce.JobSubmitter: number of splits:100
18/03/27 13:06:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1522131782827_0001
18/03/27 13:06:06 INFO impl.YarnClientImpl: Submitted application application_1522131782827_0001
18/03/27 13:06:06 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1522131782827_0001/
18/03/27 13:06:06 INFO mapreduce.Job: Running job: job_1522131782827_0001
18/03/27 13:06:16 INFO mapreduce.Job: Job job_1522131782827_0001 running in uber mode : false
18/03/27 13:06:16 INFO mapreduce.Job:  map 0% reduce 0%
18/03/27 13:06:29 INFO mapreduce.Job:  map 2% reduce 0%
18/03/27 13:06:31 INFO mapreduce.Job:  map 3% reduce 0%
18/03/27 13:06:32 INFO mapreduce.Job:  map 5% reduce 0%

....
18/03/27 13:09:27 INFO mapreduce.Job:  map 100% reduce 0%

这是打印在控制台上的最终计数器：

18/03/27 13:09:29 INFO mapreduce.Job: Counters: 30
File System Counters
    FILE: Number of bytes read=0
    FILE: Number of bytes written=10660990
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=8594
    HDFS: Number of bytes written=0
    HDFS: Number of read operations=400
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=200
Job Counters 
    Launched map tasks=100
    Other local map tasks=100
    Total time spent by all maps in occupied slots (ms)=983560
    Total time spent by all reduces in occupied slots (ms)=0
    Total time spent by all map tasks (ms)=983560
    Total vcore-milliseconds taken by all map tasks=983560
    Total megabyte-milliseconds taken by all map tasks=1007165440
Map-Reduce Framework
    Map input records=0
    Map output records=0
    Input split bytes=8594
    Spilled Records=0
    Failed Shuffles=0
    Merged Map outputs=0
    GC time elapsed (ms)=9746
    CPU time spent (ms)=0
    Physical memory (bytes) snapshot=0
    Virtual memory (bytes) snapshot=0
    Total committed heap usage (bytes)=11220811776
File Input Format Counters 
    Bytes Read=0
File Output Format Counters 
    Bytes Written=0

这是作业计划的输出：

请说明为什么没有减少任务？

Answer 1

您的运行命令表明您正在运行teragen而不是terasort。 teragen只会生成您可以用于terasort的数据，因此不需要缩减器。

要对刚刚生成的数据运行terasort，请运行：

hadoop jar /Users/karan.verma/Documents/backups/h/hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar terasort random-data terasort-output

然后你应该看到减速器。

Answer 2

执行teragen时没有运行减少任务。这是文档：

  

TeraGen将运行地图任务以生成数据，并且不会运行任何减少任务。地图任务的默认数量由＆＃34; mapreduce.job.maps = 2＆＃34;定义。 PARAM。这里唯一的目的是以下列格式生成1TB随机数据＆＃34; 10字节键| 2个字节中断| 32字节acsii / hex | 4个字节中断| 48字节填充| 4个字节中断| \ r \ n＆＃34;