HiBench wordcount工作挂在hadoop 2.9上

时间：2018-02-21 04:44:27

标签： hadoop mapreduce hdfs yarn

我正在使用：

  • HiBench 7.0
  • Hadoop 2.9
  • Java版本1.8.0_161
  • Scala代码运行器版本2.11.6
  • Apache Maven 3.5.2

所有在OpenStack VM的三节点Hadoop集群上都有详细信息：

  • Ubuntu 16.04.3 LTS
  • VCPUs：8
  • RAM：16GB
  • 大小：10GB

每个都有一个100GB的卷附加dfs存储

当试图运行wordcount准备脚本时，我被困在：

hadoop@hadoop0:~/HiBench$ /home/hadoop/HiBench/bin/workloads/micro/wordcount/prepare/prepare.sh
patching args=
Parsing conf: /home/hadoop/HiBench/conf/hadoop.conf
Parsing conf: /home/hadoop/HiBench/conf/hibench.conf
Parsing conf: /home/hadoop/HiBench/conf/workloads/micro/wordcount.conf
probe sleep jar: /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0-tests.jar
start HadoopPrepareWordcount bench
hdfs rm -r: /home/hadoop/hadoop/bin/hadoop --config /home/hadoop/hadoop/etc/hadoop fs -rm -r -skipTrash hdfs://node-master:9000/HiBench/Wordcount/Input
rm: 'hdfs://node-master:9000/HiBench/Wordcount/Input': No such file or directory
Submit MapReduce Job: /home/hadoop/hadoop/bin/hadoop --config /home/hadoop/hadoop/etc/hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar randomtextwriter -D mapreduce.randomtextwriter.totalbytes=32000 -D mapreduce.randomtextwriter.bytespermap=4000 -D mapreduce.job.maps=8 -D mapreduce.job.reduces=8 hdfs://node-master:9000/HiBench/Wordcount/Input
18/02/21 04:03:22 INFO mapreduce.Job: Running job: job_1519185680180_0001

这是一个众所周知的错误，但解决方案是针对我没有的问题：

  1. 错误的YARN和MapReduce配置导致容器分配资源不足，但我的配置文件应该正确设置以防止这种情况发生。
  2. 磁盘空间不足，但如图所示，我的hdfs超过200GB，足以支持简单的wordcount工作。

    3. 这是一个HiBench问题吗？我不满意HiBench wordcount的最低资源要求吗？

    以下是作业挂起时的一些报告的输出：

    hadoop@hadoop0:~$ yarn node -list    
18/02/21 04:03:33 INFO client.RMProxy: Connecting to ResourceManager at node-master/10.10.10.7:8032
Total Nodes:3
         Node-Id         Node-State Node-Http-Address   Number-of-Running-Containers
31-24-168.neu.massopencloud.org:34500           RUNNING 31-24-168.neu.massopencloud.org:8042                               0
31-24-168.neu.massopencloud.org:39297           RUNNING 31-24-168.neu.massopencloud.org:8042                               0
31-24-168.neu.massopencloud.org:43353           RUNNING 31-24-168.neu.massopencloud.org:8042                               1

hadoop@hadoop0:~$ yarn application -list    
18/02/21 04:04:05 INFO client.RMProxy: Connecting to ResourceManager at node-master/10.10.10.7:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):1
                Application-Id      Application-Name        Application-Type          User       Queue               State         Final-State         Progress                        Tracking-URL
application_1519185680180_0001    random-text-writer               MAPREDUCE        hadoop     default            ACCEPTED           UNDEFINED               0%                                 N/A


hadoop@hadoop0:~$ hdfs dfsadmin -report
Configured Capacity: 316665593856 (294.92 GB)
Present Capacity: 300320362240 (279.70 GB)
DFS Remaining: 300319195136 (279.69 GB)
DFS Used: 1167104 (1.11 MB)
DFS Used%: 0.00%
Under replicated blocks: 2
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0

Live datanodes (3):

Name: 10.10.10.10:50010 (node1)
Hostname: 31-24-168.neu.massopencloud.org
Decommission Status : Normal
Configured Capacity: 105555197952 (98.31 GB)
DFS Used: 331302 (323.54 KB)
Non DFS Used: 62558682 (59.66 MB)
DFS Remaining: 100106821632 (93.23 GB)
DFS Used%: 0.00%
DFS Remaining%: 94.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 21 04:04:30 UTC 2018
Last Block Report: Wed Feb 21 03:59:45 UTC 2018


Name: 10.10.10.11:50010 (node2)
Hostname: 31-24-168.neu.massopencloud.org
Decommission Status : Normal
Configured Capacity: 105555197952 (98.31 GB)
DFS Used: 331302 (323.54 KB)
Non DFS Used: 62554586 (59.66 MB)
DFS Remaining: 100106825728 (93.23 GB)
DFS Used%: 0.00%
DFS Remaining%: 94.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 21 04:04:30 UTC 2018
Last Block Report: Wed Feb 21 03:59:45 UTC 2018


Name: 10.10.10.7:50010 (localhost)
Hostname: 31-24-168.neu.massopencloud.org
Decommission Status : Normal
Configured Capacity: 105555197952 (98.31 GB)
DFS Used: 504500 (492.68 KB)
Non DFS Used: 63659340 (60.71 MB)
DFS Remaining: 100105547776 (93.23 GB)
DFS Used%: 0.00%
DFS Remaining%: 94.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 21 04:04:30 UTC 2018
Last Block Report: Wed Feb 21 03:59:45 UTC 2018

    我的yarn-site.xml如下：

    ＆＃13;
    ＆＃13; 
    <configuration>
    <property>
            <name>yarn.acl.enable</name>
            <value>0</value>
    </property>
    <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>node-master</value>
    </property>
    <property>
            <name>yarn.nodemanager.resource.memory-mb</name>
            <value>14392</value>
    </property>
    <property>
            <name>yarn.nodemanager.resource.cpu-vcores</name>
            <value>8</value>
    </property>
    <property>
            <name>yarn.scheduler.minimum-allocation-mb</name>
            <value>1024</value>
    </property>
    <property>
            <name>yarn.scheduler.minimum-allocation-vcores</name>
            <value>1</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>2</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
</configuration>
    ＆＃13;
    ＆＃13;
    ＆＃13;

    我的mapred-site.xml是：

    ＆＃13;
    ＆＃13; 
    <configuration>
    <property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.command-opts</name>
        <value>-Xmx768m</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        <description>Execution framework.</description>
    </property>
    <property>
        <name>mapreduce.map.cpu.vcores</name>
        <value>1</value>
        <description>The number of virtual cores required for each map task.</description>
    </property>
    <property>
        <name>mapreduce.reduce.cpu.vcores</name>
        <value>1</value>
        <description>The number of virtual cores required for each reduce task.</description>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>1024</value>
        <description>Larger resource limit for maps.</description>
    </property>
    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx768m</value>
        <description>Heap-size for child jvms of maps.</description>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>1024</value>
        <description>Larger resource limit for reduces.</description>
    </property>
    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>-Xmx768m</value>
        <description>Heap-size for child jvms of reduces.</description>
    </property>
</configuration>
    ＆＃13;
    ＆＃13;
    ＆＃13;

    非常感谢你的帮助。

0 个答案:

没有答案
相关问题
最新问题