使用Hadoop 2x进行hadoop-streaming的堆空间问题

时间:2014-07-09 12:22:05

标签: hadoop heap-memory hadoop-streaming

我遇到了Hadoop 2.4的严重问题。我正在尝试运行我的用例,这是一个使用hadoop-streaming jar的groovy脚本。它会导致错误:Java heap space - No memory

这是我的配置:

纱-site.xml中

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:8025</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>master:8040</value>
</property>
<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>128</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>6144</value>
</property>
<property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>2</value>
</property>
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>4096</value>
</property>
<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>4</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
</property>

mapred-site.xml中

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>4096</value>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>4096</value>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx2048m</value>
</property>

我在每个节点上都有8 GB RAM,并且没有减少任务。

我运行作业的命令行如下: -

  

$ HADOOP_INSTALL / bin / hadoop jar   $ HADOOP_INSTALL / hadoop-streaming-2.4.0.jar -libjars $ {LIBJARS} -input   /input.txt -mapper&#34; Launcher.groovy $ {args}&#34; -inputformat   org.apache.hadoop.mapred.lib.NLineInputFormat -output / $ {dir}

有人能告诉我哪里出错了吗?我对hadoop 1.2.1没有任何问题。

0 个答案:

没有答案