我遇到了Hadoop 2.4的严重问题。我正在尝试运行我的用例,这是一个使用hadoop-streaming jar的groovy脚本。它会导致错误:Java heap space - No memory
。
这是我的配置:
纱-site.xml中
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8040</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>6144</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>2</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>4</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
mapred-site.xml中
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx2048m</value>
</property>
我在每个节点上都有8 GB RAM,并且没有减少任务。
我运行作业的命令行如下: -
$ HADOOP_INSTALL / bin / hadoop jar $ HADOOP_INSTALL / hadoop-streaming-2.4.0.jar -libjars $ {LIBJARS} -input /input.txt -mapper&#34; Launcher.groovy $ {args}&#34; -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat -output / $ {dir}
有人能告诉我哪里出错了吗?我对hadoop 1.2.1没有任何问题。