Question

更新：我解决了这个问题，我发现警告说像can't connect from cp26/127.0.1.1 to cp26:some_port，我在/ etc / hosts中删除了127.0.1.1。

我在三台机器上安装hadoop-2.7.3，所有ubuntu14.04，一个namenode和两个datanode，每台机器有48GB RAM。我跑完后

hadoop jar hadoop-install/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount input output

我在shell中遇到这些错误

INFO mapreduce.Job: Task Id : attempt_1475857350322_0003_m_000062_2, Status : FAILED

我在日志中收到这些警告和错误

WARN [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://master:54310/user/hadoop/output/_temporary/1/_temporary/attempt_1475857350322_0002_m_000020_0
ERROR [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not deallocate container for task attemptId attempt_1475857350322_0002_r_000000_0
ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1475857350322_0002_01_000170

我的mapred-site.xml

<configuration>

  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>2048</value>
  </property>

  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>4096</value>
  </property>

  <property>
    <name>mapred.task.timeout</name>
    <value>1800000</value> <!-- 30 minutes -->
  </property>

  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx1639m</value>
  </property>

  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx3277m</value>
  </property>

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobtracker.address</name>
    <value>master:54311</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>master:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master:19888</value>
  </property>
</configuration>

my yarn-site.xml

<configuration>

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>43008</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>43008</value>
  </property>

  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>4096</value>
  </property>

  <property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx3277m</value>
  </property>

  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
  </property>

  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>master:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:8033</value>
  </property>
</configuration>

Answer 1

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 5 10

nano inputdata # write anything in it for processing.

hadoop fs -put inputdata .

hadoop fs -ls -R

df -h inputdata

hadoop fs -df -h inputdata

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount inputdata outputresults

hadoop fs -ls

hadoop fs -mkdir yourname

hadoop fs -ls

运行hadoop wordcount示例失败

1 个答案: