Question

安装hadoop 2.2并尝试启动管道示例后，我得到了以下错误（尝试启动hadoop jar hadoop-mapreduce-examples-2.2.0.jar wordcount someFile.txt /out后显示相同的错误）：

/usr/local/hadoop$ hadoop pipes -Dhadoop.pipes.java.recordreader=true -Dhadoop.pipes.java.recordwriter=true -input someFile.txt -output /out -program bin/wordcount
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.

13/12/14 20:12:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
13/12/14 20:12:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
13/12/14 20:12:07 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:08 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:09 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:10 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:11 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:12 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:13 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/12/14 20:12:14 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

我的yarn-site.xml：

<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<!-- Site specific YARN configuration properties -->
</configuration>

核心-site.xml中：

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

mapred-site.xml中：

<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
</configuration>

HDFS-site.xml中：

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/home/hduser/mydata/hdfs/namenode</value>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/home/hduser/mydata/hdfs/datanode</value>
</property>
</configuration>

我已经发现我的IPv6已经被禁用了。也许我的/ etc / hosts不正确？

/ etc / hosts中：

fe00::0         ip6-localnet
ff00::0         ip6-mcastprefix
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

127.0.0.1 localhost.localdomain localhost hduser
# Auto-generated hostname. Please do not remove this comment.
79.98.30.76 356114.s.dedikuoti.lt  356114
::1             localhost ip6-localhost ip6-loopback

Answer 1

连接资源管理器的问题是因为我需要向yarn-site.xml添加一些属性：

<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>

然而，我的乔布斯没有运行，但现在连接成功了

Answer 2

确保您已开始使用纱线。使用此命令启动它：

start-yarn.sh

然后使用此命令验证资源管理器是否正在运行：

JPS

输出应该如下所示：

17542 NameNode

17920 SecondaryNameNode

22064 Jps

17703 DataNode

18226 ResourceManager

18363 NodeManager

Answer 3

正确的方法可能是在 yarn-site.xml 中添加以下行：

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>127.0.0.1</value>
</property>

因为值字段主机表示可以设置的单个主机名，而不是设置所有 yarn.resourcemanager * 地址资源。导致ResourceManager组件的默认端口。

Apache Hadoop 2.7.1 - Configurations for ResourceManager

参数：yarn.resourcemanager.hostname
值：ResourceManager主机。
注意： host 可以设置单个主机名来代替设置所有yarn.resourcemanager *地址资源。导致ResourceManager组件的默认端口。

Answer 4

我通过将127.0.0.1:*

中的值更改为yarn-site.xml来解决了同样的问题

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
    <name>yarn.resourcemanager.address</name>
    <value>127.0.0.1:8032</value>
     </property>
      <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>127.0.0.1:8030</value>
     </property>
    <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>127.0.0.1:8031</value>
     </property>

Answer 5

Configuration conf = HBaseConfiguration.create();
 conf.set("yarn.resourcemanager.address", "127.0.0.1:8032");

在conf中你可以设置yarn.resourcemanager.address

Answer 6

就我而言，我的xml配置文件中有一个错字。您可以在$ HADOOP_HOME / logs / yarn-rdmaHB-resourcemanager-（yourhostname）.log中查看日志，其中可能会有有用的堆栈跟踪。

Answer 7

发生此错误是因为资源管理器无法启动。如果您做了其他人所说的更改配置文件并仍然出现错误的操作，请参考该错误。

注意：-Windows 10，Hadoop 3.1.3验证

因此，如果您是Windows用户，请转到hadoop-3.1.3 / sbin /。执行stop-all.cmd 然后启动all.cmd

现在将打开许多终端，它们是节点管理器，数据节点，名称节点和资源管理器。在资源管理器的终端中查看错误消息。该错误应该是您的问题。

如果错误消息是这样的：-
java.lang.NoClassDefFoundError：org / apache / hadoop / yarn / server / timelineservice / collector / TimelineCollectorManager

复制以下文件
来自〜/ hadoop-3.1.3 / share / hadoop / yarn / timelineservice
文件： hadoop-yarn-server-timelineservice-3.1.3.jar
到〜/ hadoop-3.1.3 / share / hadoop / yarn

这应该可以解决您的问题。

Answer 8

此问题可能是由于缺少HADOOP_CONF_DIR，MapReduce应用程序需要将其连接到yarn-site.xml中提到的资源管理器。因此，在运行MapReduce作业之前，尝试使用适当的Hadoop Conf目录手动设置/导出HADOOP_CONF_DIR，例如export HADOOP_CONF_DIR = / etc / hadoop / conf。这种方式对我有用：）

Answer 9

在/etc/hosts中使用以下设置，添加主机名代替your_host_name：

127.0.0.1   localhost
127.0.0.1   your_host_name

Hadoop：连接到ResourceManager失败

9 个答案: