Hadoop连接被拒绝与本地主机上的ResourceManager：8032

时间：2019-03-01 08:17:20

标签： java macos hadoop hdfs

我是Hadoop的新手，并且自最近几个小时以来一直在尝试使其在我的机器上运行。我主要遵循这些指南中的安装和设置说明-1，2，3，4，5，6。

我也多次在Hadoop: Connecting to ResourceManager failed上阅读过这样的问题，但没有任何帮助。

这是我的配置文件：

core-site.xml -

<configuration>
<property>
   <name>hadoop.tmp.dir</name>
   <!-- <value>/tmp/hadoop-${user.name}</value> -->
   <value>/app/hadoop/tmp</value>
   <description>A base for other temporary directories.</description>
 </property> 
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
 </property>
</configuration>

hdfs-site.xml -

<configuration>

 <property>
     <name>dfs.replication</name>
     <value>1</value>
 </property>
 <property>
  <name>dfs.namenode.name.dir</name>
  <value>${user.home}/hadoop/data/namenode</value>
</property>

<property>
  <name>dfs.datanode.data.dir</name>
  <value>${user.home}/hadoop/data/datanode</value>
</property>
</configuration>

mapred-site.xml -

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

yarn-site.xml -

<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>localhost:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>localhost:8031</value>
    </property>
</configuration>

安装指南还要求在hadoop-env.sh中进行更改，但是对于OSX，此case中已经存在必需的更改，因此我没有做任何进一步的更改-

case ${HADOOP_OS_TYPE} in
  Darwin*)
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.kdc= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf= "
  ;;
esac

在我的.zshrc文件中，我像这样添加了hstart和hstop别名-

# hadoop config
alias hstart="/usr/local/Cellar/hadoop/3.1.1/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/3.1.1/sbin/start-yarn.sh"
alias hstop="/usr/local/Cellar/hadoop/3.1.1/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/3.1.1/sbin/stop-dfs.sh"

我认为我的hstart命令正常工作-

➜  ~ hstart
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [Manish-Macbook-Pro-15.local]
2019-03-01 02:26:13,502 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers

这是jps-

的输出 
➜  ~ jps
30548 NameNode
31172 Jps
30796 SecondaryNameNode
30655 DataNode

此后，我尝试使用其中一份指南中的示例作业来运行示例jar文件（hadoop-mapreduce-examples-3.1.1.jar）-

➜  ~ cd /usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/mapreduce
➜  mapreduce hadoop jar hadoop-mapreduce-examples-3.1.1.jar pi 2 5

这是ConnectionRefused错误出现的时间-

Number of Maps  = 2
Samples per Map = 5
2019-03-01 02:26:39,491 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
2019-03-01 02:26:42,134 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
2019-03-01 02:26:43,440 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:44,443 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:45,449 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:46,452 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:47,456 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:48,461 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:49,463 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:50,469 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:51,475 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:52,481 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:53,496 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:54,501 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:55,507 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:56,510 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:57,513 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:58,517 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:59,520 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:00,526 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:01,528 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:02,530 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:02,533 INFO retry.RetryInvocationHandler: java.net.ConnectException: Call From Manish-Macbook-Pro-15.local/192.168.200.79 to localhost:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ApplicationClientProtocolPBClientImpl.getNewApplication over null after 1 failover attempts. Trying to failover after sleeping for 31316ms.

此后，它会继续尝试反复连接。我必须打开一个新的终端选项卡才能执行hstop。

我不知道是什么原因导致此错误。我从头开始使用hdfs namenode -format了好几次，但是每次都出现相同的错误。

我还有其他一些发现-

  • 指南提到，一旦hadoop启动并运行，我就可以使用相应的默认端口在浏览器中查看事物的状态（例如ResourceManager），但是我已经尝试了多个端口-localhost:50070 ，localhost:8088等更多内容，但没有任何效果。

  • 在yarn-site.xml文件中，我使用localhost:8032作为<name>yarn.resourcemanager.address</name>的值，但是在我的终端日志中，它显示了2019-03-01 02:26:48,461 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032.。我不知道为什么是localhost/127.0.0.1:8032而不是localhost:8032。

以下是/etc/hosts-

的内容 
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost

希望对此有所帮助。

0 个答案:

没有答案
相关问题
最新问题