Question

我遇到与以下链接相同的问题。

请帮我解决一下这个问题，我在Apache Hadoop 2.7.3 EC2安装上遇到同样的问题。链接中提到的属性是否需要添加到名称和数据节点配置文件中？如果是，所有.xmls？提前谢谢。

此外，应用程序正在尝试访问EC2上的内部IP，如下所示，我是否需要打开任何端口？它在Web UI上显示为8042。

所有节点和Nodemanager和资源管理器（RM）都显示为以jps运行。

当我尝试运行地图时，来自Namenode的错误缩小示例如下：

作业job_1506038808044_0002因状态失败而失败，原因是：由于错误启动appattempt_1506038808044_0002_000002，应用程序application_1506038808044_0002失败了2次。得到异常：org.apache.hadoop.net.ConnectTimeoutException：调用从ip-172-31-1-10 / 172.31.1.10到ip-172-31-5-59.ec2.internal：43555在套接字超时异常时失败： org.apache.hadoop.net.ConnectTimeoutException：等待通道准备连接时超时20000毫秒。 ch：java.nio.channels.SocketChannel [connection-pending remote = ip-172-31-5-59.ec2.internal / 172.31.5.59：43555]

最后，RM Web UI在作业运行的所有时间都有以下消息：

状态：等待AM容器分配，启动并注册RM。

谢谢，阿沙

Answer 1

在尝试Hadoop上出现的解决方案，Socket Timeout Error（我的问题中的链接）并在下面添加到hdfs-site.xml文件后，通过允许所有ICMP和UDP规则到ec2实例来解决问题，以便他们可以ping彼此之间。

<property>
  <name>dfs.namenode.name.dir</name>
  <value>/usr/local/hadoop/hadoop_work/hdfs/namenode</value>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>/usr/local/hadoop/hadoop_work/hdfs/datanode</value>
</property>
<property>
  <name>dfs.namenode.checkpoint.dir</name>
  <value>/usr/local/hadoop/hadoop_work/hdfs/namesecondary</value>
</property>
<property>
  <name>dfs.block.size</name>
  <value>134217728</value>
</property>
<property>
  <name>dfs.client.use.datanode.hostname</name>
  <value>true</value>
</property>
<property>
  <name>dfs.datanode.socket.write.timeout</name>
  <value>2000000</value>
</property>
<property>
  <name>dfs.socket.timeout</name>
  <value>2000000</value>
</property>

<property>
  <name>dfs.datanode.use.datanode.hostname</name>
  <value>true</value>
  <description>Whether datanodes should use datanode hostnames when
    connecting to other datanodes for data transfer.
  </description>
</property>

<property>
  <name>dfs.namenode.rpc-bind-host</name>
  <value>0.0.0.0</value>
  <description>
    The actual address the RPC server will bind to. If this optional address is
    set, it overrides only the hostname portion of dfs.namenode.rpc-address.
    It can also be specified per name node or name service for HA/Federation.
    This is useful for making the name node listen on all interfaces by
    setting it to 0.0.0.0.
  </description>
</property>

<property>
  <name>dfs.namenode.servicerpc-bind-host</name>
  <value>0.0.0.0</value>
  <description>
    The actual address the service RPC server will bind to. If this optional address is
    set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.
    It can also be specified per name node or name service for HA/Federation.
    This is useful for making the name node listen on all interfaces by
    setting it to 0.0.0.0.
  </description>
</property>

<property>
  <name>dfs.namenode.http-bind-host</name>
  <value>0.0.0.0</value>
  <description>
    The actual address the HTTP server will bind to. If this optional address
    is set, it overrides only the hostname portion of dfs.namenode.http-address.
    It can also be specified per name node or name service for HA/Federation.
    This is useful for making the name node HTTP server listen on all
    interfaces by setting it to 0.0.0.0.
  </description>
</property>

<property>
  <name>dfs.namenode.https-bind-host</name>
  <value>0.0.0.0</value>
  <description>
    The actual address the HTTPS server will bind to. If this optional address
    is set, it overrides only the hostname portion of dfs.namenode.https-address.
    It can also be specified per name node or name service for HA/Federation.
    This is useful for making the name node HTTPS server listen on all
    interfaces by setting it to 0.0.0.0.
  </description>
</property>

Apache Hadoop 2.7.3，套接字超时错误

1 个答案: