Hadoop - 资源是<memory：0，vcores：0 =“”>

Question

我正在尝试在AWS上设置的群集上运行wordcount示例。它挂了，只是说跑步工作。

我在resourcemanager日志中发现了这个错误

我可以通过HDFS UI查看我的所有节点（namenode：50070）。

但是当我尝试通过namenode查看有关集群的更多信息时：8088 / cluster / nodes它说有0个节点？

有什么想法吗？我已经尝试编辑yarn-site.xml来指定最小/最大内存和内核，但它不起作用。

** edit以下是NodeManager日志文件中的错误

2018-02-08 19:28:41,110 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8042 2018-02-08 19:28:41,111 INFO org.mortbay.log: jetty-6.1.26 2018-02-08 19:28:41,246 INFO org.mortbay.log: Extract jar:file:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar!/webapps/node to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp 2018-02-08 19:28:42,777 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042 2018-02-08 19:28:42,777 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app node started at 8042 2018-02-08 19:28:42,783 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID assigned is : ec2-34-227-117-73.compute-1.amazonaws.com:39885 2018-02-08 19:28:42,797 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8031 2018-02-08 19:28:42,798 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor 2018-02-08 19:28:42,861 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: [] 2018-02-08 19:28:42,866 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[] 2018-02-08 19:28:43,935 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:44,936 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:45,937 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:46,937 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:47,938 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:48,939 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Answer 1

您在理解Hadoop时遇到了一个常见的错误。 Hadoop由文件系统（HDFS）和计算引擎（YARN）组成。 Datanodes仅显示HDFS功能。要运行作业，您需要资源管理器，并且还需要节点管理器来提供计算功能。

资源管理器的屏幕截图显示了这一点。您没有运行节点管理器，因此您没有可用于计算的vcores或内存。