Hadoop NodeManager无法启动

时间：2020-08-07 08:57:45

标签： hadoop yarn nodemanager

我正在设置一个单节点集群。我正在关注https://dev.to/awwsmm/installing-and-running-hadoop-and-spark-on-ubuntu-18-393h教程。但是，我的nodemanager无法启动。

14468 SecondaryNameNode
14648 ResourceManager
14074 NameNode
15293 Jps
14223 DataNode

以下是节点管理器日志：


2020-08-07 14:13:52,796 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize mapreduce_shuffle 
java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:114)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:255)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:480)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:527)
2020-08-07 14:13:52,800 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed in state INITED; cause: java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:114)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:255)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:480)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:527)
2020-08-07 14:13:52,801 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause: java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:114)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:255)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:480)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:527)
2020-08-07 14:13:52,802 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
java.lang.IllegalArgumentException: The ServiceName: mapreduce_shuffle  set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:114)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:255)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 

2020-08-07 14:13:52,816 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at es/127.0.1.1
************************************************************/

这是我的yarn-site.xml：

<?xml version="1.0"?>


<configuration>

<!-- Site specific YARN configuration properties -->

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

我看到很多帖子，其中mapreduce.shuffle必须更改为mapreduce_shuffle。但是，在我的yarn-site.xml中，我已经有mapreduce_shuffle。

1 个答案:

答案 0 :(得分：0)

  1. 从日志mapreduce_shuffle 中可以看到，在mapreduce_shuffle值之后添加了两个空格。您能否正确键入并复制到yarn-site.xml中，而不是直接从博客中复制它。这可能是由于mapreduce_shuffle字符串中出现了一些特殊字符
  2. 将yarn.nodemanager.auxservices.mapreduce.shuffle.class替换为yarn.nodemanager.aux-services.mapreduce_shuffle.class标签，并启动应有的hadoop。
<configuration>
<!-- Site specific YARN configuration properties -->

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>
相关问题
最新问题