Question

我已使用相同的问题检查了多个帖子，解决方案始终将以下内容添加到yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarm.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

我在配置中重新启动了两个属性并重新启动了纱线。问题仍然存在。

错误是：

17/02/15 15:43:34 INFO mapreduce.Job: Task Id : attempt_1487202110321_0001_m_000000_2, Status : FAILED
Container launch failed for container_1487202110321_0001_01_000007 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

我希望有一个错字，但似乎无法找到它。尝试直接将stackoverflow上的内容复制到xml文件中，仍然无法正常工作。

我还能尝试什么？

编辑：

由于错误说aux_service应该是auxService，我修改了yarn-site.xml，将所有aux-service更改为auxService，但它仍然不能正常工作

EDIT2：

如果有人感兴趣，我会调用此命令

hadoop jar hadoop-streaming-2.7.1.jar \
       -input /user/myfolder/input1/* \
       -output /user/myfolder/output1 \
       -mapper <path>/<to>/<mapper>/mapper.py \
       -reducer <path>/<to>/<reducer>/reducer.py

虽然我已经在/usr/local/cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/

编辑3：

我是个笨蛋。证明 - 阅读脚本家伙！

Answer 1

将yarn-site.xml中的媒体资源名称更新为yarn.nodemanager.aux-services，

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

Hadoop MapReduce作业因为auxService：mapreduce_shuffle不存在而停滞不前

1 个答案: