Question

我试图在集群hadoop 2.7.1（一个主服务器和4个从服务器）上运行一个wordcount jar，但是MapReduce作业被阻止了：

$ hadoop jar wc.jar WordCount /input /output_hocine 
17/03/13 09:41:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
17/03/13 09:41:43 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 
17/03/13 09:41:43 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 
17/03/13 09:41:44 INFO input.FileInputFormat: Total input paths to process : 3 
17/03/13 09:41:44 INFO mapreduce.JobSubmitter: number of splits:3 
17/03/13 09:41:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489393376058_0003 
17/03/13 09:41:44 INFO impl.YarnClientImpl: Submitted application application_1489393376058_0003 
17/03/13 09:41:44 INFO mapreduce.Job: The url to track the job: http://ibnbadis21:8088/proxy/application_1489393376058_0003/ 
17/03/13 09:41:44 INFO mapreduce.Job: Running job: job_1489393376058_0003

通过导航器，通过导航器输出如下图所示：

以下是配置文件的内容：

核心-site.xml中：

<configuration>
<!--    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://ibnbadis21:9000</value>
    </property>-->
    <property>
        <name>fs.default.name</name>
        <value>hdfs://ibnbadis21:9000</value>
    </property>
   <property>
      <name>dfs.permissions</name>
      <value>false</value>
   </property>
 </configuration>

纱-site.xml中：

<?xml version="1.0"?> <configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

mapred-site.xml中：

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. -->
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>ibnbadis21:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>ibnbadis21:19888</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.staging-dir</name>
                <value>/user/app</value>
        </property>
</configuration>

HDFS-site.xml中：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
        <name>dfs.replication</name>
        <value>2</value>
 </property>
 <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
 </property>
 <property> <name>dfs.namenode.checkpoint.dir</name>
        <value>file:/usr/local/hadoop_data/hdfs/namesecondary</value>
 </property>
 <property> <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop_data/hdfs/datanode</value>
 </property>
</configuration>

有人能告诉我怎么能解决这个问题吗？

Answer 1

在/0.0.0.0:8032
连接到ResourceManager

0.0.0.0（默认值）不是有效的主机名。

所以，请在yarn-site.xml

中添加

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value> YOUR VALUE HERE </value> <!-- Needs Fully Qualified Domain Name -->
</property>

您可能没有设置很多值。

参考Hadoop | Configuring the Hadoop Daemons

顺便说一句，fs.defaultFS是正确使用的属性。

Answer 2

最后问题是关于访问权限。该框架无权访问我的yarn-site.xml文件。这就是它使用默认值0.0.0.0/8030的原因。因此当我用特权（sudo）执行命令时：

sudo hadoop jar wc.jar WordCount /input /output

我的工作MapReduce已成功执行！

运行Mapreduce问题

2 个答案: