启动Hadoop Streaming作业时出错

时间:2016-07-24 18:41:43

标签: hadoop hadoop2 hadoop-streaming

我正在推出失败的Hadoop流媒体作业。启动它的行是:

hadoop jar $HADOOP_HOME/hadoop-streaming-2.6.0.2.2.5.3-1.jar -conf ~/HADOOP/conf/hadoop-cluster.xml -files aggregation_jobs -input /epcot -output /crowd_analytics/event_count/ -mapper "aggregation_jobs/streaming/event_count_map.py" -reducer "aggregation_jobs/streaming/event_count_reduce.py" -verbose >> output

hadoop-cluster.xml文件的内容如下:

<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://fsdala12080.test.domain.com/</value> </property> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>fsdala12081.test.domain.com:8032</value> </property> </configuration>

我收到以下错误:

java.net.UnknownHostException: ww-am04035180-010082030080.test.domain.com: ww-am04035180-010082030080.test.domain.com: unknown error at java.net.InetAddress.getLocalHost(InetAddress.java:1484) at org.apache.hadoop.streaming.Environment.getHost(Environment.java:121) at org.apache.hadoop.streaming.StreamUtil.<clinit>(StreamUtil.java:176) at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:822) at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:128) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.net.UnknownHostException: ww-am04035180-010082030080.test.domain.com: unknown error at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302) at java.net.InetAddress.getLocalHost(InetAddress.java:1479) ... 13 more

其中未知主机字符串的第一部分是我的本地计算机,它不在我正在谈论的集群中。顺便说一句,我可以从我的计算机与群集通信,并将文件从本地硬盘驱动器复制到该群集上的hdfs。为什么这可能不起作用的任何想法?为什么它会使我的本地主机达到峰值?我提交工作时有任何错误吗?任何指针都会有所帮助。

0 个答案:

没有答案