Mapper作业在Hadoop-python流中失败

时间:2016-02-11 04:39:13

标签: python hadoop streaming

我正在尝试在Hadoop流上执行Python脚本,但在执行命令映射器后,作业失败了。请帮忙。

我的命令:

hduser@R9HEX3R:~/hadoop-2.5.0-cdh5.3.2$ hadoop jar /home/hduser/hadoop-2.5.0-cdh5.3.2/share/hadoop/mapreduce1/contrib/streaming/hadoop-streaming-2.5.0-mr1-cdh5.3.2.jar -file /home/hduser/mapper.py /home/hduser/reducer.py -mapper  /home/hduser/mapper.py -reducer  /home/hduser/reducer.py -input /user/hduser/data -output /user/hduser/data_output

错误:

  16/02/11 09:52:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  16/02/11 09:52:57 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
  packageJobJar: [/home/hduser/mapper.py, /home/hduser/reducer.py] [] /tmp/streamjob2731352439731142145.jar tmpDir=null
 16/02/11 09:52:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
 16/02/11 09:52:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
 16/02/11 09:52:59 INFO mapred.FileInputFormat: Total input paths to process : 1
 16/02/11 09:52:59 INFO mapreduce.JobSubmitter: number of splits:2
 16/02/11 09:53:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1455116478271_0005
 16/02/11 09:53:00 INFO impl.YarnClientImpl: Submitted application application_1455116478271_0005
 16/02/11 09:53:00 INFO mapreduce.Job: The url to track the job: http://R9HEX3R:8088/proxy/application_1455116478271_0005/
 16/02/11 09:53:00 INFO streaming.StreamJob: getLocalDirs(): [/home/hduser/hadata/mapred/local]
 16/02/11 09:53:00 INFO streaming.StreamJob: Running job: job_1455116478271_0005
 16/02/11 09:53:00 INFO streaming.StreamJob: Job running in-process (local Hadoop)
16/02/11 09:53:02 INFO streaming.StreamJob:  map 0%  reduce 0%
16/02/11 09:53:17 INFO streaming.StreamJob:  map 50%  reduce 0%
16/02/11 09:53:18 INFO streaming.StreamJob:  map 0%  reduce 0%
16/02/11 09:53:29 INFO streaming.StreamJob:  map 100%  reduce 0%
16/02/11 09:53:32 INFO streaming.StreamJob:  map 0%  reduce 0%
16/02/11 09:53:48 INFO streaming.StreamJob:  map 50%  reduce 0%
16/02/11 09:53:49 INFO streaming.StreamJob:  map 100%  reduce 0%
16/02/11 09:53:50 INFO streaming.StreamJob:  map 0%  reduce 0%
16/02/11 09:53:59 INFO streaming.StreamJob:  map 100%  reduce 100%
16/02/11 09:54:03 INFO streaming.StreamJob: Job running in-process (local Hadoop)
16/02/11 09:54:03 ERROR streaming.StreamJob: Job not successful.    Error: Task failed task_1455116478271_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/02/11 09:54:03 INFO streaming.StreamJob: killJob...
16/02/11 09:54:03 INFO impl.YarnClientImpl: Killed application    application_1455116478271_0005
Streaming Command Failed!

0 个答案:

没有答案