我正在尝试在Hadoop流上执行Python脚本,但在执行命令映射器后,作业失败了。请帮忙。
我的命令:
hduser@R9HEX3R:~/hadoop-2.5.0-cdh5.3.2$ hadoop jar /home/hduser/hadoop-2.5.0-cdh5.3.2/share/hadoop/mapreduce1/contrib/streaming/hadoop-streaming-2.5.0-mr1-cdh5.3.2.jar -file /home/hduser/mapper.py /home/hduser/reducer.py -mapper /home/hduser/mapper.py -reducer /home/hduser/reducer.py -input /user/hduser/data -output /user/hduser/data_output
错误:
16/02/11 09:52:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/11 09:52:57 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
packageJobJar: [/home/hduser/mapper.py, /home/hduser/reducer.py] [] /tmp/streamjob2731352439731142145.jar tmpDir=null
16/02/11 09:52:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/11 09:52:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/11 09:52:59 INFO mapred.FileInputFormat: Total input paths to process : 1
16/02/11 09:52:59 INFO mapreduce.JobSubmitter: number of splits:2
16/02/11 09:53:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1455116478271_0005
16/02/11 09:53:00 INFO impl.YarnClientImpl: Submitted application application_1455116478271_0005
16/02/11 09:53:00 INFO mapreduce.Job: The url to track the job: http://R9HEX3R:8088/proxy/application_1455116478271_0005/
16/02/11 09:53:00 INFO streaming.StreamJob: getLocalDirs(): [/home/hduser/hadata/mapred/local]
16/02/11 09:53:00 INFO streaming.StreamJob: Running job: job_1455116478271_0005
16/02/11 09:53:00 INFO streaming.StreamJob: Job running in-process (local Hadoop)
16/02/11 09:53:02 INFO streaming.StreamJob: map 0% reduce 0%
16/02/11 09:53:17 INFO streaming.StreamJob: map 50% reduce 0%
16/02/11 09:53:18 INFO streaming.StreamJob: map 0% reduce 0%
16/02/11 09:53:29 INFO streaming.StreamJob: map 100% reduce 0%
16/02/11 09:53:32 INFO streaming.StreamJob: map 0% reduce 0%
16/02/11 09:53:48 INFO streaming.StreamJob: map 50% reduce 0%
16/02/11 09:53:49 INFO streaming.StreamJob: map 100% reduce 0%
16/02/11 09:53:50 INFO streaming.StreamJob: map 0% reduce 0%
16/02/11 09:53:59 INFO streaming.StreamJob: map 100% reduce 100%
16/02/11 09:54:03 INFO streaming.StreamJob: Job running in-process (local Hadoop)
16/02/11 09:54:03 ERROR streaming.StreamJob: Job not successful. Error: Task failed task_1455116478271_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
16/02/11 09:54:03 INFO streaming.StreamJob: killJob...
16/02/11 09:54:03 INFO impl.YarnClientImpl: Killed application application_1455116478271_0005
Streaming Command Failed!