在hadoop上运行作业时,出现以下错误

时间:2018-04-03 09:16:00

标签: python hadoop mapreduce hdfs

我试图运行地图减少工作。我的mapper代码和reducer代码(python代码)在我的本地系统中工作正常。但是当我试图在hadoop上运行它时,我收到以下错误

错误:java.lang.RuntimeException:PipeMapRed.waitOutputThreads():子进程失败,代码为127         在org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)         在org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)         在org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)         在org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)         在org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)         在org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)         在org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)         在org.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:167)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:415)         在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)         在org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

15/01/22 23:32:06 INFO mapreduce.Job:任务ID:attempt_1414619043632_78083_m_000000_0,状态:未通过 错误:java.lang.RuntimeException:PipeMapRed.waitOutputThreads():子进程失败,代码为127         在org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)         在org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)         在org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)         在org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)         在org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)         在org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)         在org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)         在org.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:167)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:415)         在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)         在org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

15/01/22 23:32:14 INFO mapreduce.Job:任务ID:attempt_1414619043632_78083_m_000000_1,状态:未通过 错误:java.lang.RuntimeException:PipeMapRed.waitOutputThreads():子进程失败,代码为127         在org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)         在org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)         在org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)         在org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)         在org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)         在org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)         在org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)         在org.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:167)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:415)         在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)         在org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

15/01/22 23:32:14 INFO mapreduce.Job:任务ID:attempt_1414619043632_78083_m_000001_1,状态:未通过 错误:java.lang.RuntimeException:PipeMapRed.waitOutputThreads():子进程失败,代码为127         在org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)         在org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)         在org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)         在org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)         在org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)         在org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)         在org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)         在org.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:167)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:415)         在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)         在org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

15/01/22 23:32:21 INFO mapreduce.Job:任务ID:attempt_1414619043632_78083_m_000000_2,状态:未通过 错误:java.lang.RuntimeException:PipeMapRed.waitOutputThreads():子进程失败,代码为127         在org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)         在org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)         在org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)         在org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)         在org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)         在org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)         在org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)         在org.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:167)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:415)         在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)         在org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

15/01/22 23:32:22 INFO mapreduce.Job:任务ID:attempt_1414619043632_78083_m_000001_2,状态:未通过 错误:java.lang.RuntimeException:PipeMapRed.waitOutputThreads():子进程失败,代码为127         在org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)         在org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)         在org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)         在org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)         在org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)         在org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)         在org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)         在org.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:167)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:415)         在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)         在org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

15/01/22 23:32:30 INFO mapreduce.Job:地图100%减少100% 15/01/22 23:32:31 INFO mapreduce.Job:作业job_1414619043632_78083因状态失败而失败,原因是:任务失败task_1414619043632_78083_m_000001 任务失败,作业失败。 failedMaps:1次失败减少:0

15/01/22 23:32:31 INFO mapreduce.Job:专柜:12         工作柜台                 失败的地图任务= 7                 被杀死的地图任务= 1                 启动地图任务= 8                 其他本地地图任务= 8                 占用时隙中所有映射所花费的总时间(ms)= 51787                 所有花费的总时间减少了占用的时隙(ms)= 0                 所有地图任务花费的总时间(ms)= 51787                 所有地图任务占用的总核心秒数= 51787                 所有映射任务占用的总兆字节数= 450754048         Map-Reduce框架                 花费的CPU时间(ms)= 0                 物理内存(字节)snapshot = 0                 虚拟内存(字节)snapshot = 0 15/01/22 23:32:31 ERROR streaming.StreamJob:工作不成功!

0 个答案:

没有答案