Hadoop作业在两次之间失败

时间:2018-07-25 12:52:59

标签: hadoop hadoop2 cloudera-cdh hadoop-streaming

我在运行hadoop作业时对Hadoop完全陌生。我的工作在map 67% reduce 0%处停止,出现600秒超时错误,无法理解是什么问题。我确实从堆栈溢出答案之一修改了yarn-site.xml。我的mapper和reducer在python中,我在cloudera VM 5.13中使用hadoop流jar运行它。

我的yarn-site.xml如下

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <property>
    <name>yarn.dispatcher.exit-on-error</name>
    <value>true</value>
  </property>

  <property>
    <description>List of directories to store localized files in.</description>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
  </property>

 <property>
    <description>Where to store container logs.</description>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/var/log/hadoop-yarn/containers</value>
  </property>

  <property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/var/log/hadoop-yarn/apps</value>
  </property>

  <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
     <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
     </value>
  </property>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
   <description>Whether virtual memory limits will be enforced for containers</description>
</property>

<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
   <value>4</value>
   <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

该作业的日志为:

2018-07-25 05:19:54,036 INFO [IPC Server handler 0 on 46431] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1532520855082_0001_m_000000_0 is : 0.667
2018-07-25 05:19:55,335 INFO [IPC Server handler 1 on 46431] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1532520855082_0001_m_000001_0 is : 0.667
2018-07-25 05:29:55,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1532520855082_0001_m_000001_0: AttemptID:attempt_1532520855082_0001_m_000001_0 Timed out after 600 secs
2018-07-25 05:29:55,782 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000001_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2018-07-25 05:29:55,783 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1532520855082_0001_m_000000_0: AttemptID:attempt_1532520855082_0001_m_000000_0 Timed out after 600 secs
2018-07-25 05:29:55,783 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2018-07-25 05:29:55,791 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1532520855082_0001_01_000003 taskAttempt attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:55,791 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:55,798 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1532520855082_0001_01_000002 taskAttempt attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:55,799 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:55,997 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000001_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2018-07-25 05:29:55,997 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2018-07-25 05:29:56,007 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2018-07-25 05:29:56,023 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2018-07-25 05:29:56,123 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://quickstart.cloudera:8020/user/cloudera/wd_count3/_temporary/1/_temporary/attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,124 WARN [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://quickstart.cloudera:8020/user/cloudera/wd_count3/_temporary/1/_temporary/attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:56,167 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000001_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2018-07-25 05:29:56,168 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2018-07-25 05:29:56,193 WARN [IPC Server handler 2 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,194 INFO [IPC Server handler 2 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 2 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000001_0, org.apache.hadoop.mapred.MapTaskStatus@5451d861), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59696 Call#105 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,209 WARN [IPC Server handler 1 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,209 INFO [IPC Server handler 1 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 1 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000001_0, org.apache.hadoop.mapred.MapTaskStatus@128bf171), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59696 Call#106 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,214 WARN [IPC Server handler 4 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,215 INFO [IPC Server handler 4 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 4 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000001_0, org.apache.hadoop.mapred.MapTaskStatus@11586e11), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59696 Call#107 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,222 WARN [IPC Server handler 6 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,222 INFO [IPC Server handler 6 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 6 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000001_0, org.apache.hadoop.mapred.MapTaskStatus@71ad8d97), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59696 Call#108 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,224 WARN [IPC Server handler 5 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,224 INFO [IPC Server handler 5 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 5 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000001_0, org.apache.hadoop.mapred.MapTaskStatus@31a4755b), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59696 Call#109 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,235 WARN [IPC Server handler 0 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
2018-07-25 05:29:56,235 INFO [IPC Server handler 0 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 0 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000001_0, org.apache.hadoop.mapred.MapTaskStatus@72d39e39), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59696 Call#110 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000001_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,269 WARN [IPC Server handler 7 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:56,271 INFO [IPC Server handler 7 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 7 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000000_0, org.apache.hadoop.mapred.MapTaskStatus@18711ad), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59692 Call#105 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,278 WARN [IPC Server handler 10 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:56,278 INFO [IPC Server handler 10 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 10 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000000_0, org.apache.hadoop.mapred.MapTaskStatus@381636de), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59692 Call#106 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,284 WARN [IPC Server handler 14 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:56,284 INFO [IPC Server handler 14 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 14 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000000_0, org.apache.hadoop.mapred.MapTaskStatus@72e024cb), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59692 Call#107 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,291 WARN [IPC Server handler 8 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:56,291 INFO [IPC Server handler 8 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 8 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000000_0, org.apache.hadoop.mapred.MapTaskStatus@6e26e957), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59692 Call#108 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,317 WARN [IPC Server handler 9 on 46431] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:job_1532520855082_0001 (auth:TOKEN) cause:java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
2018-07-25 05:29:56,317 INFO [IPC Server handler 9 on 46431] org.apache.hadoop.ipc.Server: IPC Server handler 9 on 46431, call statusUpdate(attempt_1532520855082_0001_m_000000_0, org.apache.hadoop.mapred.MapTaskStatus@12e07f77), rpc version=2, client version=19, methodsFingerPrint=937413979 from 10.0.2.15:59692 Call#109 Retry#0
java.io.IOException: java.lang.IllegalStateException: Status update was called with illegal TaskAttemptId: attempt_1532520855082_0001_m_000000_0
    at org.apache.hadoop.mapred.TaskAttemptListenerImpl.statusUpdate(TaskAttemptListenerImpl.java:349)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:516)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2018-07-25 05:29:56,421 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node quickstart.cloudera
2018-07-25 05:29:56,451 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000001_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-07-25 05:29:56,453 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node quickstart.cloudera
2018-07-25 05:29:56,453 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1532520855082_0001_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-07-25 05:29:56,453 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1532520855082_0001_m_000001_1 to list of failed maps
2018-07-25 05:29:56,454 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1532520855082_0001_m_000000_1 to list of failed maps
2018-07-25 05:29:56,818 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:1 HostLocal:2 RackLocal:0
2018-07-25 05:29:56,821 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1532520855082_0001: ask=1 release= 0 newContainers=0 finishedContainers=2 resourcelimit=<memory:6144, vCores:7> knownNMs=1
2018-07-25 05:29:56,822 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1532520855082_0001_01_000003
2018-07-25 05:29:56,822 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1532520855082_0001_m_000001_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

我是一个完整的初学者。谢谢

0 个答案:

没有答案