无法杀死YARN的工作

时间:2016-05-19 07:30:48

标签: java hadoop yarn apache-samza

我有一个简单的Samza工作,我提交给我们的YARN集群。该作业分配一个容器并运行没有任何问题。

然而,当尝试杀死作业时,AM和作业容器都在NM上运行,即使RM声称该作业已成功杀死:

$ yarn application -kill application_1461969364354_5761
Killing application application_1461969364354_5761
16/05/19 06:48:49 INFO impl.YarnClientImpl: Killed application application_1461969364354_5761

从NM日志中,我能够看到:

2016-05-19 06:48:50,051 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e34_1461969364354_5761_01_000002 transitioned from RUNNING to KILLING
2016-05-19 06:48:50,051 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e34_1461969364354_5761_01_000002
2016-05-19 06:48:50,060 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1461969364354_5761 transitioned from RUNNING to FINISHING_CONTAI
NERS_WAIT

状态永远不会从FINISHING_CONTAINERS_WAIT转出,我不得不kill -9容器进程。

我使用的是Samza版本0.10.0和YARN版本Hadoop 2.6.0-cdh5.4.9

有什么想法吗?

更新:

经过一番挖掘后,我能够看到这一点:

2016-05-20 03:14:59,497 INFO org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking finishApplicationMaster of class ApplicationMasterProtocolPBClientImpl over rm157 after 2326 fail over attempts. Trying to fail over immediately.
org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1463512986427_0017_000001 not found in AMRMTokenSecretManager.
        at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
        at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:94)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy16.finishApplicationMaster(Unknown Source)
        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:378)
        at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:157)
        at org.apache.samza.job.yarn.SamzaAppMasterLifecycle.onShutdown(SamzaAppMasterLifecycle.scala:63)
        at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$5.apply(SamzaAppMaster.scala:133)
        at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$5.apply(SamzaAppMaster.scala:132)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.samza.job.yarn.SamzaAppMaster$.run(SamzaAppMaster.scala:132)
        at org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:104)
        at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): appattempt_1463512986427_0017_000001 not found in AMRMTokenSecretManager.        
        at org.apache.hadoop.ipc.Client.call(Client.java:1468)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy15.finishApplicationMaster(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:91)
        ... 15 more

0 个答案:

没有答案