Question

15/08/04 10:58:14 INFO streaming.CheckpointWriter: Saving checkpoint for time 1438666094000 ms to file 'hdfs://Bootcamp2015-1/user/root/checkpoint/checkpoint-1438666094000'
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, Bootcamp2015-1, PROCESS_LOCAL, 1358 bytes)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 116 ms on Bootcamp2015-1 (32/50)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 135 ms on Bootcamp2015-2 (33/50)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, Bootcamp2015-2, PROCESS_LOCAL, 1358 bytes)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, Bootcamp2015-1, PROCESS_LOCAL, 1358 bytes)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 125 ms on Bootcamp2015-1 (34/50)
15/08/04 10:58:14 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/root/checkpoint/temp (inode 146905): File does not exist. [Lease.  Holder: DF$
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3605)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3402)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3258)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:668)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

        at org.apache.hadoop.ipc.Client.call(Client.java:1468)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, Bootcamp2015-2, PROCESS_LOCAL, 1358 bytes)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 113 ms on Bootcamp2015-2 (35/50)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, Bootcamp2015-1, PROCESS_LOCAL, 1358 bytes)
15/08/04 10:58:14 INFO scheduler.TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 87 ms on Bootcamp2015-1 (36/50)
15/08/04 10:58:14 WARN streaming.CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://Bootcamp2015-1/user/root/checkpoint/checkpoint-1438666094000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/root/checkpoint/temp (inode 146905): File does not exist. [Lease.  Holder: DF$

Answer 1

我知道这是一个老问题，但我遇到了同样的问题。发生这种情况是因为2个不同的流应用程序使用相同的检查点目录。在为每个应用程序附加随机字符串后，问题就消失了。

运行Kafka SparkStreaming简单字数计数示例时出错

1 个答案: