无法在缓存

时间:2017-09-18 04:58:33

标签: hadoop hdfs spark-streaming kerberos kerberos-delegation

我正在运行一个部署在纱线客户端模式的火花流工作,它经常处理HDFS,我们的hadoop集群版本是hadoop-2.6.0-cdh5.7.3,并且jira HDFS-9276中的补丁文件已经引入到此版本,但在几天后(大多数是7天)我仍然会遇到如下错误:

  

18-09-2017 10:05:48 CST crm_user_select ERROR - 17/09/18 10:05:48 WARN security.UserGroupInformation:PriviledgedActionException as:bd_recom @ FHC(auth:KERBEROS)cause:org.apache.hadoop .ipc.RemoteException(org.apache.hadoop.security.token.SecretManager $ InvalidToken):token(bd_recom的令牌:HDFS_DELEGATION_TOKEN owner = bd_recom @ FHC,renewer = yarn,realUser =,issueDate = 1505095524480,maxDate = 1505700324480,sequenceNumber = 2244503,masterKeyId = 504)已过期   18-09-2017 10:05:48 CST crm_user_select错误 - 17/09/18 10:05:48 WARN ipc.Client:连接服务器时遇到异常:org.apache.hadoop.ipc.RemoteException(org.apache) .hadoop.security.token.SecretManager $ InvalidToken):token(bd_recom的令牌:HDFS_DELEGATION_TOKEN owner = bd_recom @ FHC,renewer = yarn,realUser =,issueDate = 1505095524480,maxDate = 1505700324480,sequenceNumber = 2244503,masterKeyId = 504)已过期   18-09-2017 10:05:48 CST crm_user_select ERROR - 17/09/18 10:05:48 WARN security.UserGroupInformation:PriviledgedActionException as:bd_recom @ FHC(auth:KERBEROS)cause:org.apache.hadoop.ipc。 RemoteException(org.apache.hadoop.security.token.SecretManager $ InvalidToken):token(bd_recom的令牌:HDFS_DELEGATION_TOKEN owner = bd_recom @ FHC,renewer = yarn,realUser =,issueDate = 1505095524480,maxDate = 1505700324480,sequenceNumber = 2244503,masterKeyId = 504)已过期   18-09-2017 10:05:48 CST crm_user_select错误 - 17/09/18 10:05:48警告hdfs.LeaseRenewer:未能续订[DFSClient_NONMAPREDUCE_-2053099090_1]的租约30秒。很快就会重试......   18-09-2017 10:05:48 CST crm_user_select错误 - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager $ InvalidToken):token(bd_recom的令牌:HDFS_DELEGATION_TOKEN owner = bd_recom @ FHC,更新者=纱线,realUser =,issueDate = 1505095524480,maxDate = 1505700324480,sequenceNumber = 2244503,masterKeyId = 504)已过期   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1471)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1408)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke(ProtobufRpcEngine.java:230)   18-09-2017 10:05:48 CST crm_user_select ERROR - at com.sun.proxy。$ Proxy14.renewLease(Unknown Source)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:576)   18-09-2017 10:05:48 CST crm_user_select ERROR - at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)   18-09-2017 10:05:48 CST crm_user_select ERROR - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)   18-09-2017 10:05:48 CST crm_user_select错误 - 在java.lang.reflect.Method.invoke(Method.java:606)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)   18-09-2017 10:05:48 CST crm_user_select ERROR - at com.sun.proxy。$ Proxy15.renewLease(Unknown Source)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:941)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.access $ 700(LeaseRenewer.java:71)   18-09-2017 10:05:48 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer $ 1.run(LeaseRenewer.java:304)   18-09-2017 10:05:48 CST crm_user_select错误 - 在java.lang.Thread.run(Thread.java:745)

并关注错误信息:

  

18-09-2017 10:19:35 CST crm_user_select ERROR - 17/09/18 10:19:35 WARN security.UserGroupInformation:PriviledgedActionException as:bd_recom @ FHC(auth:KERBEROS)cause:org.apache.hadoop .ipc.RemoteException(org.apache.hadoop.security.token.SecretManager $ InvalidToken):token(bd_recom的令牌:HDFS_DELEGATION_TOKEN owner = bd_recom @ FHC,renewer = yarn,realUser =,issueDate = 1505095524480,maxDate = 1505700324480,sequenceNumber = 2244503,masterKeyId = 504)无法在缓存中找到   18-09-2017 10:19:35 CST crm_user_select ERROR - 17/09/18 10:19:35 WARN hdfs.LeaseRenewer:无法续订[DFSClient_NONMAPREDUCE_-2053099090_1]的租约857秒。很快就会重试......   18-09-2017 10:19:35 CST crm_user_select错误 - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager $ InvalidToken):token(bd_recom的令牌:HDFS_DELEGATION_TOKEN owner = bd_recom @ FHC,更新者=纱线,realUser =,issueDate = 1505095524480,maxDate = 1505700324480,sequenceNumber = 2244503,masterKeyId = 504)无法在缓存中找到   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1471)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.ipc.Client.call(Client.java:1408)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke(ProtobufRpcEngine.java:230)   18-09-2017 10:19:35 CST crm_user_select ERROR - at com.sun.proxy。$ Proxy14.renewLease(Unknown Source)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:576)   18-09-2017 10:19:35 CST crm_user_select ERROR - at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)   18-09-2017 10:19:35 CST crm_user_select ERROR - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)   18-09-2017 10:19:35 CST crm_user_select错误 - 在java.lang.reflect.Method.invoke(Method.java:606)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)   18-09-2017 10:19:35 CST crm_user_select ERROR - at com.sun.proxy。$ Proxy15.renewLease(Unknown Source)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:941)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer.access $ 700(LeaseRenewer.java:71)   18-09-2017 10:19:35 CST crm_user_select ERROR - at org.apache.hadoop.hdfs.LeaseRenewer $ 1.run(LeaseRenewer.java:304)   18-09-2017 10:19:35 CST crm_user_select错误 - 在java.lang.Thread.run(Thread.java:745)

再见: 1. NameNode HA已启用。 2. Kerberos已启用。 3. HDFS委托令牌(不是Keytab或TGT)用于与NameNode通信。

我试过使用配置" --conf spark.hadoop.fs.hdfs.impl.disable.cache = true",但它没有用。所以任何人都可以帮助我,我真的很感激!

0 个答案:

没有答案