让Regionserver在日志

Question

我注意到我的区域服务器日志中出现以下错误：

org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/my-table/eb512b4b9f9fa9cb2a1a3930d9c9f18b/r/df1694a4542f419992f86b219541fb6fBlock token with block_token_identifier (expiryDate=1519482398334, keyId=1283446178, userId=hbase, blockPoolId=BP-1872413417-101.33.253.88-1458393583173, blockId=1133036852, access modes=[READ]) is expired.
        at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:591)
        at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:490)
        at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782)
        at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716)
        at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
        at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
        at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1118)
        at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1056)
        at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1411)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1374)
        at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:89)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1380)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1591)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1470)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:437)
        at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:259)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:217)
        at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2003)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5338)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2494)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2480)
        at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2462)
        at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6527)
        at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6506)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:579)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
        at java.lang.Thread.run(Thread.java:745)
2018-02-24 20:03:10,947 INFO  [RW.default.readRpcServer.handler=151,queue=38,port=16020] hdfs.DFSClient: Access token was invalid when connecting to /101.32.65.239:50010 : org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/my-table/eb512b4b9f9fa9cb2a1a3930d9c9f18b/r/df1694a4542f419992f86b219541fb6fBlock token with block_token_identifier (expiryDate=1519482398334, keyId=1283446178, userId=hbase, blockPoolId=BP-1872413417-101.33.253.88-1458393583173, blockId=1133036852, access modes=[READ]) is expired.

我们发现与config相关的tweaked已经JIRA了。

dfs.client.read.shortcircuit.streams.cache.expiry.ms 300000 HDFS-default.xml中

dfs.client.read.shortcircuit.streams.cache.size 4096 HDFS-site.xml中

我还看到Nick Dimiduk-2提出的已经面临此问题，该问题仍处于打开状态，并且评论中包含了面临同一问题的用户建议的配置，我们也在使用，

任何人都遇到过这个问题或知道这个问题，并对此有任何解决方案/建议吗？