几分钟后，HBase对客户没有回应

Question

我有一个分布式hbase集群，有三个regionserver和两个表，数百万行

有时群集不会响应大约 15分钟的读取和写入请求，这种情况多次发生，每次持续15分钟，大约一周一次

我不知道为什么，并且在日志中没有发现错误，hbase master和区域服务器的GC在可接受的时间内完成，在google之后找不到有用的信息

有人可以帮助我，非常感谢

我的hbase版本是0.94.18，hadoop是1.2.1，而asynchbase 1.4.1是客户端

区域服务器日志在15分钟内没有响应是列表打击：

no response 15 minutes last from 2014-08-31 20:54:45 to 2014-08-31 21:09:16
----------------------------------------------------------------------------------------

2014-08-31 20:41:02,488 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc. after a delay of 7351
2014-08-31 20:41:09,841 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc., current region memstore size 94.1k
2014-08-31 20:41:09,841 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc., syncing WAL and waiting on mvcc, flushsize=96376
2014-08-31 20:41:09,841 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2014-08-31 20:41:09,845 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b with permission=rwxrwxrwx
2014-08-31 20:41:09,846 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-08-31 20:41:09,846 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-08-31 20:41:09,879 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-08-31 20:41:09,880 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b: CompoundBloomFilterWriter
2014-08-31 20:41:09,909 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b)
2014-08-31 20:41:09,909 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315537, memsize=93.9k, into tmp file hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b
2014-08-31 20:41:09,919 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b to hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/a/97b6d77913704f56aef1d39127bea09b
2014-08-31 20:41:09,930 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/a/97b6d77913704f56aef1d39127bea09b, entries=8, sequenceid=3315537, filesize=93.7k
2014-08-31 20:41:09,932 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~94.1k/96376, currentsize=0.0/0 for region bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc. in 91ms, sequenceid=3315537, compaction requested=false
2014-08-31 20:44:42,488 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045. after a delay of 6086
2014-08-31 20:44:48,575 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045., current region memstore size 37.1k
2014-08-31 20:44:48,575 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045., syncing WAL and waiting on mvcc, flushsize=38008
2014-08-31 20:44:48,575 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2014-08-31 20:44:48,582 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e with permission=rwxrwxrwx
2014-08-31 20:44:48,583 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-08-31 20:44:48,583 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-08-31 20:44:48,601 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-08-31 20:44:48,601 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e: CompoundBloomFilterWriter
2014-08-31 20:44:48,624 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e)
2014-08-31 20:44:48,624 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315613, memsize=36.9k, into tmp file hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e
2014-08-31 20:44:48,632 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e to hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/a/f45f88b313904de49eec3f702abf811e
2014-08-31 20:44:48,650 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/a/f45f88b313904de49eec3f702abf811e, entries=8, sequenceid=3315613, filesize=36.6k
2014-08-31 20:44:48,652 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~37.1k/38008, currentsize=0.0/0 for region bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045. in 76ms, sequenceid=3315613, compaction requested=false
2014-08-31 20:45:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=18.95 MB, free=478.97 MB, max=497.92 MB, blocks=219, accesses=956789177, hits=956375329, hitRatio=99.95%, , cachingAccesses=956377890, cachingHits=956349801, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
2014-08-31 20:49:12,491 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a. after a delay of 4475
2014-08-31 20:49:16,967 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a., current region memstore size 141.4k
2014-08-31 20:49:16,967 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a., syncing WAL and waiting on mvcc, flushsize=144792
2014-08-31 20:49:16,967 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2014-08-31 20:49:16,978 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f with permission=rwxrwxrwx
2014-08-31 20:49:16,979 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-08-31 20:49:16,979 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-08-31 20:49:17,021 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-08-31 20:49:17,021 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f: CompoundBloomFilterWriter
2014-08-31 20:49:17,051 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f)
2014-08-31 20:49:17,052 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315721, memsize=141.2k, into tmp file hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f
2014-08-31 20:49:17,060 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f to hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/a/b7b2bdd1f6384466b30a4363c2b7f22f
2014-08-31 20:49:17,071 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/a/b7b2bdd1f6384466b30a4363c2b7f22f, entries=8, sequenceid=3315721, filesize=141.0k
2014-08-31 20:49:17,073 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~141.4k/144792, currentsize=0.0/0 for region bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a. in 106ms, sequenceid=3315721, compaction requested=false
2014-08-31 20:50:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=18.95 MB, free=478.97 MB, max=497.92 MB, blocks=220, accesses=956969362, hits=956555513, hitRatio=99.95%, , cachingAccesses=956558075, cachingHits=956529985, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
2014-08-31 20:54:32,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. after a delay of 21448
2014-08-31 20:54:42,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. after a delay of 21023
2014-08-31 20:54:52,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. after a delay of 17156
2014-08-31 20:54:53,943 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f., current region memstore size 140.1k
2014-08-31 20:54:53,943 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f., syncing WAL and waiting on mvcc, flushsize=143512
2014-08-31 20:54:53,943 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2014-08-31 20:54:53,948 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784 with permission=rwxrwxrwx
2014-08-31 20:54:53,949 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-08-31 20:54:53,949 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-08-31 20:54:53,985 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-08-31 20:54:53,985 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784: CompoundBloomFilterWriter
2014-08-31 20:54:54,012 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784)
2014-08-31 20:54:54,012 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315851, memsize=140.0k, into tmp file hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784
2014-08-31 20:54:54,021 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784 to hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/a/e521dc246e6c4216bbb368f0ce98e784
2014-08-31 20:54:54,032 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/a/e521dc246e6c4216bbb368f0ce98e784, entries=8, sequenceid=3315851, filesize=139.7k
2014-08-31 20:54:54,034 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~140.1k/143512, currentsize=0.0/0 for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. in 91ms, sequenceid=3315851, compaction requested=false
2014-08-31 20:55:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=18.95 MB, free=478.97 MB, max=497.92 MB, blocks=221, accesses=957103402, hits=956689552, hitRatio=99.95%, , cachingAccesses=956692115, cachingHits=956664024, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
2014-08-31 20:56:27,483 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 2176807346167982283 lease expired on region messages,07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.1a34b2e385e39cf3a82ae55f732c7a4e.
.......................
2014-08-31 20:56:38,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -6933263087733055502 lease expired on region messages,11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.198c072806a86f7103a0f3c6de11290a.
2014-08-31 20:56:38,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -2230372302736915424 lease expired on region messages,08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.3ab90beebbc002e19a2f024cb74e5e99.
2014-08-31 20:59:55,789 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of info because one (major) compacted file only and oldestTime 7395388096ms is < ttl=9223372036854775807
2014-08-31 21:00:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957113714, hits=956699856, hitRatio=99.95%, , cachingAccesses=956702427, cachingHits=956674328, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
2014-08-31 21:09:41,115 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access missing scanner -5479515347986367613
.......................
2014-08-31 21:10:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957150432, hits=956736574, hitRatio=99.95%, , cachingAccesses=956739145, cachingHits=956711046, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
2014-08-31 21:10:41,126 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 2475283676294571093 lease expired on region messages,07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.1a34b2e385e39cf3a82ae55f732c7a4e.
.........................
2014-08-31 21:10:41,127 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -1230151222882130492 lease expired on region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
2014-08-31 21:10:41,127 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -3189762669648092281 lease expired on region 
...............
2014-08-31 21:10:41,731 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 4766251358165302168 lease expired on region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
2014-08-31 21:15:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957331387, hits=956917529, hitRatio=99.95%, , cachingAccesses=956920100, cachingHits=956892001, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
2014-08-31 21:16:32,506 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. after a delay of 5644
2014-08-31 21:16:32,506 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd., current region memstore size 195.8k
2014-08-31 21:16:38,151 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting 
2014-08-31 21:16:38,156 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a with permission=rwxrwxrwx
2014-08-31 21:16:38,157 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-08-31 21:16:38,157 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-08-31 21:16:38,165 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-08-31 21:16:38,166 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a: CompoundBloomFilterWriter
2014-08-31 21:16:38,188 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a)
2014-08-31 21:16:38,189 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3316344, memsize=180.3k, into tmp file hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a
2014-08-31 21:16:38,197 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a to hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a
2014-08-31 21:16:38,208 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, entries=580, sequenceid=3316344, filesize=106.7k
2014-08-31 21:16:38,210 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~195.8k/200488, currentsize=0.0/0 for region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. in 59ms, sequenceid=3316344, compaction requested=true
2014-08-31 21:16:38,210 DEBUG org.apache.hadoop.hbase.regionserver.Store: d364a5f6bc7e54e72d915a3198636afd - a: Initiating minorcompaction
2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on a in region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in a of messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. into tmpdir=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp, seqid=3316344, totalSize=4.0m
2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on a in region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in a of messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. into tmpdir=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp, seqid=3316344, totalSize=4.0m
2014-08-31 21:16:38,222 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: Compacting hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/6f8d2fc1160245318ddec3f193982d9c, keycount=7820, bloomtype=NONE, size=3.6m, encoding=NONE
2014-08-31 21:16:38,223 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: Compacting hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b, keycount=592, bloomtype=NONE, size=250.5k, encoding=NONE
2014-08-31 21:16:38,223 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: Compacting hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, keycount=580, bloomtype=NONE, size=106.7k, encoding=NONE
2014-08-31 21:16:38,223 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction requested: regionName=messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd., storeName=a, fileCount=3, fileSize=4.0m (3.6m, 250.5k, 106.7k), priority=3, time=7733965975674385; Because: regionserver60020.cacheFlusher; compaction_queue=(0:0), split_queue=0
2014-08-31 21:16:38,224 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21 with permission=rwxrwxrwx
2014-08-31 21:16:38,225 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-08-31 21:16:38,225 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-08-31 21:16:38,229 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-08-31 21:16:38,229 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21: CompoundBloomFilterWriter
2014-08-31 21:16:38,383 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21)
2014-08-31 21:16:38,396 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21 to hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/0cfab02d23184841b0185dbcb3c7cb21
2014-08-31 21:16:38,406 DEBUG org.apache.hadoop.hbase.regionserver.Store: Removing store files after compaction...
2014-08-31 21:16:38,409 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving compacted store files.
2014-08-31 21:16:38,410 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Starting to archive files:[class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, ..........................................
for:hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b, free to archive original file.
2014-08-31 21:16:38,419 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Finished archiving file from: class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b, to: hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b
2014-08-31 21:16:38,419 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a
2014-08-31 21:16:38,420 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: No existing file in archive for:hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, free to archive original file.
2014-08-31 21:16:38,423 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Finished archiving file from: class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, to: hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a
2014-08-31 21:16:38,423 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in a of messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. into 0cfab02d23184841b0185dbcb3c7cb21, size=4.0m; total size for store is 332.2m
2014-08-31 21:16:38,423 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd., storeName=a, fileCount=3, fileSize=4.0m, priority=3, time=7733965975674385; duration=0sec
2014-08-31 21:16:38,424 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(0:0), split_queue=0
2014-08-31 21:16:40,082 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for messages,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.5d9b973aef7e10eee0e32fecdbeb37e9., current region memstore size 245.3k
2014-08-31 21:16:40,083 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting messages,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.5d9b973aef7e10eee0e32fecdbeb37e9., syncing WAL and waiting on mvcc, flushsize=251224
2014-08-31 21:16:40,083 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores

Answer 1

尝试做以下事情

运行jstat以查看是否发生了LONG GC暂停。

sudo -u username jstat -gc pid
sudo -u username jmap -histo:live pid

以下一组日志：

2014-08-31 20:56:38,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -6933263087733055502 lease expired on region messages,11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.198c072806a86f7103a0f3c6de11290a.
2014-08-31 20:56:38,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -2230372302736915424 lease expired on region messages,08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.3ab90beebbc002e19a2f024cb74e5e99.
2014-08-31 20:59:55,789 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of info because one (major) compacted file only and oldestTime 7395388096ms is < ttl=9223372036854775807
2014-08-31 21:00:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957113714, hits=956699856, hitRatio=99.95%, , cachingAccesses=956702427, cachingHits=956674328, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, e victedPerRun=Infinity
2014-08-31 21:09:41,115 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access missing scanner -5479515347986367613

显然，由于读取（扫描是您正在执行的操作），Hbase块缓存正在被搅动。

如果您使用MR扫描表格，请禁用setblockCache（false）。