Question

我试图使用createtable命令在Accumulo中创建一个表，并发现它被卡住了。在取消createtable命令之前我等了大约20分钟。

createtable test_table

我有一台主服务器和2台平板电脑服务器，发现我的主人和其中一个平板电脑已经死亡。我无法telnet到该特定平板电脑服务器的端口9997，我甚至无法telnet到端口29999（accumulo-site.xml中的master.port.client）。当我看到死服务器的服务器日志时，我看到了以下条目。

2016-05-10 02:12:07,456 [zookeeper.DistributedWorkQueue] INFO : Got unexpected z
ookeeper event: None for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/recovery
2016-05-10 02:12:23,883 [zookeeper.ZooCache] WARN : Saw (possibly) transient exc
eption communicating with ZooKeeper, will retry
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/tables
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.accumulo.fate.zookeeper.ZooCache$1.run(ZooCache.java:210)
        at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
        at org.apache.accumulo.fate.zookeeper.ZooCache.getChildren(ZooCache.java
:221)
        at org.apache.accumulo.core.client.impl.Tables.exists(Tables.java:142)
        at org.apache.accumulo.server.tabletserver.LargestFirstMemoryManager.tab
leExists(LargestFirstMemoryManager.java:149)
        at org.apache.accumulo.server.tabletserver.LargestFirstMemoryManager.get
MemoryManagementActions(LargestFirstMemoryManager.java:175)
        at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagem
entFramework.manageMemory(TabletServerResourceManager.java:408)
        at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagem
entFramework.access$400(TabletServerResourceManager.java:318)
        at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagem
entFramework$2.run(TabletServerResourceManager.java:346)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.jav
a:35)
        at java.lang.Thread.run(Thread.java:745)
2016-05-10 02:12:23,884 [zookeeper.ZooCache] WARN : Saw (possibly) transient exc
eption communicating with ZooKeeper, will retry
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/tables/!0/con
f/table.classpath.context
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
        at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
        at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
        at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
        at org.apache.accumulo.server.conf.ZooCachePropertyAccessor.get(ZooCache
PropertyAccessor.java:117)
        at org.apache.accumulo.server.conf.ZooCachePropertyAccessor.get(ZooCache
PropertyAccessor.java:103)
        at org.apache.accumulo.server.conf.TableConfiguration.get(TableConfigura
tion.java:99)
        at org.apache.accumulo.tserver.constraints.ConstraintChecker.classLoader
Changed(ConstraintChecker.java:93)
        at org.apache.accumulo.tserver.tablet.Tablet.checkConstraints(Tablet.jav
a:1225)
        at org.apache.accumulo.tserver.TabletServer$8.run(TabletServer.java:2848
)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51
1)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-05-10 02:12:23,887 [zookeeper.ZooReader] WARN : Saw (possibly) transient ex
ception communicating with ZooKeeper
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/tservers/accu
mulo.tablet.2:9997
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.accumulo.fate.zookeeper.ZooReader.getStatus(ZooReader.java
:132)
        at org.apache.accumulo.fate.zookeeper.ZooLock.process(ZooLock.java:383)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.j
ava:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2016-05-10 02:12:24,252 [watcher.MonitorLog4jWatcher] INFO : Changing monitor lo
g4j address to accumulo.master:4560
2016-05-10 02:12:24,252 [watcher.MonitorLog4jWatcher] INFO : Enabled log-forward
ing

即使主服务器的日志也有相同的堆栈跟踪。我的动物园管理员正在跑步。

起初，我认为这是一个磁盘问题。也许没有空间。但事实并非如此。我在accumulo instance.volumes上运行了fsck，它返回了HEALTHY状态。

有谁知道究竟发生了什么，如果可能的话，如何避免它？

编辑：即使是tracer_accumulo.master.log也有相同的堆栈跟踪。

Answer 1

当ZooKeeper客户端内的线程在必要时间内（默认情况下为30秒）未运行时，ZooKeeper会话到期发生，以维持ZooKeeper客户端和服务器之间的内存状态会话。对此没有单一的解释，但许多常见的罪魁祸首：

JVM垃圾回收在客户端暂停。如果遇到暂停，Accumulo应该会发出警告。
缺少CPU时间。如果主机本身负担过重，Accumulo可能没有足够的周期来及时运行所需的所有任务。
缺少套接字/文件句柄，Accumulo可能尝试连接到ZooKeeper，但无法打开新连接
ZooKeeper可能是速率限制连接，作为拒绝服务预防。检查zookeeper日志，查看有关从特定IP中删除/拒绝新连接的错误，如果发现这些错误，请考虑在zoo.cfg中增加maxClientCnxns。

Accumulo的createtable命令卡住了，不会创建表

1 个答案: