如何从Apache Cassandra CommitLogReplayException中恢复

时间:2016-04-29 16:00:03

标签: cassandra replay

我在运行apache-cassandra 3.3时意外关闭了Mac OS X.在新引导加载中重新启动cassandra时,出现以下错误:

INFO  15:25:15 Replaying /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351787.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461942807785.log, /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461943004975.log
ERROR 15:25:15 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 542295 of commit log /Users/eswenson/opt/apache-cassandra-3.5/bin/../data/commitlog/CommitLog-6-1461717351786.log, with bad position but valid CRC
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:105) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.SegmentReader$SegmentIterator.computeNext(SegmentReader.java:78) [apache-cassandra-3.5.jar:3.5]
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) [guava-18.0.jar:na]
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) [guava-18.0.jar:na]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:399) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.jar:3.5]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.jar:3.5]

我搜索了堆栈溢出和谷歌,但没有发现同样问题的帖子。我发现问题https://issues.apache.org/jira/browse/CASSANDRA-9749是相关的,但是这个问题被标记为已解决,并且与检测到问题后继续执行有关,而不是因为异常而失败。

在这种情况下我该怎么办?是否有任何恢复路径(假设没有可用的快照)?有什么方法可以在违规记录中截断提交日志?任何可用于修复错误标题的工具(例如删除记录)?

感谢。

3 个答案:

答案 0 :(得分:9)

在Mac OS上与我当地的Cassandra一起玩时,我也遇到过这种情况。

如果您不关心丢失提交日志中的数据,或者您在某处有备份,请停止Cassandra,清理commitlog文件夹并重新启动。

我担心没有其他选择。顺便说一句,您是否在Mac OS上部署Cassandra进行生产?

  • 如果是,您应该知道Mac OS不受官方支持
  • 如果不是为了生产,那么谁在乎,只需清理commitlog并重新启动

答案 1 :(得分:2)

在Windows 7(本地计算机)环境中运行Cassandra时遇到了类似的问题。我在日志中找到的例外是

错误07:14:35在初始化期间处理提交日志时因错误而退出。 org.apache.cassandra.db.commitlog.CommitLogReplayer $ CommitLogReplayException:无法读取文件C中的提交日志描述符:\ Program Files \ DataStax-DDC \ data \ commitlog \ CommitLog-6-1470969948498.log     在org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:616)[apache-cassandra-3.7.0.jar:3.7.0]     在org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:378)[apache-cassandra-3.7.0.jar:3.7.0]

由于它只是一个测试设置,我删除了该特定的提交日志文件,并从Windows服务管理器重新启动它,它开始完美。我不确定如果它发生在生产环境中该怎么办。

答案 2 :(得分:0)

我第二次在我的单节点开发人员系统上得到这个。如果jvm意外被杀,似乎能够发生。清除commitlog目录解决了它。使用Windows 10.

让我担心的是,即使没有对数据库进行更新,我也会遇到这些错误问题。 cassandra不定期冲洗。我不想因为服务器崩溃或类似问题而在生产中遇到类似这样的问题,即使我有副本节点。

在stdout日志中,最后一部分是

INFO  06:17:39 Replaying C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812251.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471411951134.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471454506802.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471532812678.log
ERROR 06:17:39 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:373) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.4.0.jar:3.4.0]