我有一个批量Python脚本将数据上传到Cassandra数据库。简而言之,它循环遍历数据行列表,并将每个数据行的时间戳上传到Cassandra。
我注意到,脚本每隔一段时间就会崩溃,并出现类似于以下内容的错误:
Failed to write to cassandra table: ('Unable to complete the operation against any hosts',{})
此时,nodetool statusbinary
返回not running
,我需要做的就是运行nodetool enablebinary
以使脚本再次运行 - 直到下一次崩溃为止。
我没有看到任何东西在原木中关闭原生运输 - 这可能是二进制自身关闭的原因 -
版本号:Cassandra:3.0.9 Java:1.8.0_162 CentOS:6.9
编辑:阅读日志,我注意到以下内容:
INFO [RMI TCP Connection(5893)-127.0.0.1] 2018-05-08 10:58:17,318 Server.java:160 - Starting listening for CQL clients on /XX.XXX.X.XX:9042 (unencrypted)...
ERROR [CompactionExecutor:29] 2018-05-08 10:58:48,700 CassandraDaemon.java:205 - Exception in thread Thread[CompactionExecutor:29,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Channel not open for writing - cannot extend file to required size
更长一点:
WARN [CompactionExecutor:4] 2018-05-09 12:10:11,094 CLibrary.java:304 - fsync(315) failed, errno (22) {}
com.sun.jna.LastErrorException: [22] h
at org.apache.cassandra.utils.CLibrary.fsync(Native Method) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.CLibrary.trySync(CLibrary.java:293) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.SyncUtil.trySync(SyncUtil.java:179) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.SyncUtil.trySyncDir(SyncUtil.java:190) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:117) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:135) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:69) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:163) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:73) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.switchCompactionLocation(DefaultCompactionWriter.java:64) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.maybeSwitchWriter(CompactionAwareWriter.java:128) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:108) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) [apache-cassandra-3.0.9.jar:3.0.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
WARN [CompactionExecutor:4] 2018-05-09 12:10:11,096 CLibrary.java:304 - fsync(316) failed, errno (22) {}
com.sun.jna.LastErrorException: [22] i
at org.apache.cassandra.utils.CLibrary.fsync(Native Method) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.CLibrary.trySync(CLibrary.java:293) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.SyncUtil.trySync(SyncUtil.java:179) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.SyncUtil.trySyncDir(SyncUtil.java:190) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:117) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:135) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:150) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.<init>(BigTableWriter.java:375) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:84) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.switchCompactionLocation(DefaultCompactionWriter.java:64) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.maybeSwitchWriter(CompactionAwareWriter.java:128) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:108) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) [apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) [apache-cassandra-3.0.9.jar:3.0.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
ERROR [Reference-Reaper:1] 2018-05-09 12:10:14,363 Ref.java:223 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4a962079) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@2019607399:[Memory@[0..4e0), Memory@[0..30c0)] was not released before the reference was garbage collected
ERROR [CompactionExecutor:4] 2018-05-09 12:14:00,394 CassandraDaemon.java:205 - Exception in thread Thread[CompactionExecutor:4,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Channel not open for writing - cannot extend file to required size
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:142) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedRegions.extend(MmappedRegions.java:131) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.updateRegions(MmappedSegmentedFile.java:130) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:111) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.SegmentedFile$Builder.buildIndex(SegmentedFile.java:198) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:244) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.9.jar:3.0.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
Caused by: java.io.IOException: Channel not open for writing - cannot extend file to required size
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:901) ~[na:1.8.0_162]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.0.9.jar:3.0.9]
... 23 common frames omitted
ERROR [CompactionExecutor:4] 2018-05-09 12:14:00,394 StorageService.java:399 - Stopping native transport
INFO [CompactionExecutor:4] 2018-05-09 12:14:00,394 Server.java:180 - Stop listening for CQL clients
dmesg
输出了很多:
CIFS VFS: Send error in SETFSUnixInfo = -11
CIFS VFS: Send error in Close = -11
CIFS VFS: Unexpected lookup error -512
显然泄漏是Cassandra 3.0.9的一个已知问题,随Netty 4.0.23一起发布 - 但即使是内存泄漏问题,我已经重新启动了Cassandra,它仍然无法正常工作。 ..