我有13个节点的Cassandra集群,现在尝试添加第14个节点。 13个节点中的每一个都保存大约1.5 TB的数据,现在当我启动新节点时,引导过程开始没有任何问题,新节点开始获得大约1Gbit的数据。但是经过一段时间后,它只是在日志文件中没有任何错误,nodetool netstats显示
/data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701 bytes(27%) received from idx:0/192.168.220.16
/data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671 bytes(38%) received from idx:0/192.168.220.22
/data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398 bytes(48%) received from idx:0/192.168.220.12
/data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610 bytes(85%) received from idx:0/192.168.220.18
/data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085 bytes(48%) received from idx:0/192.168.220.20
看起来它看起来应该看起来,但目标系统上不存在/data/XXX/XXXX/tmp-la-XX-big Chall.db文件,没有网络活动。 Boostrap暂停,节点未加入群集,它在UJ上的状态为stales。
我不时会在日志文件中看到这一点:
WARN [STREAM-IN-/192.168.220.15] 2015-09-04 04:34:20,922 StreamSession.java:638 - [Stream #2bc66b80-52cf-11e5-bec6-75928743ca32] Retrying for following error
org.apache.cassandra.serializers.MarshalException: String didn't validate.
at org.apache.cassandra.serializers.UTF8Serializer.validate(UTF8Serializer.java:35) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:91) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.cql3.ColumnIdentifier.<init>(ColumnIdentifier.java:58) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.composites.SimpleSparseCellNameType.fromByteBuffer(SimpleSparseCellNameType.java:83) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.2.0.jar:2.2.0]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:162) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) [apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56) [apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
ERROR [STREAM-IN-/192.168.220.15] 2015-09-04 04:34:20,927 StreamSession.java:518 - [Stream #2bc66b80-52cf-11e5-bec6-75928743ca32] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:90) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
INFO [STREAM-IN-/192.168.220.15] 2015-09-04 04:34:22,048 StreamResultFuture.java:182 - [Stream #2bc66b80-52cf-11e5-bec6-75928743ca32] Session with /192.168.220.15 is complete
ERROR [Thread-12048] 2015-09-04 04:34:22,063 CassandraDaemon.java:182 - Exception in thread Thread[Thread-12048,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79]
Caused by: java.nio.channels.AsynchronousCloseException: null
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205) ~[na:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412) ~[na:1.7.0_79]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.7.0_79]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.7.0_79]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.7.0_79]
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:161) ~[apache-cassandra-2.2.0.jar:2.2.0]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.0.jar:2.2.0]
... 1 common frames omitted
但我可以看到此异常的服务器不在staled流服务器列表中。
我试图重新启动节点,它开始自举,但再次卡住 我们有 : Cassandra 2.2.0 Datastx社区,Debian 7 64位,HotSpot JDK 1.7.0_79。
尝试在源节点和目标节点,修复的表上的引导期间禁用自动编译,但最终会出现同样的问题。