我在Cassandra 3.10中遇到问题，当我尝试加入/停用节点时，它无法以“流失败”异常加入/停用集群，然后它停止运行。在这段时间内，节点保持在JOINING状态。
版本：cqlsh 5.0.1 |卡桑德拉3.10 | CQL规范3.4.4 |原生协议v4
节点1错误日志（种子）
INFO [STREAM-INIT-/10.0.120.11:44986] 2019-03-04 02:25:13,844 StreamResultFuture.java:116 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f ID#0] Creating new streaming plan for Bootstrap
INFO [STREAM-INIT-/10.0.120.11:44986] 2019-03-04 02:25:13,861 StreamResultFuture.java:123 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f, ID#0] Received streaming plan for Bootstrap
INFO [STREAM-INIT-/10.0.120.11:44988] 2019-03-04 02:25:13,861 StreamResultFuture.java:123 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f, ID#0] Received streaming plan for Bootstrap
INFO [STREAM-IN-/10.0.120.11:44988] 2019-03-04 02:25:14,130 StreamResultFuture.java:173 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f ID#0] Prepare completed. Receiving 0 files(0.000KiB), sending 154 files(64.507MiB)
ERROR [STREAM-IN-/10.0.120.11:44988] 2019-03-04 02:25:14,401 StreamSession.java:706 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Remote peer 10.0.120.11 failed stream session.
INFO [STREAM-IN-/10.0.120.11:44988] 2019-03-04 02:25:14,415 StreamResultFuture.java:187 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Session with /10.0.120.11 is complete
WARN [STREAM-IN-/10.0.120.11:44988EHgks ] 2019-03-04 02:25:14,417 StreamResultFuture.java:214 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Stream failed
ERROR [STREAM-OUT-/10.0.120.11:44986] 2019-03-04 02:25:14,418 StreamSession.java:593 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Streaming error occurred on session with peer 10.0.120.11
org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:145) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.compress.CompressedStreamWriter.lambda$write$0(CompressedStreamWriter.java:85) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:350) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:85) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:101) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:52) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:50) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:408) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:380) ~[apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_152]
at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) ~[na:1.8.0_152]
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) ~[na:1.8.0_152]
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) ~[na:1.8.0_152]
at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:141) ~[apache-cassandra-3.10.jar:3.10]
... 10 common frames omitted
节点2错误日志
INFO [main] 2019-03-04 02:25:12,424 StorageService.java:1435 - JOINING: Starting to bootstrap...
INFO [main] 2019-03-04 02:25:13,832 StreamResultFuture.java:90 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Executing streaming plan for Bootstrap
INFO [StreamConnectionEstablisher:1] 2019-03-04 02:25:13,837 StreamSession.java:266 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Starting streaming to /10.0.110.11
INFO [StreamConnectionEstablisher:1] 2019-03-04 02:25:13,842 StreamCoordinator.java:264 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f, ID#0] Beginning stream session with /10.0.110.11
INFO [STREAM-IN-/10.0.110.11:5000] 2019-03-04 02:25:14,138 StreamResultFuture.java:173 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f ID#0] Prepare completed. Receiving 154 files(64.507MiB), sending 0 files(0.000KiB)
INFO [StreamReceiveTask:1] 2019-03-04 02:25:14,376 SecondaryIndexManager.java:365 - Submitting index build of pushcapabilityindx,presetsearchval for data in BigTableReader(path='/cassandra/data/abc_sub_db/sub-94fe59103afb11e9a042932fa01bd6f1/mc-1-big-Data.db'),BigTableReader(path='/cassandra/data/abc_sub_db/sub-94fe59103afb11e9a042932fa01bd6f1/mc-2-big-Data.db')
ERROR [StreamReceiveTask:1] 2019-03-04 02:25:14,400 StreamSession.java:593 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Streaming error occurred on session with peer 10.0.110.11
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.NoSuchElementException
at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:51) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:393) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:382) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:269) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:215) ~[apache-cassandra-3.10.jar:3.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Caused by: java.util.concurrent.ExecutionException: java.util.NoSuchElementException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_152]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) [na:1.8.0_152]
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:386) ~[apache-cassandra-3.10.jar:3.10]
... 9 common frames omitted
Caused by: java.util.NoSuchElementException: null
at org.apache.cassandra.utils.AbstractIterator.next(AbstractIterator.java:64) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.index.SecondaryIndexManager.lambda$indexPartition$20(SecondaryIndexManager.java:618) ~[apache-cassandra-3.10.jar:3.10]
at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_152]
at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:618) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:71) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1587) ~[apache-cassandra-3.10.jar:3.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_152]
... 6 common frames omitted
ERROR [STREAM-IN-/10.0.110.11:5000] 2019-03-04 02:25:14,407 StreamSession.java:593 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Streaming error occurred on session with peer 10.0.110.11
java.lang.RuntimeException: Outgoing stream handler has been closed
at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:143) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:655) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:523) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317) ~[apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
INFO [StreamReceiveTask:1] 2019-03-04 02:25:14,409 StreamResultFuture.java:187 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Session with /10.0.110.11 is complete
WARN [StreamReceiveTask:1] 2019-03-04 02:25:14,411 StreamResultFuture.java:214 - [Stream #bd205bb0-3e24-11e9-9d30-fb2ced1ba79f] Stream failed
ERROR [main] 2019-03-04 02:25:14,412 StorageService.java:1507 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1502) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:962) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:681) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:394) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735) [apache-cassandra-3.10.jar:3.10]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:571) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:249) ~[apache-cassandra-3.10.jar:3.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_152]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) ~[apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
WARN [StreamReceiveTask:1] 2019-03-04 02:25:14,412 StorageService.java:1497 - Error during bootstrap.
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) [guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) [guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) [guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:571) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:249) [apache-cassandra-3.10.jar:3.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
WARN [main] 2019-03-04 02:25:14,420 StorageService.java:1013 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2019-03-04 02:25:14,420 CassandraDaemon.java:694 - Waiting for gossip to settle before accepting client requests...
INFO [main] 2019-03-04 02:25:22,421 CassandraDaemon.java:725 - No gossip backlog; proceeding
INFO [main] 2019-03-04 02:25:22,482 NativeTransportService.java:70 - Netty using native Epoll event loop
INFO [main] 2019-03-04 02:25:22,531 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.39.Final.38bdf86, netty-codec=netty-codec-4.0.39.Final.38bdf86, netty-codec-haproxy=netty-codec-haproxy-4.0.39.Final.38bdf86, netty-codec-http=netty-codec-http-4.0.39.Final.38bdf86, netty-codec-socks=netty-codec-socks-4.0.39.Final.38bdf86, netty-common=netty-common-4.0.39.Final.38bdf86, netty-handler=netty-handler-4.0.39.Final.38bdf86, netty-tcnative=netty-tcnative-1.1.33.Fork19.fe4816e, netty-transport=netty-transport-4.0.39.Final.38bdf86, netty-transport-native-epoll=netty-transport-native-epoll-4.0.39.Final.38bdf86, netty-transport-rxtx=netty-transport-rxtx-4.0.39.Final.38bdf86, netty-transport-sctp=netty-transport-sctp-4.0.39.Final.38bdf86, netty-transport-udt=netty-transport-udt-4.0.39.Final.38bdf86]
INFO [main] 2019-03-04 02:25:22,531 Server.java:156 - Starting listening for CQL clients on /10.0.120.11:7042 (unencrypted)...
INFO [main] 2019-03-04 02:25:22,564 CassandraDaemon.java:528 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
在以下情况下成功。
案例1。
删除索引>节点加入/停用成功>创建索引>节点加入/停用失败
案例2。
表数据备份（复制到命令）>截断表>数据恢复（从命令复制）>节点加入/停用成功>从服务中插入新数据>节点加入/停用失败
案例3。
表数据备份（复制到命令）>删除表>创建表>数据恢复（从命令复制）>节点加入/停用成功>从服务中插入新数据>节点加入/停用失败
不幸的是，我不知道该服务中的新插入查询。
Q1。我想看看哪些查询进入了Cassandra。我可以检查服务会话中的查询吗？
Q2。您是否知道为什么流在Cassandra中失败？
答案 0 :(得分：0)
Q1。启用来自nodetool setlogginglevels TRACE的日志记录级别的跟踪以进行cassandra传输。您可以在system.log上查询。
Q2。通常，此错误是由于节点之间的网络问题而引起的，而没有发生流数据或八卦。