我们有一个超级账本应用程序。主应用程序托管在AWS VM上,而DR托管在Azure VM上。最近,Microsoft团队发现一个DR VM变得不可用,并且在大约8分钟内恢复了可用性。按照Microsoft的说法,“此意外事件是由Azure启动的自动恢复操作引起的。该自动恢复操作是由托管虚拟机的物理节点上的硬件问题触发的。按照设计,您的VM已自动移动到不同而健康的物理节点,以避免进一步的影响。” Zookeeper VM也已在同一位置重新部署
此事件发生后的第二天,我们开始注意到订购者脱机并在几秒钟后立即上线。在12小时10分钟的间隔后会定期发生这种断开/连接。
我们注意到了两件事
在日志中,我们得到
- [orderer/consensus/kafka] startThread -> CRIT 24df#033[0m [channel:
testchainid] Cannot set up channel consumer = kafka server: The
requested offset is outside the range of offsets maintained by the
server for the given topic/partition.
- panic: [channel: testchainid] Cannot set up channel consumer = kafka
server: The requested offset is outside the range of offsets
maintained by the server for the given topic/partition.
- goroutine 52 [running]:
- github.com/hyperledger/fabric/vendor/github.com/op/go-logging.(*Logger).Panicf(0xc4202748a0,
0x108dede, 0x31, 0xc420327540, 0x2, 0x2)
- /w/workspace/fabric-binaries-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/github.com/op/go-logging/logger.go:194
+0x134
- github.com/hyperledger/fabric/orderer/consensus/kafka.startThread(0xc42022cdc0)
- /w/workspace/fabric-binaries-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/kafka/chain.go:261
+0xb33
- created by
github.com/hyperledger/fabric/orderer/consensus/kafka.(*chainImpl).Start
- /w/workspace/fabric-binaries-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/kafka/chain.go:126
+0x3f
我们注意到的另一件事是,在VM故障事件之前的日志中,有3个kafka代理,但是在此事件之后,我们只能在日志中看到2个kafka代理。
有人可以指导我吗?我该如何解决这个问题?
其他信息-我们已经浏览了重新部署VM后的第二天的Kafka日志,并注意到了以下
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:132)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:231)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:192)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:528)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:469)
at org.apache.kafka.common.network.Selector.poll(Selector.java:398)
at kafka.network.Processor.poll(SocketServer.scala:535)
at kafka.network.Processor.run(SocketServer.scala:452)
at java.lang.Thread.run(Thread.java:748)
答案 0 :(得分:0)
似乎我们有解决方案,但需要进行验证。解决方案通过验证后,我会将其发布在此网站上。