KafkaStreams - 在代理失败后恢复流

时间:2017-10-23 15:41:37

标签: java apache-kafka kafka-consumer-api kafka-producer-api apache-kafka-streams

我已经实现了一个具有以下属性的KafkaStreams应用

application.id = KafkaStreams
application.server = 
bootstrap.servers = [localhost:9092,localhost:9093]
buffered.records.per.partition = 1000
cache.max.bytes.buffering = 10485760
client.id = 
commit.interval.ms = 30000
connections.max.idle.ms = 540000
default.key.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
default.timestamp.extractor = class org.apache.kafka.streams.processor.FailOnInvalidTimestamp
default.value.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
key.serde = null
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
num.standby.replicas = 0
num.stream.threads = 1
partition.grouper = class org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 40000
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 600000
state.dir = /tmp/kafka-streams
timestamp.extractor = null
value.serde = null
windowstore.changelog.additional.retention.ms = 86400000
zookeeper.connect = 

我的kafka版本是0.11.0.1。我在localhost上分别推出了两个kafka经纪人:9092和9093。在两个代理中,default.replication.factor = 2和num.partitions = 4(其余配置属性是默认值)。

我的应用从特定主题接收流数据,进行一些转换并将数据发送回另一个主题。一旦第二个代理关闭,它就会停止接收打印以下内容的数据:

INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator localhost:9093 (id: 2147483646 rack: null) for group KafkaStreams.
[KafkaStreams-38259122-0ce7-41c3-8df6-7482626fec81-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the coordinator localhost:9093 (id: 2147483646 rack: null) dead for group KafkaStreams
[KafkaStreams-38259122-0ce7-41c3-8df6-7482626fec81-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator localhost:9093 (id: 2147483646 rack: null) for group KafkaStreams.
[KafkaStreams-38259122-0ce7-41c3-8df6-7482626fec81-StreamThread-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 2147483646 could not be established. Broker may not be available.
[kafka-coordinator-heartbeat-thread | KafkaStreams] WARN org.apache.kafka.clients.NetworkClient - Connection to node 1 could not be established. Broker may not be available.

由于某种原因,它不会重新平衡以连接到第一个代理。任何建议为什么会发生这种情况?

0 个答案:

没有答案