卡夫卡消费者中有一半属于消费者群体

时间:2019-09-06 15:45:52

标签: scala apache-kafka akka alpakka

我遇到这个问题,我在这里说

  • 4个消费者属于同一组,其中只有2个消费者从代理那里获取消息。
  • 6个属于同一组的消费者,其中只有3个从经纪人那里收到消息

我有3个代理,复制因子为1。

在挂起的消费者中,我得到以下日志,该日志随着时间的推移几乎没有进展,最后一行相同:

15:01:45.255 [default-akka.kafka.default-dispatcher-6] INFO  o.a.k.c.consumer.ConsumerConfig - ConsumerConfig values: 
    auto.commit.interval.ms = 5000
    auto.offset.reset = earliest
    bootstrap.servers = [dev-cp-kafka-headless:9092]
    check.crcs = true
    client.dns.lookup = default
    client.id = 
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = false
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 10000
    fetch.min.bytes = 1
    group.id = elsevier-triple-store-builder-26
    heartbeat.interval.ms = 60000
    interceptor.classes = []
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 5000000
    max.poll.records = 10000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 300000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

15:01:45.397 [default-akka.kafka.default-dispatcher-6] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version : 2.1.1
15:01:45.397 [default-akka.kafka.default-dispatcher-6] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId : 21234bee31165527
15:01:46.133 [default-akka.kafka.default-dispatcher-22] INFO  org.apache.kafka.clients.Metadata - Cluster ID: cLqnPKhKRPuZtkhAFpcaow
15:01:46.201 [default-akka.kafka.default-dispatcher-8] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Discovered group coordinator dev-cp-kafka-2.dev-cp-kafka-headless.dev:9092 (id: 2147483645 rack: null)
15:01:46.872 [default-akka.kafka.default-dispatcher-10] INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Revoking previously assigned partitions []
15:01:46.874 [default-akka.kafka.default-dispatcher-10] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] (Re-)joining group
15:06:35.500 [default-akka.kafka.default-dispatcher-14] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Successfully joined group with generation 339
15:06:35.641 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Setting newly assigned partitions [elsevier-proxy-reaxys-document-2, elsevier-proxy-reaxys-protein-4, elsevier-proxy-reaxys-bioactivity-0, elsevier-proxy-reaxys-parameter-0, elsevier-proxy-reaxys-higestclinicalphase-4, elsevier-proxy-reaxys-chemicalentityname-2, elsevier-proxy-reaxys-proteinassembly-3, elsevier-proxy-reaxys-administrationroute-1, elsevier-proxy-reaxys-compoundtype-1, elsevier-proxy-reaxys-bioassay-1, elsevier-proxy-reaxys-putativeactiontarget-4, elsevier-proxy-reaxys-tissuetype-2, elsevier-proxy-reaxys-reactiontype-1, elsevier-proxy-reaxys-agentconfiguration-2, elsevier-proxy-reaxys-biologicalmaterial-2, elsevier-proxy-reaxys-document-1, elsevier-proxy-reaxys-parameter-1, elsevier-proxy-reaxys-proteinsubunit-0, elsevier-proxy-reaxys-bioassay-0, elsevier-proxy-reaxys-protein-3, elsevier-proxy-reaxys-chemicalentityname-3, elsevier-proxy-reaxys-bioactivity-1, elsevier-proxy-reaxys-administrationroute-0, elsevier-proxy-reaxys-proteinassembly-2, elsevier-proxy-reaxys-compoundtype-2, elsevier-proxy-reaxys-measuresettings-0, elsevier-proxy-reaxys-putativeactiontarget-3, elsevier-proxy-reaxys-tissuetype-1, elsevier-proxy-reaxys-higestclinicalphase-3, elsevier-proxy-reaxys-agentconfiguration-1, elsevier-proxy-reaxys-reactiontype-2, elsevier-proxy-reaxys-biologicalmaterial-1, elsevier-proxy-reaxys-chemicalentityname-4, elsevier-proxy-reaxys-protein-2, elsevier-proxy-reaxys-parameter-2, elsevier-proxy-reaxys-celltype-0, elsevier-proxy-reaxys-bioactivity-2, elsevier-proxy-fda-spl-document-0, elsevier-proxy-reaxys-metabolicreaction-1, elsevier-proxy-reaxys-administrationroute-3, elsevier-proxy-reaxys-proteinsubunit-1, elsevier-proxy-reaxys-organism-3, elsevier-proxy-reaxys-tissuetype-4, elsevier-proxy-reaxys-formulainformation-0, elsevier-proxy-reaxys-chemicalentity-1, elsevier-proxy-reaxys-bioactivity-3, elsevier-proxy-reaxys-agentconfiguration-4, elsevier-proxy-reaxys-biologicalmaterial-4, elsevier-proxy-reaxys-proteinnature-0, elsevier-proxy-reaxys-agentrole-0, elsevier-proxy-reaxys-higestclinicalphase-2, elsevier-proxy-reaxys-document-4, elsevier-proxy-reaxys-parameter-3, elsevier-proxy-reaxys-protein-1, elsevier-proxy-reaxys-metabolicreaction-0, elsevier-proxy-reaxys-proteinassembly-4, elsevier-proxy-reaxys-administrationroute-2, elsevier-proxy-reaxys-compoundtype-0, elsevier-proxy-reaxys-proteinsubunit-2, elsevier-proxy-reaxys-organism-4, elsevier-proxy-reaxys-tissuetype-3, elsevier-proxy-reaxys-formulainformation-1, elsevier-proxy-reaxys-bioactivity-4, elsevier-proxy-reaxys-higestclinicalphase-1, elsevier-proxy-reaxys-agentconfiguration-3, elsevier-proxy-reaxys-proteinnature-1, elsevier-proxy-reaxys-reactiontype-0, elsevier-proxy-reaxys-biologicalmaterial-3, elsevier-proxy-reaxys-document-3, elsevier-proxy-reaxys-chemicalentity-0, elsevier-proxy-reaxys-protein-0, elsevier-proxy-reaxys-parameter-4, elsevier-proxy-reaxys-celltype-2, elsevier-proxy-reaxys-organism-1, elsevier-proxy-reaxys-metabolicreaction-3, elsevier-proxy-reaxys-measuresettings-3, elsevier-proxy-reaxys-proteinsubunit-3, elsevier-proxy-reaxys-formulainformation-2, elsevier-proxy-reaxys-putativeactiontarget-0, elsevier-proxy-reaxys-agentrole-2, elsevier-proxy-reaxys-higestclinicalphase-0, elsevier-proxy-reaxys-chemicalentity-3, elsevier-proxy-reaxys-proteinnature-2, elsevier-proxy-reaxys-celltype-1, elsevier-proxy-reaxys-metabolicreaction-2, elsevier-proxy-reaxys-measuresettings-4, elsevier-proxy-reaxys-administrationroute-4, elsevier-proxy-reaxys-proteinsubunit-4, elsevier-proxy-reaxys-organism-2, elsevier-proxy-reaxys-bioassay-4, elsevier-proxy-reaxys-formulainformation-3, elsevier-proxy-reaxys-chemicalentity-2, elsevier-proxy-reaxys-agentrole-1, elsevier-proxy-reaxys-proteinnature-3, elsevier-proxy-reaxys-document-0, elsevier-proxy-reaxys-chemicalentityname-0, elsevier-proxy-reaxys-measuresettings-1, elsevier-proxy-reaxys-proteinassembly-1, elsevier-proxy-reaxys-compoundtype-3, elsevier-proxy-reaxys-bioassay-3, elsevier-proxy-reaxys-putativeactiontarget-2, elsevier-proxy-reaxys-formulainformation-4, elsevier-proxy-reaxys-tissuetype-0, elsevier-proxy-reaxys-reactiontype-3, elsevier-proxy-reaxys-agentconfiguration-0, elsevier-proxy-reaxys-biologicalmaterial-0, elsevier-proxy-reaxys-proteinnature-4, elsevier-proxy-reaxys-agentrole-4, elsevier-proxy-reaxys-celltype-4, elsevier-proxy-reaxys-celltype-3, elsevier-proxy-reaxys-organism-0, elsevier-proxy-reaxys-chemicalentityname-1, elsevier-proxy-reaxys-metabolicreaction-4, elsevier-proxy-reaxys-measuresettings-2, elsevier-proxy-reaxys-proteinassembly-0, elsevier-proxy-reaxys-bioassay-2, elsevier-proxy-reaxys-compoundtype-4, elsevier-proxy-reaxys-putativeactiontarget-1, elsevier-proxy-reaxys-reactiontype-4, elsevier-proxy-reaxys-chemicalentity-4, elsevier-proxy-reaxys-agentrole-3]
15:06:35.653 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-putativeactiontarget-3 to offset 0.
15:06:35.653 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-compoundtype-0 to offset 0.
15:06:35.653 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-higestclinicalphase-2 to offset 0.
15:06:35.653 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-compoundtype-3 to offset 0.
15:06:35.653 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-proteinnature-2 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-higestclinicalphase-4 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-proteinnature-1 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-fda-spl-document-0 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-compoundtype-2 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-proteinnature-4 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-putativeactiontarget-4 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-proteinnature-0 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-higestclinicalphase-3 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-compoundtype-1 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-proteinnature-3 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-compoundtype-4 to offset 0.
15:06:35.657 [default-akka.kafka.default-dispatcher-19] INFO  o.a.k.c.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Resetting offset for partition elsevier-proxy-reaxys-agentrole-3 to offset 0.
15:11:26.880 [default-akka.kafka.default-dispatcher-17] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 0 was unable to process the fetch request with (sessionId=2019035881, epoch=29): FETCH_SESSION_ID_NOT_FOUND.
15:11:36.913 [default-akka.kafka.default-dispatcher-8] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 2 was unable to process the fetch request with (sessionId=295384863, epoch=30): FETCH_SESSION_ID_NOT_FOUND.
15:11:36.913 [default-akka.kafka.default-dispatcher-8] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 1 was unable to process the fetch request with (sessionId=1568797267, epoch=30): FETCH_SESSION_ID_NOT_FOUND.
15:16:08.090 [default-akka.kafka.default-dispatcher-17] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 0 was unable to process the fetch request with (sessionId=424717464, epoch=28): FETCH_SESSION_ID_NOT_FOUND.
15:16:38.122 [default-akka.kafka.default-dispatcher-9] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 2 was unable to process the fetch request with (sessionId=1878455433, epoch=30): FETCH_SESSION_ID_NOT_FOUND.
15:16:38.123 [default-akka.kafka.default-dispatcher-9] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 1 was unable to process the fetch request with (sessionId=1382963023, epoch=30): FETCH_SESSION_ID_NOT_FOUND.
15:20:38.890 [default-akka.kafka.default-dispatcher-13] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 0 was unable to process the fetch request with (sessionId=861824997, epoch=27): FETCH_SESSION_ID_NOT_FOUND.
15:21:39.033 [default-akka.kafka.default-dispatcher-11] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 2 was unable to process the fetch request with (sessionId=2058171454, epoch=30): FETCH_SESSION_ID_NOT_FOUND.
15:21:39.034 [default-akka.kafka.default-dispatcher-11] INFO  o.a.k.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=elsevier-triple-store-builder-26] Node 1 was unable to process the fetch request with (sessionId=750170814, epoch=30): FETCH_SESSION_ID_NOT_FOUND.


我不知道问题来自哪里,甚至不知道从哪里开始。完全披露我正在使用Akka-stream Kafka(alpakka)版本1.0.5,以及客户端2.1.1,而我的经纪人的版本是2.3.0。

请注意,我的主题每个都有30个分区。我要同时订阅多个主题。

0 个答案:

没有答案