Question

我有一个

的AWS基础架构

1个Kafka集群在3个docker容器上，在ECS上运行，使用EFS作为存储服务（为简单起见）。
1个Kafka Streams应用程序集群，位于3个容器上。

有一个源主题＆＃34;事件＆＃34;具有16个分区，复制2. PAPI拓扑处理器＆＃34;流处理器＆＃34;产生输出到一些其他主题并使用3个状态存储。

我可以通过Kafka Manager看到数据被消耗，并且输出产生于这些其他输出主题。显然它有效（虽然很慢）。

但是通过 bin / kafka-consumer-groups.sh 查看消费者抵消，我可以看到只有一个分区同时被消费。在命令的分开连续运行中，只有一个偏移减少。

首次执行：

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID
events          6          -               4021552         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          13         5030392         5030541         149             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          2          7056462         7056462         0               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          8          671945          6046546         5374601         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          1          164123          3009191         2845068         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          12         1962842         11052506        9089664         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          5          -               4022059         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          0          -               4019992         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          4          -               5032053         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          11         5037439         5037584         145             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          15         1683056         5034689         3351633         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          7          164702          7052434         6887732         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          14         -               3011069         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          3          1927601         6044400         4116799         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          10         5031461         5031612         151             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          9          1686979         8052924         6365945         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer

第二次执行：只有分区8提升了其偏移量。在1,5或15分钟后，那个是唯一消耗的分区。

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                          HOST            CLIENT-ID
events          6          -               4021552         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          13         5030392         5030541         149             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          2          7056462         7056462         0               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          8          686685          6046546         5359861         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          1          164123          3009191         2845068         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          12         1962842         11052506        9089664         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          5          -               4022059         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          0          -               4019992         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          4          -               5032053         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          11         5037439         5037584         145             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          15         1683056         5034689         3351633         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          7          164702          7052434         6887732         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          14         -               3011069         -               stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          3          1927601         6044400         4116799         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          10         5031461         5031612         151             stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer
events          9          1686979         8052924         6365945         stream-processor-fa36-StreamThread-1-consumer-6cc2 /same.ip.here  stream-processor-fa36-StreamThread-1-consumer

查看日志，只有一个实例同时打印日志。即如果一个人在工作，另外两个人不工作。

这可能是什么问题？

Kafka＆amp; Kafka Streams版本1.1。

Kafka Streams消费者群体并发：仅从一个分区消费？

0 个答案: