Question

我有一个带有12个分区的Kafka主题gbtopic和一个消费者组gbconsumers，其中三个消费者C1，C2，C3订阅了该主题。启动所有使用者后，将为每个使用者分配4个分区。然后我停止了消费者C3。现在，将分区重新平衡为6个分区，分别分配给C1和C2。现在，我启动了使用者C3，并重新分配了分区并将其分配给C1，C2，C3。我的期望是在停止C3之前分配与C1，C2，C3相同的分区集。

我正在将StickyAssignor用于partition.assignment.strategy。我按以下顺序执行操作：

Start C1
Start C2
Start C3
Stop C3
Restart C3

以下是我的客户端程序：

public class ConsumerApp {
    public static void main(String[] args) {
        Scanner scanIn = new Scanner(System.in);
        System.out.print("Enter Consumer Name :: ");
        String consumerId = scanIn.nextLine().trim();
        scanIn.close();

        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "gbconsumers");
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, StickyAssignor.class.getName());
        props.put(ConsumerConfig.CLIENT_ID_CONFIG, consumerId);

        KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(props);
        ConsumerRebalanceListener listener = new ConsumerRebalanceListener() {
            @Override
            public void onPartitionsRevoked(Collection<TopicPartition> collection) {
                System.out.println("-------------------------------------------------------------------------------------");
                System.out.println(consumerId + " :: onPartitionsRevoked :: " + collection);
            }

            @Override
            public void onPartitionsAssigned(Collection<TopicPartition> collection) {
                System.out.println(consumerId + " :: onPartitionsAssigned :: " + collection + "\n");
            }
        };
        kafkaConsumer.subscribe(Collections.singletonList("gbtopic"), listener);
        while (true) {
            ConsumerRecords<String, String> records = kafkaConsumer.poll((100));
            for (ConsumerRecord<String, String> record : records) {
                performSomeOperation(record);
            }
        }
    }

    private static void performSomeOperation(ConsumerRecord<String, String> record) {
        // operations
    }
}

在停止C3之前，所有三个使用者都在运行时输出


    C1 :: onPartitionsAssigned :: [gbtopic-10, gbtopic-11, gbtopic-8, gbtopic-9]
    C2 :: onPartitionsAssigned :: [gbtopic-2, gbtopic-3, gbtopic-4, gbtopic-5]
    C3 :: onPartitionsAssigned :: [gbtopic-0, gbtopic-1, gbtopic-6, gbtopic-7]

停止并启动C3之后的输出：


    C1 :: onPartitionsAssigned :: [gbtopic-0, gbtopic-10, gbtopic-11, gbtopic-6]
    C2 :: onPartitionsAssigned :: [gbtopic-4, gbtopic-5, gbtopic-1, gbtopic-7]
    C3 :: onPartitionsAssigned :: [gbtopic-2, gbtopic-3, gbtopic-8, gbtopic-9]

我希望在停止/重新启动C3之后分配给每个使用者的分区与停止/重新启动之前的分区相同。我是否缺少任何财产，还是有其他方法可以实现这一目标？

Kafka：重新平衡后如何为消费者提供以前的分区

0 个答案: