如何在属性文件中正确外部化spring-boot kafka-streams配置?

时间:2019-06-13 00:52:15

标签: spring-boot apache-kafka-streams spring-kafka

我正在尝试将我目前用Java代码编写的spring-kafka应用程序的配置外部化。 我应该将ProducerConfigConsumerConfig值放到spring.kafka.streams.properties中,还是如果我通过spring.kafka.producerspring.kafka.consumer提供它们,是否可以正确配置它们?

到目前为止,看来我应该将所有配置都放入KafkaStreamsConfiguration类型的Bean中,以便配置我的kafka-streams应用程序。目前,我是通过直接在代码中设置ProducerConfigConsumerConfig值来做到这一点的。

当我外部化此配置时,似乎在ProducerConfig文件中的ConsumerConfigapplication.properties中设置属性值与在创建的KafkaStreamsConfiguration中的属性值不相关通过spring-boot(我通过在某处自动装配配置并进行了观察来确认这一点)。

如果我改为通过ProducerConfig提供ConsumerConfigspring.kafka.streams.properties值,它们将显示在KafkaStreamsConfiguration中。

这是我以前的Java配置:

@Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
    public KafkaStreamsConfiguration kStreamsConfigs() {
        Map<String, Object> props = new HashMap<>();

        props.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        props.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, WallclockTimestampExtractor.class.getName());
        props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, commitInterval);
        props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, registryUrl);
        props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, LogAndContinueExceptionHandler.class.getName());
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
        props.put("replication.factor", replicationFactor);
        props.put(StreamsConfig.STATE_DIR_CONFIG, "/var/lib/kafka-streams");
        props.put(StreamsConfig.STATE_CLEANUP_DELAY_MS_CONFIG, "600000");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

        return new KafkaStreamsConfiguration(props);
    }

在运行时,结果为ProducerConfigConsumerConfig值不在KafkaStreamsConfiguration

spring.kafka.streams.bootstrap-servers=localhost:9092
spring.kafka.streams.properties.schema.registry.url=http://localhost:8081
spring.kafka.streams.application-id=<application_id>
spring.kafka.consumer.group-id=<group_id> #this won't show up in KafkaStreamsConfiguration
spring.kafka.streams.replication-factor=1
spring.kafka.streams.properties.commit.interval.ms=100
spring.kafka.streams.properties.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
spring.kafka.streams.properties.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
spring.kafka.streams.properties.default.deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
spring.kafka.producer.compression-type=lz4 #this won't show up in KafkaStreamsConfiguration
spring.kafka.streams.properties.state.dir=/var/lib/kafka-streams
spring.kafka.streams.properties.state.cleanup.delay.ms=600000
spring.kafka.consumer.auto-offset-reset=latest #this won't show up in KafkaStreamsConfiguration
spring.kafka.streams.properties.timestamp.extractor=org.apache.kafka.streams.processor.WallclockTimestampExtractor

但是,这确实导致KafkaStreamsConfiguration具有期望的值:

spring.kafka.streams.bootstrap-servers=localhost:9092
spring.kafka.streams.properties.schema.registry.url=http://localhost:8081
spring.kafka.streams.application-id=<application_id>
spring.kafka.streams.properties.group-id=<group_id> #this shows up in KafkaStreamsConfiguration
spring.kafka.streams.replication-factor=1
spring.kafka.streams.properties.commit.interval.ms=100
spring.kafka.streams.properties.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
spring.kafka.streams.properties.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
spring.kafka.streams.properties.default.deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
spring.kafka.streams.properties.compression-type=lz4 #this shows up in KafkaStreamsConfiguration
spring.kafka.streams.properties.state.dir=/var/lib/kafka-streams
spring.kafka.streams.properties.state.cleanup.delay.ms=600000
spring.kafka.streams.properties.auto-offset-reset=latest #this shows up in KafkaStreamsConfiguration
spring.kafka.streams.properties.timestamp.extractor=org.apache.kafka.streams.processor.WallclockTimestampExtractor

我期望分别通过ProducerConfigConsumerConfig进行设置时,KafkaStreamsConfigurationspring.kafka.producer的值会传播到spring.kafka.consumer。尤其是因为我在IntelliJ中获得了application.properties中生产者和消费者配置的Intellisense。

也就是说,我是否需要确保通过spring.kafka.streams.properties进行设置,以便正确配置该应用?

1 个答案:

答案 0 :(得分:0)

  

spring.kafka.consumer.group-id=<group_id> #this won't show up in KafkaStreamsConfiguration

流将group.id设置为application.id属性。

  

公共静态最终字符串APPLICATION_ID_CONFIG =“ application.id”;

     

private static final String APPLICATION_ID_DOC =“流处理应用程序的标识符。在Kafka集群中必须是唯一的。它用作1)默认客户端ID前缀,2)成员资格管理的组ID,3 )changelog主题前缀。“;

请参见KafkaProperties

streamsproducerconsumer属性是互不相关的。

  

spring.kafka.producer.compression-type=lz4 #this won't show up in KafkaStreamsConfiguration

compression-type未公开为流的一流引导属性。您可以使用

进行设置
spring.kafka.streams.properties.compression.type=gzip