如何避免在Kafka流中创建更改日志?

时间:2019-07-09 15:13:41

标签: apache-kafka apache-kafka-streams

我试图避免使用inMemoryWindowStore在Kafka流中创建更改日志主题(我正在使用Kafka 2.3.0和Streams DSL),我也在调用withLoggingDisabled(),但是在应用程序以某种方式调用时开始时,会创建changelog主题,并使用它们,因为我可以看到其中的数据。我究竟做错了什么?如何避免创建变更日志?

    WindowBytesStoreSupplier storeSupplier = Stores.inMemoryWindowStore("in-mem-store-" + index,
            Duration.ofSeconds(windowRetentionPeriodInSeconds),
            Duration.ofSeconds(aggregationWindowSizeInSeconds),
            false);

    myStream.filter((key, val) -> val!=null)
            .selectKey((key, val) -> val.getId())
            .groupByKey(Grouped.as("key-grouper").with(Serdes.String(), new MyDtoSerde()))
            .aggregate(MyDto::new,
                    new MyUpdater(),
                    Materialized.as(storeSupplier)
                            .withCachingDisabled()
                            .withLoggingDisabled()
                            .with(Serdes.String(), new MyDtoSerde()))

1 个答案:

答案 0 :(得分:0)

如Bill Bejeck here所述,在2.3.0中使用Materialized的静态方法有点棘手。

我已经通过这种方式解决了这个问题:

    Materialized<String, MyDto, WindowStore<Bytes, byte[]>> materialized;
    materialized = Materialized.with(Serdes.String(), new MyDtoSerde());
    if (withLoggingDisabled) {
        materialized.withLoggingDisabled();
    }

    myStream.filter((key, val) -> val!=null)
        .selectKey((key, val) -> val.getId())
        .groupByKey(Grouped.as("key-grouper").with(Serdes.String(), new MyDtoSerde()))
        .windowedBy(TimeWindows.of(Duration.ofSeconds(aggregationWindowSizeInSeconds))
                   .grace(Duration.ofSeconds(windowRetentionPeriodInSeconds)))
        .aggregate(MyDto::new,
                   new MyUpdater(),
                   materialized)