Question

我以json字符串生成输入数据。

主题 - myinput

{"time":"2017-11-28T09:42:26.776Z","name":"Lane1","oclass"
     :"myClass","id":112553,"Scope":"198S"}

我的班级看起来像这样：

public class App {
    static public class CountryMessage {

        public String time;
        public String Scope;
        public String id;
        public String oclass;
        public String name; 
    }

    private static final String APP_ID = "countries-streaming-analysis-app";

    public static void main(String[] args) {
        System.out.println("Kafka Streams Demonstration");


        StreamsConfig config = new StreamsConfig(getProperties());
        final Serde < String > stringSerde = Serdes.String();
        final Serde < Long > longSerde = Serdes.Long();

        Map < String, Object > serdeProps = new HashMap < > ();
        final Serializer < CountryMessage > countryMessageSerializer = new JsonPOJOSerializer < > ();
        serdeProps.put("JsonPOJOClass", CountryMessage.class);
        countryMessageSerializer.configure(serdeProps, false);

        final Deserializer < CountryMessage > countryMessageDeserializer = new JsonPOJODeserializer < > ();
        serdeProps.put("JsonPOJOClass", CountryMessage.class);
        countryMessageDeserializer.configure(serdeProps, false);
        final Serde < CountryMessage > countryMessageSerde = Serdes.serdeFrom(countryMessageSerializer,countryMessageDeserializer);

        KStreamBuilder kStreamBuilder = new KStreamBuilder();
        KStream<String, CountryMessage> countriesStream = kStreamBuilder.stream(stringSerde, countryMessageSerde, "vanitopic");

        KGroupedStream<String, CountryMessage> countries = countriesStream.selectKey((k, traffic) -> traffic.Scope).groupByKey();

        KTable<Windowed<String>, Long> aggregatedStream = countries.count(TimeWindows.of(60 * 1000L), "UserCountStore");

        System.out.println("Starting Kafka Streams Countries Example");
        KafkaStreams kafkaStreams = new KafkaStreams(kStreamBuilder, config);
        kafkaStreams.start();
        System.out.println("Now started CountriesStreams Example");
    }

    private static Properties getProperties() {
        Properties settings = new Properties();

        settings.put(StreamsConfig.APPLICATION_ID_CONFIG, APP_ID);
        settings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "10.106.9.235:9092,10.106.9.235:9093,10.106.9.235:9094");
        settings.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "10.106.9.235:2181");
        //settings.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        //settings.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        return settings;
    }
}

我得到轰鸣声阶级演员异常：

线程中的异常 “国家串流分析-APP-f7f95119-4401-4a6e-8060-5a138ffaddb2-StreamThread-1” org.apache.kafka.streams.errors.StreamsException：陷入异常处理。 taskId = 0_0，processor = KSTREAM-SOURCE-0000000000， topic = vanitopic，partition = 0，offset = 2036 at org.apache.kafka.streams.processor.internals.StreamTask.process（StreamTask.java:203）在 org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate（StreamThread.java:679）在 org.apache.kafka.streams.processor.internals.StreamThread.runLoop（StreamThread.java:557）在 org.apache.kafka.streams.processor.internals.StreamThread.run（StreamThread.java:527）引起：org.apache.kafka.streams.errors.StreamsException：A 序列化器（关键： org.apache.kafka.common.serialization.ByteArraySerializer / value： org.apache.kafka.common.serialization.ByteArraySerializer）不是兼容实际的键或值类型（键类型：java.lang.String / value type：com.cisco.streams.countries.App $ CountryMessage）。更改 StreamConfig中的默认Serdes或提供正确的Serdes via 方法参数。在 org.apache.kafka.streams.processor.internals.SinkNode.process（SinkNode.java:91）在 org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward（ProcessorContextImpl.java:82）在 org.apache.kafka.streams.kstream.internals.KStreamFilter $ KStreamFilterProcessor.process（KStreamFilter.java:43）在 org.apache.kafka.streams.processor.internals.ProcessorNode $ 1.run（ProcessorNode.java:47）在 org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs（StreamsMetricsImpl.java:187）在 org.apache.kafka.streams.processor.internals.ProcessorNode.process（ProcessorNode.java:133）在 org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward（ProcessorContextImpl.java:82）在 org.apache.kafka.streams.kstream.internals.KStreamMap $ KStreamMapProcessor.process（KStreamMap.java:42）在 org.apache.kafka.streams.processor.internals.ProcessorNode $ 1.run（ProcessorNode.java:47）在 org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs（StreamsMetricsImpl.java:187）在 org.apache.kafka.streams.processor.internals.ProcessorNode.process（ProcessorNode.java:133）在 org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward（ProcessorContextImpl.java:82）在 org.apache.kafka.streams.processor.internals.SourceNode.process（SourceNode.java:80）在 org.apache.kafka.streams.processor.internals.StreamTask.process（StreamTask.java:189） ... 3更多引起：java.lang.ClassCastException：java.lang.String 不能被投到[B at org.apache.kafka.common.serialization.ByteArraySerializer.serialize（ByteArraySerializer.java:21）在 org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send（RecordCollectorImpl.java:88）在 org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send（RecordCollectorImpl.java:76）在 org.apache.kafka.streams.processor.internals.SinkNode.process（SinkNode.java:87） ......还有16个

我需要帮助来了解如何以及在何处应用我创建的自定义Serdes

Answer 1

将序列化器添加到 groupByKey

KGroupedStream<String, CountryMessage> countries = countriesStream.selectKey((k, traffic) -> traffic.Scope).groupByKey(Grouped.with(Serdes.String(), new ObjectMapperSerde<>(CountryMessage.class)));

Answer 2

在您的代码中，

KGroupedStream<String, CountryMessage> countries = countriesStream.selectKey((k, traffic) -> traffic.Scope).groupByKey();

groupByKey()需要设置两个序列化器，因为这将触发数据重新分区。或者，您将默认服务设置为String和CountryMessage类型。

正如我的评论中所提到的，每个运算符都不使用来自StreamsConfig的默认Serdes，需要设置正确的Serdes。

因此，count()操作也需要指定相应的String和Long Serdes：

countries.count(TimeWindows.of(60 * 1000L), "UserCountStore");

所有可能需要Serdes的运算符都有适当的重载。只需检查您正在使用的所有操作员的所有过载情况。

查看文档了解更多详情：https://docs.confluent.io/current/streams/developer-guide/dsl-api.html

在Kafka Stream API中获取Class Cast异常

2 个答案: