Question

我正在尝试将Json Objects生成为Kafka并手动使用它们，我在org.apache.kafka.streams.examples.pageview中使用JSONPOJO Serdes。

我的制作人代码是：

package JsonProducer;

imports ...

public class jsnPdc {

    public static void main(String[] args) throws IOException {

        byte[] arr= "XXXX       THIS IS TEST DATA \n XYZ".getBytes();    
        JSONObject jsn = new JSONObject();
        jsn.put("Header_Title", (Arrays.copyOfRange(arr, 0, 4)));
        jsn.put("Data_Part", (Arrays.copyOfRange(arr, 4, arr.length)));


        Properties props = new Properties();
        props.put("bootstrap.servers", "xxxxxxxxxxxxxxxxxxxxx:xxxx");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.streams.examples.pageview.JsonPOJOSerializer");

        KafkaProducer<String, JSONObject> pdc = new KafkaProducer<>(props);
        pdc.send(new ProducerRecord<String,JSONObject>("testoutput", jsn));

        System.in.read();


    }

}

并且消费者的代码是：

package testConsumer;

imports ...

public class consumer_0 {
    static public void main(String[] argv) throws ParseException {

        //Configuration
        Properties props = new Properties();
        props.put("bootstrap.servers", "xxxxxxxxxxxxxxxxxxx:xxxx");
        props.put("group.id", "test");
        props.put("enable.auto.commit", "false");
        props.put("auto.commit.interval.ms", "1000");
        props.put("session.timeout.ms", "30000");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.streams.examples.pageview.JsonPOJODeserializer");


        //Create Consumer Object
        KafkaConsumer<String, JSONObject> consumer = new KafkaConsumer<String, JSONObject>(props);
        consumer.subscribe(Arrays.asList("testoutput"));


        //Keep Polling Records
        System.out.println("Polling new record...\n");
        while (true) {
            ConsumerRecords<String, JSONObject> records = consumer.poll(100);

            //Print Each Record
            for (ConsumerRecord<String, JSONObject> record : records){
                JSONObject json = record.value();

                //Some print code, print(json) ...

            }
        }
    }
}

我遇到了这个问题：

Exception in thread "main" org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition testoutput-0 at offset 20491
Caused by: org.apache.kafka.common.errors.SerializationException: java.lang.IllegalArgumentException: Unrecognized Type: [null]
Caused by: java.lang.IllegalArgumentException: Unrecognized Type: [null]
    at com.fasterxml.jackson.databind.type.TypeFactory._fromAny(TypeFactory.java:1170)
    at com.fasterxml.jackson.databind.type.TypeFactory.constructType(TypeFactory.java:618)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2929)
    at org.apache.kafka.streams.examples.pageview.JsonPOJODeserializer.deserialize(JsonPOJODeserializer.java:49)
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:882)
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:788)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:480)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1061)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
    at testConsumer.consumer_0.main(consumer_0.java:43)

我需要json的value字段类型为字节数组。知道为什么会这样吗？

Answer 1

你误解了序列化价值观的责任。你告诉Kafka使用org.apache.kafka.streams.examples.pageview.JsonPOJOSerializer序列化你给它的值，它期望一个普通的java对象，比如

class Data {
    private String headerTitle;
    private String dataPart;
    //... constructors, getters, setters
}

但实际上你已经将JSONObject传递给了ProducerRecord（换句话说，在你将数据交给Kafka之前，你已经自己序列化了数据，然后游戏尝试将其序列化< EM>再次）。

您可以自己序列化jsn vale，但可以使用org.apache.kafka.common.serialization.StringDeserializer作为value.serializer，或者您可以使用org.apache.kafka.streams.examples.pageview.JsonPOJOSerializer来查找，并定义类{上面的{1}}并将该类的内容传递给Data。

从Kafka使用JSON对象时序列化出错

1 个答案: