Question

所以我想实现向kafka consumer发送通知kafka生产者的简单应用程序。到目前为止，我已成功向生产者发送String消息给消费者。但是当我尝试发送通知对象时，kafka消费者没有收到任何对象。这个是我用过的代码。

public class Notification implements Serializable{

    private String name;
    private String message;
    private long currentTimeStamp;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

    public long getCurrentTimeStamp() {
        return currentTimeStamp;
    }

    public void setCurrentTimeStamp(long currentTimeStamp) {
        this.currentTimeStamp = currentTimeStamp;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Notification that = (Notification) o;

        if (currentTimeStamp != that.currentTimeStamp) return false;
        if (message != null ? !message.equals(that.message) : that.message != null) return false;
        if (name != null ? !name.equals(that.name) : that.name != null) return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result = name != null ? name.hashCode() : 0;
        result = 31 * result + (message != null ? message.hashCode() : 0);
        result = 31 * result + (int) (currentTimeStamp ^ (currentTimeStamp >>> 32));
        return result;
    }

    @Override
    public String toString() {
        return "Notification{" +
                "name='" + name + '\'' +
                ", message='" + message + '\'' +
                ", currentTimeStamp=" + currentTimeStamp +
                '}';
    }
}

这是制片人

public class KafkaProducer {
    static String topic = "kafka-tutorial";


    public static void main(String[] args) {
        System.out.println("Start Kafka producer");
        Properties properties = new Properties();
        properties.put("metadata.broker.list", "localhost:9092");
        properties.put("serializer.class", "dev.innova.kafka.tutorial.producer.CustomSerializer");
        ProducerConfig producerConfig = new ProducerConfig(properties);
        kafka.javaapi.producer.Producer<String, Notification> producer = new kafka.javaapi.producer.Producer<String, Notification>(producerConfig);
        KeyedMessage<String, Notification> message = new KeyedMessage<String, Notification>(topic, createNotification());
        System.out.println("send Message to broker");
        producer.send(message);
        producer.close();

    }

    private static Notification createNotification(){
        Notification notification = new Notification();
        notification.setMessage("Sample Message");
        notification.setName("Sajith");
        notification.setCurrentTimeStamp(System.currentTimeMillis());
        return notification;
    }
}

这是消费者

public class KafkaConcumer extends Thread {
    final static String clientId = "SimpleConsumerDemoClient";
    final static String TOPIC = "kafka-tutorial";
    ConsumerConnector consumerConnector;


    public KafkaConcumer() {
        Properties properties = new Properties();
        properties.put("zookeeper.connect","localhost:2181");
        properties.put("group.id","test-group");
        properties.put("serializer.class", "dev.innova.kafka.tutorial.producer.CustomSerializer");
        properties.put("zookeeper.session.timeout.ms", "400");
        properties.put("zookeeper.sync.time.ms", "200");
        properties.put("auto.commit.interval.ms", "1000");
        ConsumerConfig consumerConfig = new ConsumerConfig(properties);
        consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
    }

    @Override
    public void run() {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(TOPIC, new Integer(1));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector.createMessageStreams(topicCountMap);
        KafkaStream<byte[], byte[]> stream =  consumerMap.get(TOPIC).get(0);
        ConsumerIterator<byte[], byte[]> it = stream.iterator();
        System.out.println("It :" + it.size());
        while(it.hasNext()){
            System.out.println(new String(it.next().message()));
        }
    }

    private static void printMessages(ByteBufferMessageSet messageSet) throws UnsupportedEncodingException {
        for(MessageAndOffset messageAndOffset: messageSet) {
            ByteBuffer payload = messageAndOffset.message().payload();
            byte[] bytes = new byte[payload.limit()];
            payload.get(bytes);
            System.out.println(new String(bytes, "UTF-8"));
        }
    }
}

最后我使用了customserializer来序列化和反序列化对象。

public class CustomSerializer implements Encoder<Notification>, Decoder<Notification> {
    public CustomSerializer(VerifiableProperties verifiableProperties) {
        /* This constructor must be present for successful compile. */
    }
    @Override
    public byte[] toBytes(Notification o) {
        return new byte[0];
    }

    @Override
    public Notification fromBytes(byte[] bytes) {
        return null;
    }
}

有人可以告诉我这是什么问题吗？这是正确的方法吗？

Answer 1

你有两个问题。

首先，你的解串器没有任何逻辑。它为每个序列化的对象返回一个空字节数组，并在要求反序列化对象时返回一个空对象。您需要在那里放置实际序列化和反序列化对象的代码。

其次，如果您计划使用JVM中的本机JVM序列化和反序列化逻辑，则需要向将要传输的Bean添加serialVersionUID。像这样：

private static final long serialVersionUID = 123L;

您可以使用您喜欢的任何值。当JVM对对象进行反序列化时，会将对象中的serialVersionId与加载的类定义中指定的值进行比较。如果两者不同，那么JVM假定即使您加载了类定义，也没有加载正确版本的类定义，并且序列化将失败。如果你没有在类定义中为serialVersionID指定一个值，那么JVM将为你创建一个值，两个不同的JVM（生成器和带消费者的JVM）几乎肯定会生成为你起到不同的价值。

修改

如果要利用默认的Java序列化，您需要使序列化程序看起来像这样：

public class CustomSerializer implements Encoder<Notification>, Decoder<Notification> {
    public CustomSerializer(VerifiableProperties verifiableProperties) {
        /* This constructor must be present for successful compile. */
    }

@Override
public byte[] toBytes(Notification o) {
    try {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(baos);
        oos.writeObject(o);
        oos.close();
        byte[] b = baos.toByteArray();
        return b;
    } catch (IOException e) {
        return new byte[0];
    }
}

@Override
public Notification fromBytes(byte[] bytes) {
    try {
        return (Notification) new ObjectInputStream(new ByteArrayInputStream(b)).readObject();
    } catch (Exception e) {
        return null;
    }    
}

Answer 2

创建自定义反序列化器，Kafka需要一种序列化和反序列化的方法。到目前为止，我们必须提供这两种实现方式需要添加库才能获得对象映射器类

FasterXML jackson - 2.8.6

示例 - 序列化程序

public class PayloadSerializer implements org.apache.kafka.common.serialization.Serializer { @Override public byte[] serialize(String arg0, Object arg1) { byte[] retVal = null; ObjectMapper objectMapper = new ObjectMapper(); TestModel model =(TestModel) arg1; try { retVal = objectMapper.writeValueAsString(model).getBytes(); } catch (Exception e) { e.printStackTrace(); } return retVal; } @Override public void close() { } @Override public void configure(Map map, boolean bln) { } }

反序列化程序

public class PayloadDeserializer implements Deserializer { @Override public void close() { } @Override public TestModel deserialize(String arg0, byte[] arg1) { ObjectMapper mapper = new ObjectMapper(); TestModel testModel = null; try { testModel = mapper.readValue(arg1, TestModel.class); } catch (Exception e) { e.printStackTrace(); } return testModel; } @Override public void configure(Map map, boolean bln) { } } 最后，我们必须将反序列化器类传递给接收器

ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG - PayloadDeserializer.class

或

deserializer.class - classpath.PayloadDeserializer

Answer 3

我强烈建议您在发送之前将对象转换为Avro对象。

这并不困难，而且是卡夫卡传播物体的方式。

Kafka消费者无法接收序列化对象？

3 个答案: