只有很少的序列化器可用,
org.apache.kafka.common.serialization.StringSerializer
org.apache.kafka.common.serialization.StringSerializer
我们如何创建自己的自定义序列化程序?
答案 0 :(得分:11)
这里有一个示例,可以使用您自己的序列化器/解串器来获取Kafka消息值。对于Kafka,消息密钥也是一样的。
我们希望将MyMessage的序列化版本作为Kafka值发送,并将其再次反序列化为消费者端的MyMessage对象。
在生产者端序列化MyMessage。
您应该创建一个实现org.apache.kafka.common.serialization.Serializer的序列化程序类
serialize()方法完成工作,接收对象并将序列化版本作为字节数组返回。
public class MyValueSerializer implements Serializer<MyMessage>
{
private boolean isKey;
@Override
public void configure(Map<String, ?> configs, boolean isKey)
{
this.isKey = isKey;
}
@Override
public byte[] serialize(String topic, MyMessage message)
{
if (message == null) {
return null;
}
try {
(serialize your MyMessage object into bytes)
return bytes;
} catch (IOException | RuntimeException e) {
throw new SerializationException("Error serializing value", e);
}
}
@Override
public void close()
{
}
}
final IntegerSerializer keySerializer = new IntegerSerializer();
final MyValueSerializer myValueSerializer = new MyValueSerializer();
final KafkaProducer<Integer, MyMessage> producer = new KafkaProducer<>(props, keySerializer, myValueSerializer);
int messageNo = 1;
int kafkaKey = messageNo;
MyMessage kafkaValue = new MyMessage();
ProducerRecord producerRecord = new ProducerRecord<>(topic, kafkaKey, kafkaValue);
producer.send(producerRecord, new DemoCallBack(logTag, startTime, messageNo, strValue));
在消费者方面反序列化MyMessage。
您应该创建一个实现org.apache.kafka.common.serialization.Deserializer
的反序列化器类deserialize()方法完成工作,接收序列化值作为字节数组并返回对象。
public class MyValueDeserializer implements Deserializer<MyMessage>
{
private boolean isKey;
@Override
public void configure(Map<String, ?> configs, boolean isKey)
{
this.isKey = isKey;
}
@Override
public MyMessage deserialize(String s, byte[] value)
{
if (value == null) {
return null;
}
try {
(deserialize value into your MyMessage object)
MyMessage message = new MyMessage();
return message;
} catch (IOException | RuntimeException e) {
throw new SerializationException("Error deserializing value", e);
}
}
@Override
public void close()
{
}
}
然后像这样使用它:
final IntegerDeserializer keyDeserializer = new IntegerDeserializer();
final MyValueDeserializer myValueDeserializer = new MyValueDeserializer();
final KafkaConsumer<Integer, MyMessage> consumer = new KafkaConsumer<>(props, keyDeserializer, myValueDeserializer);
ConsumerRecords<Integer, MyMessage> records = consumer.poll(1000);
for (ConsumerRecord<Integer, MyMessage> record : records) {
int kafkaKey = record.key();
MyMessage kafkaValue = record.value();
...
}
答案 1 :(得分:3)
没有单词,只有代码
某些对象,您将其发送到Kafka
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.ToString;
@Data
@AllArgsConstructor
@NoArgsConstructor
@ToString
public class TestDto {
private String name;
private String version;
}
创建序列化程序,供生产者使用
@Slf4j
public class KafkaValueSerializer implements Serializer<TestDto> {
private ObjectMapper objectMapper = new ObjectMapper();
@Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
@Override
public byte[] serialize(String topic, TestDto data) {
try {
return objectMapper.writeValueAsBytes(data);
} catch (JsonProcessingException e) {
log.error("Unable to serialize object {}", data, e);
return null;
}
}
@Override
public void close() {
}
}
共同的用户,不要为消费者使用反序列化器
@Slf4j
public class KafkaValueDeserializer implements Deserializer<TestDto> {
private ObjectMapper objectMapper = new ObjectMapper();
@Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
@Override
public TestDto deserialize(String topic, byte[] data) {
try {
return objectMapper.readValue(new String(data, "UTF-8"), TestDto.class);
} catch (Exception e) {
log.error("Unable to deserialize message {}", data, e);
return null;
}
}
@Override
public void close() {
}
}
最后,将序列化器/反序列化器添加到 application.yml
spring:
kafka:
bootstrap-servers: 192.168.192.168:9092
producer:
value-serializer: com.package.service.kafka.KafkaValueSerializer
consumer:
group-id: groupId
value-deserializer: com.package.service.kafka.KafkaValueDeserializer
仅此而已。不需要任何配置文件或与Tamboirine跳舞:)
发送
KafkaTemplate<String, TestDto> kafkaTemplate;
TestDto test = new TestDto("test name", "test-version");
kafkaTemplate.send(topic, testDto);
听
@KafkaListener(topics = "${ktp-agent.kafka.request-topic}", groupId = "${spring.kafka.consumer.group-id}")
public void listen(TestDto message) {
log.info("Received message '{}' from Kafka.", message.toString());
}
答案 2 :(得分:-1)
You must create your own serializer which implements the interface 'Serializer' (org.apache.kafka.common.serialization.Serializer) and then set the producer option 'key.serializer/value.serializer' to it.