我正在尝试根据ConsumerRecord字段之一上的内容在使用前过滤ConsumerRecord消息。
在应用过滤器之前对消费者记录进行采样(在值中查找GP_ID):
ConsumerRecord(topic = jdbc-project, partition = 0, offset = 0, CreateTime = 1551118248440, serialized key size = -1, serialized value size = 69, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {"GP_ID": {"bytes": "@"}, "PROJECT_ID": {"bytes": "\u001E\u008C"}, "START_DATE": 1009843200000, "END_DATE": 1041292800000, "TITLE": "Project- FPH", "STATUS_CODE": "INACTIVE"})
KafkaRecordVO(projectId=7820, gpId=64)
当我在kafkaListenerContainerFactory()中按如下所示设置recordFilterStrategy时:
@Bean
ConcurrentKafkaListenerContainerFactory<String, GenericRecord> kafkaListenerContainerFactoryProject() {
ConcurrentKafkaListenerContainerFactory<String, GenericRecord> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setRecordFilterStrategy(new RecordFilterStrategy<String, GenericRecord>() {
@Override
public boolean filter(ConsumerRecord<String, GenericRecord> consumerRecord) {
long gpId= KafkaRecordVO.convertByteBufferToLong(consumerRecord.value().get("GP_ID"));
if(gpId == 10766L || gpId == 10823L || gpId == 10459L || gpId == 10649L)
return false;
else
return true;
}
});
factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
return factory;
}
KafkaRecordVO.convertByteBufferToLong正在将字节缓冲区的值转换为long值。
这是正确的评估,并返回true / false。
但是,当卡夫卡监听器使用它时,如下所示:
@KafkaListener(id = "project", topics = "jdbc-project", containerFactory = "kafkaListenerContainerFactoryProject")
public void consumeProject(ConsumerRecord<String, GenericRecord> record,Acknowledgment acknowledgment) {
log.debug(record.toString());
KafkaRecordVO recordVo = new KafkaRecordVO().projectId(record.value().get("PROJECT_ID"))
.budgetYear(record.value().get("GP_ID"));
log.debug(recordVo.toString());
}
这将返回删除我过滤的字段值的记录:“ GP_ID”
这些是应用过滤器后生成的示例日志(在值中查找GP_ID):
ConsumerRecord(topic = jdbc-project, partition = 0, offset = 171275, CreateTime = 1551118279371, serialized key size = -1, serialized value size = 181, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = { "GP_ID": {"bytes": ""}, "PROJECT_ID": {"bytes": "\u0005â^"}, "START_DATE": 1470009600000, "END_DATE": 1532995200000, "TITLE": "Project 2016 - 2016", "STATUS_CODE": "INACTIVE"})
KafkaRecordVO(projectId=385630, gpId=0)
我在kafkaListener中通过其他字段在日志中找到此字段:“ GP_ID”:{“ bytes”:“”}。如何不剥离价值?这是怎么了?
更新:当我使用实用程序方法将字节转换为long时,发生了此问题。它正在将位置设置为数组的末尾。这就是为什么它返回空数组:
public static Long convertByteBufferToLong(Object byteBuff) {
//After adding below line, the issue got resolved
ByteBuffer buf = ((ByteBuffer) byteBuff).duplicate();
byte[] arr = new byte[buf.remaining()];
buf.get(arr);
BigInteger bi =new BigInteger(1,arr);
return bi.longValue();
}
答案 0 :(得分:0)
您的建议没有任何意义;过滤器适配器具有此代码...
@Override
public void onMessage(ConsumerRecord<K, V> consumerRecord, Acknowledgment acknowledgment, Consumer<?, ?> consumer) {
if (!filter(consumerRecord)) {
switch (this.delegateType) {
case ACKNOWLEDGING_CONSUMER_AWARE:
this.delegate.onMessage(consumerRecord, acknowledgment, consumer);
break;
case ACKNOWLEDGING:
this.delegate.onMessage(consumerRecord, acknowledgment);
break;
case CONSUMER_AWARE:
this.delegate.onMessage(consumerRecord, consumer);
break;
case SIMPLE:
this.delegate.onMessage(consumerRecord);
}
}
else {
ackFilteredIfNecessary(acknowledgment);
}
}
它不对记录进行任何操作。
尝试在调试器中运行以查看是否可以看到正在发生的事情。