Question

我使用下面的代码使用Java生成器应用程序将消息发布到我的Kafka主题中。

String mySchema = "{"type": "record","name": "MyData","namespace": "com.qwe.rty","doc": "MyData Schema","fields": [{"name": "f1","type": ["null", "string"],"default" : null}, {"name": "f2","type": ["null", "string"],"default" : null}, {"name": "f3","type": ["null", "string"],"default" : null}]}";


Properties props = new Properties();

props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.0.1:9092");

props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);

props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);

props.put("schema.registry.url", "http://192.168.0.1:8081");



KafkaProducer producer = new KafkaProducer(props);

Schema.Parser parser = new Schema.Parser();

Schema schema = parser.parse(mySchema);



GenericRecord avroRecord = new GenericData.Record(schema);

avroRecord.put("f1", "data1");

avroRecord.put("f2", "data2");

avroRecord.put("f3", "data3");



ProducerRecord<String, GenericRecord> record = new ProducerRecord<>("my_topic_1",avroRecord);



try {

    System.out.println("send from producer.serialized.avro.Sender6n");

    producer.send(record);

} catch (Exception e) {

    e.printStackTrace();

}



try{

    System.out.println("flush from producer.serialized.avro.Sender6n");

    producer.flush();

}catch(Exception e){

    e.printStackTrace();

}

消息在Avro中。我正在使用Kafka connect jdbc sink将其插入到我的Oracle表中，使用下面的配置;

name=jdbc-sink-avro

connector.class=io.confluent.connect.jdbc.JdbcSinkConnector

tasks.max=1

topics=my_topic_1

connection.url=jdbc:oracle:thin:@192.168.0.1:1521:user01

connection.user=user_01

connection.password=user_01

auto.create=true

table.name.format=my_table_1

key.converter=io.confluent.connect.avro.AvroConverter

key.converter.schema.registry.url=http://192.168.0.1:8081

value.converter=io.confluent.connect.avro.AvroConverter

value.converter.schema.registry.url=http://192.168.0.1:8081

producer.retries=1

这很好用。但我不希望f3插入到我的表中，这意味着，我只需要一个2列表。换句话说，我想阻止插入f3。我怎样才能做到这一点？

Answer 1

使用Kafka Connect的Single Message Transform（SMT）功能。具体为ReplaceField和blacklist。

您可以看到示例here（＆＃34;字段屏蔽和白名单/黑名单＆＃34;）。

如何防止使用Kafka连接接收器插入Kafka消息中的不需要的属性？

1 个答案: