Kafka JDBC Connect:根据消息字段中的多个值插入密钥

时间:2018-10-24 21:35:34

标签: jdbc apache-kafka apache-kafka-connect confluent

关于JDBC连接器发布到的主题,我有以下json

{"APP_SETTING_ID":9,"USER_ID":10,"APP_SETTING_NAME":"my_name","SETTING_KEY":"my_setting_key"}

这是我的连接器文件

name=data.app_setting
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
poll.interval.ms=500
tasks.max=4
mode=timestamp
query=SELECT APP_SETTING_ID, APP_SETTING_NAME, SETTING_KEY,FROM MY_TABLE with (nolock)
timestamp.column.name=LAST_MOD_DATE
topic.prefix=data.app_setting

key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

我现在想通过乘以两个整数字段APP_SETTING_IDUSER_ID来向此消息插入密钥。因此,此消息的密钥变为9*10 = 90

通过Connect可以进行这种转换吗?如果可以,请给大家说明一下

1 个答案:

答案 0 :(得分:1)

我会试着看看你能得到多远

query=SELECT APP_SETTING_ID, APP_SETTING_NAME, SETTING_KEY, (APP_SETTING_ID*USER_ID) as _key FROM MY_TABLE with (nolock)

然后添加一个ExtractKey转换

transforms=AddKeys,ExtractKey
# this make a map
transforms.AddKeys.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.AddKeys.fields=_key  
# this gets one field from the map 
transforms.ExtractKey.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.ExtractKey.field=_key