Question

我有一个示例Json文件，已在mongo db中加载该文件，然后使用debezium连接器将其发送到Kafka connect。我的问题是，尽管原始json输入文件中包含有效数据，RowKey和id键列值如何转换为0（零）？以下是产生此错误的步骤。

第1步：创建示例json文件

{"id":1,"first_name":"Devinne","last_name":"Stubbe","email":"dstubbe0@java.com","gender":"Female","club_status":"platinum","comments":"Extended interactive initiative"}
                    {"id":2,"first_name":"Loise","last_name":"Salt","email":"lsalt1@apple.com","gender":"Female","club_status":"silver","comments":"Optimized neutral standardization"}
                    {"id":3,"first_name":"Benjamen","last_name":"Spittle","email":"bspittle2@howstuffworks.com","gender":"Male","club_status":"platinum","comments":"De-engineered systemic customer loyalty"}
                    {"id":4,"first_name":"Helena","last_name":"Pogosian","email":"hpogosian3@pagesperso-orange.fr","gender":"Female","club_status":"gold","comments":"Phased hybrid definition"}
                    {"id":5,"first_name":"Miller","last_name":"Karolewski","email":"mkarolewski4@g.co","gender":"Male","club_status":"platinum","comments":"Monitored systematic software"}

将此数据加载到customerprofile集合中的mongodb中

rs0:PRIMARY> db.collection.find();
        rs0:PRIMARY> db.customerprofile.find();
        { "_id" : ObjectId("5b6213ba78aae5dc09ef8b5b"), "id" : 1, "first_name" : 
        "Devinne", "last_name" : "Stubbe", "email" : "dstubbe0@java.com", "gender" : 
       "Female", "club_st
       atus" : "platinum", "comments" : "Extended interactive initiative" }
       { "_id" : ObjectId("5b6213ba78aae5dc09ef8b5c"), "id" : 2, "first_name" : 
       "Loise", "last_name" : "Salt", "email" : "lsalt1@apple.com", "gender" : 
       "Female", "club_status"
        : "silver", "comments" : "Optimized neutral standardization" }
        { "_id" : ObjectId("5b6213ba78aae5dc09ef8b5d"), "id" : 3, "first_name" : 
        "Benjamen", "last_name" : "Spittle", "email" : "bspittle2@howstuffworks.com", 
        "gender" : "Male"
        , "club_status" : "platinum", "comments" : "De-engineered systemic customer 
        loyalty" }
        { "_id" : ObjectId("5b6213ba78aae5dc09ef8b5e"), "id" : 4, "first_name" : 
        "Helena", "last_name" : "Pogosian", "email" : "hpogosian3@pagesperso- 
       orange.fr", "gender" : "Fe
       male", "club_status" : "gold", "comments" : "Phased hybrid definition" }
       { "_id" : ObjectId("5b6213ba78aae5dc09ef8b5f"), "id" : 5, "first_name" : 
        "Miller", "last_name" : "Karolewski", "email" : "mkarolewski4@g.co", "gender"    
      : "Male", "club_status" : "platinum", "comments" : "Monitored systematic 
       software" }
       { "_id" : ObjectId("5b6213ba78aae5dc09ef8b60"), "id" : 6, "first_name" : 
       "Cammy", "last_name" : "Suche", "email" : "csuche5@cbsnews.com", "gender" : 
       "Male", "club_status" : "gold", "comments" : "Inverse client-server alliance" 
        }

第2步：创建mongodb debezium连接器配置并激活连接器在kafka connect

{
  "name": "mongodb-jsondb-connector",
  "config":{
   "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
   "mongodb.hosts": "rs0/169.254.57.118:27017",
   "mongodb.name": "jsondb",
   "mongodb.members.auto.discover": "false",
   "database.whitelist": "jsondb",
    "transforms": "unwrap",
   "transforms.unwrap.type": 
 "io.debezium.connector.mongodb.transforms.UnwrapFromMongoDbEnvelope",
    "database.history.kafka.topic": "schema-changes.jsondb",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
   "key.converter.schemas.enable": "false",
   "value.converter": "org.apache.kafka.connect.json.JsonConverter",
   "value.converter.schemas.enable": "false"
 }
}

第3步：检查主题队列

  ksql> print 'jsondb.jsondb.customerprofile' from beginning;
   Format:JSON
   {"ROWTIME":1533154567091,"ROWKEY":"

正如您在下面看到的那样，输出debezium由id列转换为像

这样的objectid

 \"id\":\"5b6213ba78aae5dc09ef8b5b\"}","id":"5b6213ba78aae5dc09ef8b5b".

****我的问题是，因为我们使用JsonConvetor作为密钥转换器，所以如何用序列ID密钥更新我的id列？ ****

 {\"id\":\"5b6213ba78aae5dc09ef8b5b\"}","id":"5b6213ba78aae5dc09ef8b5b","first_name":"Devinne","last_name":"Stubbe","email":"dstubbe0@java.com","gender":"Female","club_status":"platinum","comments":"Extended interactive initiative"}
    {"ROWTIME":1533154567096,"ROWKEY":" 

  {\"id\":\"5b6213ba78aae5dc09ef8b5c\"}","id":"5b6213ba78aae5dc09ef8b5c","first_name":"Loise","last_name":"Salt","email":"lsalt1@apple.com","gender":"Female","club_status":"silver","comments":"Optimized neutral standardization"}
    {"ROWTIME":1533154567096,"ROWKEY":" 

 {\"id\":\"5b6213ba78aae5dc09ef8b5d\"}","id":"5b6213ba78aae5dc09ef8b5d","first_name":"Benjamen","last_name":"Spittle","email":"bspittle2@howstuffworks.com","gender":"Male","club_status":"platinum","comments":"De-engineered systemic customer loyalty"}
{"ROWTIME":1533154567097,"ROWKEY":"{\"id\":\"5b6213ba78aae5dc09ef8b5e\"}","id":"5b6213ba78aae5dc09ef8b5e","first_name":"Helena","last_name":"Pogosian","email":"hpogosian3@pagesperso-orange.fr","gender":"Female","club_status":"gold","comments":"Phased hybrid definition"}
{"ROWTIME":1533154567097,"ROWKEY":"{\"id\":\"5b6213ba78aae5dc09ef8b5f\"}","id":"5b6213ba78aae5dc09ef8b5f","first_name":"Miller","last_name":"Karolewski","email":"mkarolewski4@g.co","gender":"Male","club_status":"platinum","comments":"Monitored systematic software"}
{"ROWTIME":1533154567099,"ROWKEY":"{\"id\":\"5b6213ba78aae5dc09ef8b60\"}","id":"5b6213ba78aae5dc09ef8b60","first_name":"Cammy","last_name":"Suche","email":"csuche5@cbsnews.com","gender":"Male","club_status":"gold","comments":"Inverse client-server alliance"}

第4步：从主题队列创建流

   CREATE STREAM customers_profile \
    (id integer, first_name string, last_name string, \
     email string, gender string, club_status string, \
    comments string) WITH \ 
    (KAFKA_TOPIC='jsondb.jsondb.customerprofile',VALUE_FORMAT='json');

步骤5：这是流从CUSTOMER_REPART接收零（0）的地方

ksql> CREATE STREAM customers_stream WITH 
 (KAFKA_TOPIC='CUSTOMERS_REPART',VALUE_FORMAT='json',PARTITIONS=1) as SELECT * 
 FROM customers_profile PARTITION BY id;

第6步：请检查此打印命令的输出，其中显示零在ROWKEY和id键列值中。他们来自哪里？我需要的确保ROWKEY和id键列显示原始值正确最初在mongodb中加载的json文件值？

ksql> SET 'auto.offset.reset' = 'earliest';
        Successfully changed local property 'auto.offset.reset' from 'null' to 
        'earliest'
        ksql> print 'CUSTOMERS_REPART' from beginning;
        Format:JSON

{“ ROWTIME”：1533154567091，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“白金”，“ GENDER”：“女性”，“ COMMENTS”：“扩展的互动计划”，“ ID”：0， “ LAST_NAME”：“ Stubbe”，“ EMAIL”：“ dstubbe0 @ {“ ROWTIME”：1533154567096，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“银”，“ GENDER”：“女性”，“ COMMENTS”：“优化的中性标准化”，“ ID”：0，“ LAST_NAME” ：“ Salt”，“ EMAIL”：“ lsalt1 @ appl {“ ROWTIME”：1533154567096，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“白金”，“ GENDER”：“男性”，“ COMMENTS”：“经过精心设计的系统客户忠诚度”，“ ID”：0， “ LAST_NAME”：“说话”，“ EMAIL”：“ bs”：“本杰明”} {“ ROWTIME”：1533154567097，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“黄金”，“ GENDER”：“女性”，“ COMMENTS”：“逐步混合定义”，“ ID”：0，“ LAST_NAME” ：“ Pogosian”，“ EMAIL”：“ hpogosian3 @ pagespe {“ ROWTIME”：1533154567097，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“白金”，“ GENDER”：“男性”，“ COMMENTS”：“受监控的系统软件”，“ ID”：0，“ LAST_NAME” ：“ Karolewski”，“ EMAIL”：“ mkarolews {“ ROWTIME”：1533154567099，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“黄金”，“ GENDER”：“男性”，“ COMMENTS”：“反向客户端-服务器联盟”，“ ID”：0，“ LAST_NAME”：“搜寻”，“ EMAIL”：“ csuche5@cbsnews.c {“ ROWTIME”：1533154567104，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“白金”，“ GENDER”：“男性”，“ COMMENTS”：“阶段化统一软件”，“ ID”：0，“ LAST_NAME” ：“ Suttling”，“ EMAIL”：“ msuttling7 @ baidu。 {“ ROWTIME”：1533154567104，“ ROWKEY”：“ 0”，“ CLUB_STATUS”：“黄金”，“ GENDER”：“女性”，“ COMMENTS”：“注重质量的全球软件”，“ ID”：0“，” LAST_NAME”：“ Stepney”，“ EMAIL”：“ estepney8 @ we

Answer 1

关于第一个问题，我认为您看到的是https://github.com/debezium/debezium/blob/master/debezium-connector-mongodb/src/main/java/io/debezium/connector/mongodb/transforms/UnwrapFromMongoDbEnvelope.java#L163

的效果

关于您的第二个问题，我认为问题是id integer，因为id现在包含字符串。

即使数据包含有效值，Mongodb debezium连接器也会将Rowkey填充为0（零），将Key填充为0零

1 个答案: