kafka 从 mysql 下沉到 postgres

时间：2021-06-02 21:40:43

标签： mysql jdbc apache-kafka apache-kafka-connect debezium

我想使用 debezium 和 jdbc sink（confluent）将数据从 mysql 同步到 postgres。

我已使用此教程将 mysql 更改更改为 kafka 主题：https://debezium.io/documentation/reference/1.5/tutorial.html#starting-mysql-command-line-client

这是 mysql debezium 连接器的配置：

{
  "name": "inventory-connector",  
  "config": {  
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",  
    "database.hostname": "mysql",  
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "dbz",
    "database.server.id": "184054",  
    "database.server.name": "dbserver1",  
    "database.include.list": "inventory",  
    "database.history.kafka.bootstrap.servers": "kafka:9092",  
    "database.history.kafka.topic": "schema-changes.inventory"  
  }
}

我已将 jdbc sink 连接器添加到 kafka：

https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc

目前一切正常。

我在 IP 上运行的 Postgres docker 容器：172.17.0.4

因此，我已将此接收器配置添加到 kafka 连接器：

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 172.17.0.3:8083/connectors/ -d '
{
  "name": "posgres-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "connection.url": "jdbc:postgresql://172.17.0.4:3306/users",
    "connection.user": "postgres",
    "connection.password": "postgres",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "tasks.max": "1",
    "auto.create": "true",
    "insert.mode": "insert",
    "topics": "users.accounts",
    "auto.create": "true",
    "pk.mode": "record_key",
    "pk.fields": "id",
    "batch.size": "1"
  }
}'

我使用 dbserver1.inventory.customers 作为主题，因为我正在从 tuto 运行此命令

<块引用>

docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.5 watch-topic -a -k dbserver1.inventory.customers

但是，我在 kafka 日志中没有错误，当我将数据插入 mysql 时，我在 kafka 中获得了更改日志，但在 postgres 中没有：

[2021-06-02 21:37:20,965] INFO JdbcSinkConfig values: 
    auto.create = false
    auto.evolve = false
    batch.size = 3000
    connection.attempts = 3
    connection.backoff.ms = 10000
    connection.password = [hidden]
    connection.url = jdbc:postgresql://172.17.0.7:5432/users
    connection.user = postgres
    db.timezone = UTC
    delete.enabled = false
    dialect.name = 
    fields.whitelist = []
    insert.mode = insert
    max.retries = 10
    pk.fields = []
    pk.mode = none
    quote.sql.identifiers = ALWAYS
    retry.backoff.ms = 3000
    table.name.format = ${topic}
    table.types = [TABLE]
 (io.confluent.connect.jdbc.sink.JdbcSinkConfig:361)
[2021-06-02 21:37:20,967] INFO Initializing writer using SQL dialect: PostgreSqlDatabaseDialect (io.confluent.connect.jdbc.sink.JdbcSinkTask:67)
[2021-06-02 21:37:20,968] INFO WorkerSinkTask{id=jdbc-sink-connector-0} Sink task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:309)
[2021-06-02 21:37:20,989] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Cluster ID: HociFKJoRFa7q6LSbTnrDA (org.apache.kafka.clients.Metadata:279)
[2021-06-02 21:37:20,991] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Discovered group coordinator 172.17.0.3:9092 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:847)
[2021-06-02 21:37:20,999] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540)
[2021-06-02 21:37:21,029] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540)
[2021-06-02 21:37:21,041] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Successfully joined group with generation Generation{generationId=3, memberId='connector-consumer-jdbc-sink-connector-0-23347945-6417-42e3-9c95-8aaaeef1629d', protocol='range'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596)
[2021-06-02 21:37:21,041] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Finished assignment for group at generation 3: {connector-consumer-jdbc-sink-connector-0-23347945-6417-42e3-9c95-8aaaeef1629d=Assignment(partitions=[users.accounts-0])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626)
[2021-06-02 21:37:21,048] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Successfully synced group in generation Generation{generationId=3, memberId='connector-consumer-jdbc-sink-connector-0-23347945-6417-42e3-9c95-8aaaeef1629d', protocol='range'} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756)
[2021-06-02 21:37:21,049] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Notifying assignor about the new Assignment(partitions=[users.accounts-0]) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276)
[2021-06-02 21:37:21,049] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Adding newly assigned partitions: users.accounts-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288)
[2021-06-02 21:37:21,052] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Found no committed offset for partition users.accounts-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1354)
[2021-06-02 21:37:21,061] INFO [Consumer clientId=connector-consumer-jdbc-sink-connector-0, groupId=connect-jdbc-sink-connector] Resetting offset for partition users.accounts-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[172.17.0.3:9092 (id: 1 rack: null)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState:396)
[2021-06-02 21:37:21,222] INFO WorkerSourceTask{id=users-manager-mysql-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:478)
[2021-06-02 21:37:21,223] INFO WorkerSourceTask{id=users-manager-mysql-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:495)
[2021-06-02 21:37:31,224] INFO WorkerSourceTask{id=users-manager-mysql-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:478)
[2021-06-02 21:37:31,224] INFO WorkerSourceTask{id=users-manager-mysql-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:495)

1 个答案:

答案 0 :(得分：0)

使用这个例子，所有配置都可用：

[https://github.com/debezium/debezium-examples/tree/master/unwrap-smt#jdbc-sink][1]

相关问题
最新问题