用于Kakfa Connect的Debezium SQLServerConnector无法在Kafka中创建主题

时间：2019-04-16 17:14:57

标签： sql-server apache-kafka apache-kafka-connect debezium

我正在尝试让Debezium的SQLServerConnector在Kafka Connect中运行，并且除未创建任何主题外，一切似乎运行良好。我想念什么吗？

它使用正确的配置启动任务... 

Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,424] INFO Kafka version : 2.1.1-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,424] INFO Kafka commitId : f5b753880d5460f1 (org.apache.kafka.common.utils.AppInfoParser:110)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,425] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:860)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO Starting SqlServerConnectorTask with configuration: (io.debezium.connector.common.BaseSourceTask:42)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    connector.class = io.debezium.connector.sqlserver.SqlServerConnector (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    database.user = cdc_test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    database.dbname = kafka_sql_test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    task.class = io.debezium.connector.sqlserver.SqlServerConnectorTask (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.hostname = mbvmsqlsvr001-t (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.history.kafka.bootstrap.servers = horton1:9092,horton2:9092,horton3:9092,horton4:9092 (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.history.kafka.topic = dbhistory.sql2016test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.password = ******** (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    name = sql-server-test-connector (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.server.name = mbvmsqlsvr001-t (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.port = 1433 (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    table.whitelist = dbo.message_table (io.debezium.connector.common.BaseSourceTask:44)

获取快照设置... 

Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,430] INFO Kafka version : 2.1.1-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,430] INFO Kafka commitId : f5b753880d5460f1 (org.apache.kafka.common.utils.AppInfoParser:110)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,522] INFO Requested thread factory for connector SqlServerConnector, id = mbvmsqlsvr001-t named = change-event-source-coordinator (io.debezium.util.Threads:23
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO Creating thread debezium-sqlserverconnector-mbvmsqlsvr001-t-change-event-source-coordinator (io.debezium.util.Threads:247)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO WorkerSourceTask{id=sql-server-test-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:19
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO No previous offset has been found (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource:56)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO According to the connector configuration both schema and data will be snapshotted (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSourc
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO Snapshot step 1 - Preparing (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:85)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,524] INFO Snapshot step 2 - Determining captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:94)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,528] INFO Snapshot step 3 - Locking captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:100)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Schema locking was disabled in connector configuration (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource:84)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 4 - Determining snapshot offset (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:106)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 5 - Reading structure of captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 6 - Persisting schema history (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:113)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,531] INFO Snapshot step 7 - Snapshotting data (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:125)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,531] INFO Snapshot step 8 - Finalizing (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:149)

连接器任务正在运行... 

# curl  http://172.24.32.10:8083/connectors/sql-server-test-connector/status | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   175  100   175    0     0  73160      0 --:--:-- --:--:-- --:--:-- 87500
{
  "name": "sql-server-test-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "horton5:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "horton5:8083"
    }
  ],
  "type": "source"
}

用户名和密码在SQL Server上具有SPID，并运行CDC查询以获取更改，但在Kafka主题中未显示任何内容。

我不明白什么？我是否应该先创建主题，然后如何配置连接器以向其发布消息？

更新：这是连接器json 

    curl -X POST -H "Content-Type: application/json" --data '{
  "name": "sql-server-test-connector",
  "config": {
    "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
    "database.hostname": "mbvmsqlsvr001-t",
    "database.port": "1433",
    "database.user": "cdc_test",
    "database.password": "Password!123",
    "database.dbname": "kafka_sql_test",
    "database.server.name": "mbvmsqlsvr001-t",
    "table.whitelist": "dbo.message_table",
    "database.history.kafka.bootstrap.servers": "horton1:9092,horton2:9092,horton3:9092,horton4:9092",
    "database.history.kafka.topic": "dbhistory.sql2016test"
    }
  }' http://172.24.32.10:8083/connectors

1 个答案:

答案 0 :(得分：0)

已修复。

我自己的愚蠢错误。我正在使用debezium-connector-sqlserver-0.9.0.Alpha1-plugin.tar.gz发行版。我将debezium-connector-sqlserver-0.9.4.Final-plugin.tar.gz放置在适当的位置，一切正常（我可以看到用户的变化几乎是实时的）。

我什至不知道我如何获得Alpha版本。……

相关问题
最新问题