使用kafka连接jdbc接收器

时间:2018-03-12 07:57:37

标签: apache-kafka apache-kafka-connect confluent

我正在尝试将kafka与postgresql连接。我正在使用JDBC sink连接器。

这是我的 sink.properties 文件:

name=test-sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=orders
connection.url=jdbc:postgresql://localhost:5432/casb
auto.create=true

我在postgres中创建了一个名为orders的表,然后我使用以下命令生成一个主题:

./bin/kafka-avro-console-producer \ 
--broker-list localhost:9092 --topic orders \ 
--property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"id","type":"int"},{"name":"product", "type": "string"}, {"name":"quantity", "type": "int"}, {"name":"price", "type": "float"}]}'

我没有在表格中看到任何行。

修改

Kafka Connect Logs:

在发送消息之前,kafka-connect在打印这些日志后正在等待。

[2018-03-12 14:16:55,258] INFO Initializing writer using SQL dialect: PostgreSqlDialect (io.confluent.connect.jdbc.sink.JdbcSinkTask:52)
[2018-03-12 14:16:55,260] INFO WorkerSinkTask{id=test-sink-0} Sink task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:268)
[2018-03-12 14:16:55,436] WARN [Consumer clientId=consumer-1, groupId=connect-test-sink] Error while fetching metadata with correlation id 2 : {orders=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:246)
[2018-03-12 14:16:55,553] WARN [Consumer clientId=consumer-1, groupId=connect-test-sink] Error while fetching metadata with correlation id 4 : {orders=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient:246)
[2018-03-12 14:16:57,910] INFO [Consumer clientId=consumer-1, groupId=connect-test-sink] Discovered coordinator jojo:9092 (id: 2147483647 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:341)
[2018-03-12 14:16:57,913] INFO [Consumer clientId=consumer-1, groupId=connect-test-sink] Revoking previously assigned partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:341)
[2018-03-12 14:16:57,913] INFO [Consumer clientId=consumer-1, groupId=connect-test-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:336)
[2018-03-12 14:16:57,995] INFO [Consumer clientId=consumer-1, groupId=connect-test-sink] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:341)
[2018-03-12 14:16:57,997] INFO [Consumer clientId=consumer-1, groupId=connect-test-sink] Setting newly assigned partitions [orders-0] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:341)

发送消息后,我收到以下错误:

[2018-03-12 14:20:59,770] ERROR WorkerSinkTask{id=test-sink-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. (org.apache.kafka.connect.runtime.WorkerSinkTask:515)
java.lang.NoClassDefFoundError: Could not initialize class org.apache.calcite.jdbc.Driver
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556)
    at java.sql.DriverManager.getConnection(DriverManager.java:661)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at io.confluent.connect.jdbc.util.CachedConnectionProvider.newConnection(CachedConnectionProvider.java:85)
    at io.confluent.connect.jdbc.util.CachedConnectionProvider.getValidConnection(CachedConnectionProvider.java:68)
    at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:56)
    at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:69)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:495)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:288)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:198)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:166)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2018-03-12 14:20:59,774] ERROR WorkerSinkTask{id=test-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:517)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:288)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:198)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:166)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.calcite.jdbc.Driver
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556)
    at java.sql.DriverManager.getConnection(DriverManager.java:661)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at io.confluent.connect.jdbc.util.CachedConnectionProvider.newConnection(CachedConnectionProvider.java:85)
    at io.confluent.connect.jdbc.util.CachedConnectionProvider.getValidConnection(CachedConnectionProvider.java:68)
    at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:56)
    at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:69)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:495)
    ... 10 more
[2018-03-12 14:20:59,777] ERROR WorkerSinkTask{id=test-sink-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:173)
[2018-03-12 14:20:59,777] INFO Stopping task (io.confluent.connect.jdbc.sink.JdbcSinkTask:96)

请帮助我。感谢。

0 个答案:

没有答案