我们有一个3节点融合的企业kafka集群（Linux on-prem），一个节点运行了kafka connect服务。我们想使用mysql将数据提取到kafka主题中。
尝试了以下内容-
1。在本地Windows桌面上安装mysql，创建db，table并在其中插入一些数据。
2。使用以下详细信息创建了source-quickstart-mysql.properties文件-
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://<IPAddressOfLocalMachine>:3306/test_db?user=root&password=pwd
tables.whitelist=emp
mode=incrementing
incrementing.column.name=empid
topic.prefix=test-mysql-jdbc-
connect-standalone.properties包含以下信息：
bootstrap.servers=IPaddressOfKCnode:9092
plugin.path=/usr/share/java
重新启动kafka连接服务
试图向kafka连接服务提交连接到我的sql的请求-
curl -X POST -H "Accept:application/json" -H "Content-Type:application/json" IPaddressOfKCnode:8083/connectors/ -d '{"name": "emp-connector", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": "1", "connection.url": "jdbc:mysql://IPaddressOfLocalMachine:3306/test_db?user=root&password=pwd","table.whitelist": "emp","mode": "timestamp","topic.prefix": "mysql-" } }'
在此出现以下错误：
{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nInvalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=pwd for configuration Couldn't open connection to jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=pwd\nInvalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=admin for configuration Couldn't open connection to jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=pwd\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`"}
我还尝试了以下操作-
a。停止kafka连接服务并手动运行-
systemctl stop confluent-kafka-connect
b。像这样跑连接
/usr/bin/connect-standalone /etc/kafka/connect-standalone.properties /etc/kafka-connect-jdbc/source-quickstart-mysql.properties
此过程从一开始就成功启动，但过了一段时间便死了。这是日志：
[2018-11-10 19:42:53,027] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55)
[2018-11-10 19:42:53,048] INFO AbstractConfig values:
batch.max.rows = 100
catalog.pattern = null
connection.attempts = 3
connection.backoff.ms = 10000
connection.password = null
connection.url = jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=pwd
connection.user = null
dialect.name =
incrementing.column.name = empid
mode = incrementing
numeric.mapping = null
numeric.precision.mapping = false
poll.interval.ms = 5000
query =
schema.pattern = null
table.blacklist = []
table.poll.interval.ms = 60000
table.types = [TABLE]
table.whitelist = []
timestamp.column.name = []
timestamp.delay.interval.ms = 0
topic.prefix = test-mysql-jdbc-
validate.non.null = true
(org.apache.kafka.common.config.AbstractConfig:279)
[2018-11-10 19:45:00,439] INFO AbstractConfig values:
batch.max.rows = 100
catalog.pattern = null
connection.attempts = 3
connection.backoff.ms = 10000
connection.password = null
connection.url = jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=admin
connection.user = null
dialect.name =
incrementing.column.name = empid
mode = incrementing
numeric.mapping = null
numeric.precision.mapping = false
poll.interval.ms = 5000
query =
schema.pattern = null
table.blacklist = []
table.poll.interval.ms = 60000
table.types = [TABLE]
table.whitelist = []
timestamp.column.name = []
timestamp.delay.interval.ms = 0
topic.prefix = test-mysql-jdbc-
validate.non.null = true
(org.apache.kafka.common.config.AbstractConfig:279)
[2018-11-10 19:47:07,666] ERROR Failed to create job for /etc/kafka-connect-jdbc/source-quickstart-mysql.properties (org.apache.kafka.connect.cli.ConnectStandalone:102)
[2018-11-10 19:47:07,668] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 2 error(s):
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://192.168.178.14:3306/test_db?user=root&password=admin
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://192.168.178.14:3306/test_db?user=root&password=admin
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:110)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 2 error(s):
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://192.168.178.14:3306/test_db?user=root&password=admin
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://192.168.178.14:3306/test_db?user=root&password=admin
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:415)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:189)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)
[2018-11-10 19:47:07,669] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)
我无法在融合的网站上找到流畅且适当的文档，以使用Kafka Connect服务来处理其各种连接器，配置等。 请帮助获取正确的步骤以实施数据提取管道：mySQL--kafkaconnect--kafka
最后，我期望mysql表中的插入会在kafka主题中产生数据，而kafka使用者将显示这些记录。 摄取似乎很简单，但我错过了一些基本的连接属性：（
谢谢！
答案 0 :(得分：1)
JDBC连接器似乎有问题。您正在运行哪个MySQL版本？ 要解决此问题，您需要：
答案 1 :(得分：1)
首先，您的错误是从curl命令输出中返回给您的：
Connector configuration is invalid and contains the following 2 error(s)java.sql.SQLException: No suitable driver found for jdbc:mysql://X.X.X.X:3306/test_db?user=root&password=pwd
因此，您在Kafka Connect路径中缺少用于MySQL的JDBC驱动程序。
第二个错误是在您发布的输出中：
Connector configuration is invalid and contains the following 2 error(s):Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure.The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://192.168.178.14:3306/test_db?user=root&password=admin
这表明Kafka Connect能够访问您的MySQL机器。
您在哪里运行Confluent平台，是在Docker中，还是在MySQL本地的机器上，等等？ 192.168.178.14是您的MySQL服务器的地址，可以从运行Kafka Connect的主机上访问该地址吗？
您可以找到一些使用Kafka设置MySQL的示例：
有关JDBC连接器与基于日志的CDC的优缺点，请参见https://www.confluent.io/blog/no-more-silos-how-to-integrate-your-databases-with-apache-kafka-and-cdc。
免责声明：我写了以上博客文章。
答案 2 :(得分：0)
谢谢Robin＆Giorgos的回答！这很有帮助。
这个问题与几件事有关-
1.缺少MySQL JDBC连接器jar。
我们必须将MySQL Connector / J 8.0.13放在/usr/share/java/kafka-connect-jdbc/下。
2。连接问题是由于kafka connect试图连接的MySQL用户没有足够的特权来连接到远程连接服务。 为此，我创建了一个具有完全权限并可以访问远程服务器（Kafka connect）的新mySQL用户。
完成上述步骤后，重新启动kafka-connect，提取管道开始工作。