创建了JDBC源连接器
catalog.pattern = test_01
源连接器配置
{
"name": "jdbcsource",
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
"connection.user": "root",
"connection.password": "********",
"catalog.pattern": "test_01",
"mode": "timestamp",
"timestamp.column.name": "UpdateDate",
"topic.prefix": "jdbcsource-"
}
在test_01数据库example01（父表）examplerole（子表）下，外部引用为user_id（请参见下面的架构）
mysql> describe example01;
+------------------+--------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+-------------------+-----------------------------+
| user_id | int(11) | NO | PRI | NULL | auto_increment |
| user_name | varchar(255) | NO | | NULL | |
| user_description | text | YES | | NULL | |
| CreationDate | timestamp | NO | | CURRENT_TIMESTAMP | |
| UpdateDate | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+------------------+--------------+------+-----+-------------------+-----------------------------+
5 rows in set (0.00 sec)
mysql> describe examplerole;
+--------------+---------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------+------+-----+-------------------+-----------------------------+
| prd_id | int(11) | NO | PRI | NULL | auto_increment |
| prd_name | varchar(355) | NO | | NULL | |
| prd_price | decimal(10,0) | YES | | NULL | |
| user_id | int(11) | NO | MUL | NULL | |
| CreationDate | timestamp | NO | | CURRENT_TIMESTAMP | |
| UpdateDate | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+--------------+---------------+------+-----+-------------------+-----------------------------+
6 rows in set (0.00 sec)
使用表中的键参考创建带有topic.regex选项的接收器连接器时
curl -XPOST -H 'Accept: application/json' -H "Content-type: application/json" -d '{
"name": "MySQL-JDBC-sink-connector",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics.regex": "jdbcsource*",
"connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
"connection.user": "root",
"connection.password": "********",
"insert.mode": "insert",
"pk.mode": "record_value",
"pk.fields": "user_id",
"auto.create": "true",
"auto.evolve": "true"
}
}' 'localhost:8083/connectors'
echo "\n"
上述接收器配置出错，但配置中提供了主题正则表达式选项
[2019-06-10 04:54:56,678] ERROR Uncaught exception in REST call to /connector-plugins/io.confluent.connect.jdbc.JdbcSinkConnector/config/validate (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper)
org.apache.kafka.common.config.ConfigException: Must configure one of topics or topics.regex
at org.apache.kafka.connect.runtime.SinkConnectorConfig.validate(SinkConnectorConfig.java:96)
at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:269)
at org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource.validateConfigs(ConnectorPluginsResource.java:81)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.
尝试使用选项topics:<t1,t2>创建多个jdbc接收器连接器，请参考以下配置
{
"name": "jdbcsink",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"topics": [
"jdbcsource-example01",
"jdbcsource-examplerole"
],
"connection.url": "jdbc:mysql://192.168.1.8/test_01?nullCatalogMeansCurrent=true",
"connection.user": "root",
"connection.password": "********",
"auto.create": "true",
"auto.evolve": "true"
}
遇到错误
[2019-06-10 04:53:19,738] WARN [Consumer clientId=consumer-6, groupId=connect-jdbcsink] Error while fetching metadata with correlation id 2 : { jdbcsource-examplerole=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient)
[2019-06-10 04:53:19,739] ERROR WorkerSinkTask{id=jdbcsink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.common.errors.InvalidTopicException: Invalid topics: [ jdbcsource-examplerole]
[2019-06-10 04:53:19,739] ERROR WorkerSinkTask{id=jdbcsink-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2019-06-10 04:53:19,740] INFO Stopping task (io.confluent.connect.jdbc.sink.JdbcSinkTask)
[2019-06-10 04:53:19,749] INFO After filtering the tables are: `test_01`.`example01`,`test_01`.`examplerole` (io.confluent.connect.jdbc.source.TableMonitorThread)
尝试使用单个主题创建接收器连接器，我可以创建接收器连接器。在单个配置中创建多个接收器连接器时面临上述问题
建议使用topics.regex and auto.create选项
答案 0 :(得分：0)
如果您指定topics.regex，则不要指定topics。代替
"topics": [],
"topics.regex": "example*",
您只需要
"topics.regex": "example*",