卡夫卡到雪花的连接问题

时间:2019-10-11 14:21:16

标签: apache-kafka snowflake-data-warehouse

(代表Snowflake客户提交...)

.........................

我正在尝试使用Snowflake Connector for Kafka将Kafka连接到雪花。

参考本文档:https://docs.snowflake.net/manuals/user-guide/kafka-connector.html

运行Kafka时,它正在初始化Snowflake插件。

例如:

[2019-08-31 21:52:09,448] INFO Added aliases 'SnowflakeSinkConnector' and 'SnowflakeSink' to plugin 'com.snowflake.kafka.connector.SnowflakeSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:396)

[2019-08-31 21:52:09,456] INFO Added aliases 'SnowflakeJsonConverter' and 'SnowflakeJson' to plugin 'com.snowflake.kafka.connector.records.SnowflakeJsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:396)

但是之后,它无法读取其他工作程序配置属性。

[2019-08-31 21:52:10,373] WARN The configuration 'connector.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,373] WARN The configuration 'snowflake.topic2table.map' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,375] WARN The configuration 'tasks.max' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,378] WARN The configuration 'topics' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,381] WARN The configuration 'snowflake.private.key.passphrase' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,385] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,386] WARN The configuration 'buffer.flush.time' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,386] WARN The configuration 'snowflake.url.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,387] WARN The configuration 'value.converter.basic.auth.credentials.source' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,387] WARN The configuration 'snowflake.database.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,387] WARN The configuration 'snowflake.schema.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,387] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,389] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,392] WARN The configuration 'value.converter.basic.auth.user.info' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,392] WARN The configuration 'buffer.count.records' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,393] WARN The configuration 'snowflake.private.key' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,393] WARN The configuration 'snowflake.user.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,393] WARN The configuration 'name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,394] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,394] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

[2019-08-31 21:52:10,394] WARN The configuration 'buffer.size.bytes' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:287)

我意识到这些都是警告,但是在那之后,我们遇到了失败。因此我认为它失败了,因为它无法初始化上述配置值。

WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored. 

Sep 04, 2019 11:55:52 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime

WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored. 

Sep 04, 2019 11:55:52 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime

WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.RootResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored. 

Sep 04, 2019 11:55:52 AM org.glassfish.jersey.internal.Errors logErrors

WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.

WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.



[2019-09-04 11:55:52,788] INFO Started o.e.j.s.ServletContextHandler@2be818da{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:850)

[2019-09-04 11:55:52,800] INFO Started http_8083@798deee8{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:292)

[2019-09-04 11:55:52,801] INFO Started @9514ms (org.eclipse.jetty.server.Server:408)

[2019-09-04 11:55:52,802] INFO Advertised URI: http://10.10.25.86:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:267)

[2019-09-04 11:55:52,802] INFO REST server listening at http://10.10.25.86:8083/, advertising URL http://10.10.25.86:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:217)

[2019-09-04 11:55:52,802] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55)

[2019-09-04 11:55:52,807] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113)

org.apache.kafka.common.config.ConfigException: Must configure one of topics or topics.regex

at org.apache.kafka.connect.runtime.SinkConnectorConfig.validate(SinkConnectorConfig.java:96)

at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:269)

at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:189)

at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)

[2019-09-04 11:55:52,808] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)

[2019-09-04 11:55:52,808] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:223)

[2019-09-04 11:55:52,820] INFO Stopped http_8083@798deee8{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:341)

[2019-09-04 11:55:52,821] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)

[2019-09-04 11:55:52,827] INFO Stopped o.e.j.s.ServletContextHandler@2be818da{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1040)

[2019-09-04 11:55:52,829] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:241)

[2019-09-04 11:55:52,829] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:95)

[2019-09-04 11:55:52,829] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:184)

[2019-09-04 11:55:52,829] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)

[2019-09-04 11:55:52,830] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:205)

[2019-09-04 11:55:52,830] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:112)

[2019-09-04 11:55:52,830] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:70)

我了解这可能是一个配置问题,因为在较新版本的Kafka中,“主题”的配置已更新为“主题”,但是还有其他/其他说明,纠正措施或建议的解决方法?

谢谢!

1 个答案:

答案 0 :(得分:4)

您可以忽略所有这些配置警告,它们只是警告(尽管嘈杂和令人困惑的警告!)。

失败的原因是您确定的:

Must configure one of topics or topics.regex

您必须在配置中指定其中之一。