Question

我正在使用 Snowflake Kafka 连接器将 JSON 记录从 Kafka 主题推送到我的 Snowflake 实例。我正在使用 Strimzi Kafka operators 来设置和配置 Kafka 集群和 Kafka 连接集群（使用包含雪花连接器的自定义容器，如 here 所述）。我使用 minikube 作为 K8S 集群。

我观察到的是记录从我的本地 minikube Kafka 连接器容器推送到 Snowflake，但 Snowflake 连接器在几秒钟后自动退出。从日志中可以看出，Snowflake 连接器自行正常关闭。这里是相关的日志。

记录被推送到 Snowflake：

[SF_KAFKA_CONNECTOR] SnowflakeSinkTask[ID:0]:put 1 records (com.snowflake.kafka.connector.SnowflakeSinkTask) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] SnowflakeSinkTask[ID:0]:put 0 records (com.snowflake.kafka.connector.SnowflakeSinkTask) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] searched 1 files in ingest report, found 1 (com.snowflake.kafka.connector.internal.SnowflakeIngestionServiceV1) [pool-10-thread-1]
[SF_KAFKA_CONNECTOR] purge 1 files from stage: SNOWFLAKE_KAFKA_CONNECTOR_snow_sink_test_444250050_STAGE_test_result_records (com.snowflake.kafka.connector.internal.SnowflakeConnectionServiceV1) [pool-10-thread-1]
[SF_KAFKA_CONNECTOR] pipe SNOWFLAKE_KAFKA_CONNECTOR_snow_sink_test_444250050_PIPE_test_result_records_0, flush pipe: snow_sink_test_444250050/test_result_records/0/2_2_1614862161400.json.gz (com.snowflake.kafka.connector.internal.SnowflakeSinkServiceV1) [pool-11-thread-1]
[SF_KAFKA_CONNECTOR] SnowflakeSinkTask[ID:0]:put 1 records (com.snowflake.kafka.connector.SnowflakeSinkTask) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] pipe SNOWFLAKE_KAFKA_CONNECTOR_snow_sink_test_444250050_PIPE_test_result_records_0, flush pipe: snow_sink_test_444250050/test_result_records/0/3_3_1614862192771.json.gz (com.snowflake.kafka.connector.internal.SnowflakeSinkServiceV1) [pool-11-thread-1]

但几秒钟后，连接器关闭：

[SF_KAFKA_CONNECTOR] SnowflakeSinkConnector:stop (com.snowflake.kafka.connector.SnowflakeSinkConnector) [connector-thread-snow-sink-test]
[SF_KAFKA_CONNECTOR] SnowflakeSinkTask[ID:0]:close (com.snowflake.kafka.connector.SnowflakeSinkTask) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] pipe SNOWFLAKE_KAFKA_CONNECTOR_snow_sink_test_444250050_PIPE_test_result_records_0: cleaner terminated (com.snowflake.kafka.connector.internal.SnowflakeSinkServiceV1) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] pipe SNOWFLAKE_KAFKA_CONNECTOR_snow_sink_test_444250050_PIPE_test_result_records_0: flusher terminated (com.snowflake.kafka.connector.internal.SnowflakeSinkServiceV1) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] IngestService Closed (com.snowflake.kafka.connector.internal.SnowflakeIngestionServiceV1) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] pipe SNOWFLAKE_KAFKA_CONNECTOR_snow_sink_test_444250050_PIPE_test_result_records_0: service closed (com.snowflake.kafka.connector.internal.SnowflakeSinkServiceV1) [task-thread-snow-sink-test-0]
[SF_KAFKA_CONNECTOR] SnowflakeSinkTask[ID:0]:stop (com.snowflake.kafka.connector.SnowflakeSinkTask) [task-thread-snow-sink-test-0]

我已经使用两个连接器版本测试了此工作流程：1.2.3 和 1.5.2，并且它们是相同的。

有什么我遗漏的吗？

这是我正在使用的连接器配置文件（已删除敏感信息）：

{
 "name":"snow-sink-test",
 "config":{
   "connector.class":"com.snowflake.kafka.connector.SnowflakeSinkConnector",
   "tasks.max":"1",
   "topics":"test-topic",
   "snowflake.topic2table.map": "test-topic:test_topic",
   "buffer.count.records":"1000",
   "buffer.size.bytes":"1048576",
   "snowflake.url.name":"#####.snowflakecomputing.com",
   "snowflake.user.name":"#####",
   "snowflake.private.key":"#####",
   "snowflake.database.name":"SNOW_KAFKA_TEST",
   "snowflake.schema.name":"PUBLIC",
   "key.converter":"org.apache.kafka.connect.storage.StringConverter",
   "value.converter":"com.snowflake.kafka.connector.records.SnowflakeJsonConverter",
   "value.converter.schema.registry.url":"http://localhost:8081"
 }
}

提前感谢您的指点！

雪花 Kafka 连接器快速终止

0 个答案: