Kafka-Connect：启动S3接收器连接器时出现无法识别的错误

时间：2020-10-20 19:08:44

标签： amazon-s3 apache-kafka apache-kafka-connect kafka-cluster s3-kafka-connector

我正在尝试为具有3个节点的Kafka Connect集群设置第3个工作线程。 worker在第3个节点上正常运行，我可以进行REST调用以获取现有连接器（目前，我在每个节点上有2个连接器）。但是，当我尝试使用以下命令进行POST调用以创建第3个连接器时：

curl -X POST -H "Content-Type: application/json" --data @test-s3-sink-config.json http://<my-host>:<my-port>/connectors

我收到以下TimeoutException响应：

{"error_code":500,"message":"IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

当我查看工作堆栈跟踪时，它显示以下内容：

[2020-10-20 18:27:04,062] INFO AbstractConfig values:
 (org.apache.kafka.common.config.AbstractConfig:354)
[2020-10-20 18:27:19,081] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestClient:143)
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout
        at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
        at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
        at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:711)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:125)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:65)
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:369)
        at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:164)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.jav
a:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherPr
ovider.java:176)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:500)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Connect Timeout
        at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:812)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

跟踪的第一个日志是困扰我的事情，因为我没有看到关于我做错事情的任何相关信息，第二个日志只是TimeoutException。我到处寻找有类似问题的人，或者了解'AbstractConfig'类，但找不到有用的东西，来自Kafka的this is the AbstractConfig class（我使用的是Kafka 2.0.0版）。

最后，这是我正在使用的配置文件：

{"name":"s3-connector-orderbooks",
"config":{
"connector.class":"io.confluent.connect.s3.S3SinkConnector",
"file":"snapshots-test",
"format.class":"io.confluent.connect.s3.format.json.JsonFormat",
"flush.size":"1000000",
"tasks.max":"1",
"topics":"binance-full-snaps-test",
"timezone":"UTC",
"storage.class":"io.confluent.connect.s3.storage.S3Storage",
"rotate.schedule.interval.ms":"3600000",
"s3.bucket.name":"pfc-data",
"timestamp.extractor":"Record",
"partitioner.class":"io.confluent.connect.storage.partitioner.HourlyPartitioner",
"locale":"en-US",
"s3.compression.type":"gzip"
}
}

如果您觉得有任何其他我应该包括的信息，请随时询问，我是堆栈溢出的新手。

我想知道是否有人遇到过这种情况，或者是否有人对可能导致此问题的原因有所了解。谢谢！

1 个答案:

答案 0 :(得分：0)

在Kafka连接集群中，领导节点是负责服务REST请求的节点。因此，值得检查领导者节点是否可以到达群集中的所有可用工作节点。您可以在{ "version": "2.0", "extensions": { "queues": { "batchSize": 1 } } } 中检查rest.advertised.host.name，以确保在连接群集中是否可以访问节点的播发主机名。罗宾·莫法特（Robin Moffatt）在他的blog中有一篇写得很好的文章。请阅读以获取深入的了解。

相关问题
最新问题