Question

我正在使用kafka connect replicator在同一个kafka连接群集中复制主题。

以下是配置：

/etc/kafka/connect-standalone.properties：

bootstrap.servers=xxxxx:6667,xxxxx:6667,xxxxx:6667
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect_new.offsets
offset.flush.interval.ms=10000
plugin.path=/usr/share/java
security.protocol=PLAINTEXT
offset.flush.timeout.ms=10000
producer.buffer.memory=900

/etc/kafka-connect-replicator/quickstart-replicator.properties：

name=replicator-source-test
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
tasks.max=1

key.converter=io.confluent.connect.replicator.util.ByteArrayConverter
value.converter=io.confluent.connect.replicator.util.ByteArrayConverter
header.converter=io.confluent.connect.replicator.util.ByteArrayConverter

src.kafka.bootstrap.servers=xxxxx:6667,xxxxx:6667,xxxxx:6667
dest.kafka.bootstrap.servers=xxxxx:6667,xxxxx:6667,xxxxx:6667

topic.whitelist=rep-test
topic.rename.format=${topic}.replica

topic.create.backoff.ms=10000

在运行复制器时，我遇到错误，当我在源主题中放入一些消息，即rep-test

org.apache.kafka.connect.errors.ConnectException: OffsetStorageWriter is already flushing
        at org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:110)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:340)
        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.commit(SourceTaskOffsetCommitter.java:108)
        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.access$000(SourceTaskOffsetCommitter.java:45)
        at org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter$1.run(SourceTaskOffsetCommitter.java:82)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2018-06-15 21:47:06,405] ERROR WorkerSourceTask{id=replicator-source-test-0} Failed to flush, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:350)
[2018-06-15 21:47:06,405] ERROR WorkerSourceTask{id=replicator-source-test-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)

似乎复制者的消费者在向目标主题制作或发布消息时工作正常并且失败。

任何帮助？

使用kafka连接器（复制器）

0 个答案: