Question

这可能是一个非常简单的问题，所以我会提前道歉。我要为kafka主题（此处为conf文件）添加s3接收器连接器：

{
  "schemas.enable": "false",
  "name": "my-s3-sink",
  "connector.class": "io.confluent.connect.s3.S3SinkConnector",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "value.converter": "org.apache.kafka.connect.storage.StringConverter",
  "errors.log.enable": "true",
  "errors.log.include.messages": "true",
  "topics": [
    "my-topic-name"
  ],
  "errors.deadletterqueue.context.headers.enable": "true",
  "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
  "flush.size": "2000",
  "rotate.schedule.interval.ms": "600000",
  "s3.bucket.name": "my-bucket-name",
  "s3.object.tagging": "true",
  "s3.region": "region",
  "s3.part.size": "5242880",
  "aws.access.key.id": "****",
  "aws.secret.access.key": "****",
  "s3.ssea.name": "AES256",
  "s3.compression.type": "gzip",
  "storage.class": "io.confluent.connect.s3.storage.S3Storage",
  "topics.dir": "",
  "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
  "partition.duration.ms": "3600000",
  "path.format": "YYYY/MM/dd/HH",
  "locale": "en-GB",
  "timezone": "UTC"
}

这将以{topic_name}/YYYY/MM/DD/HH/{message}格式输出消息，我希望密钥只是YYYY/MM/DD/HH/{message}。经过一些研究，我发现为了从键中删除主题名称，我必须编写一个自定义分区程序，该分区程序扩展并覆盖TimeBasedPartitioner的各个部分。（下面是一个示例https://github.com/confluentinc/kafka-connect-storage-cloud/issues/321）

我的问题是我现在不知道该分区程序的实际代码在哪里写，它应该放在哪里？基于时间的分区器似乎链接到Confluent拥有的某种注册表，但是自定义注册表会去哪里，我将如何在conf文件中为连接器引用该代码？

Answer 1

您将代码编写在一个单独的项目中，将其编译为JAR，然后将其放置在每个connect worker的类路径上。

然后您可以从<script src="https://unpkg.com/vue@2.6.12/dist/vue.min.js"></script> <script src="https://unpkg.com/bootstrap-vue@2.17.3/dist/bootstrap-vue.js"></script> <link href="https://unpkg.com/bootstrap@4.5.2/dist/css/bootstrap.min.css" rel="stylesheet" /> <link href="https://unpkg.com/bootstrap-vue@2.17.3/dist/bootstrap-vue.css" rel="stylesheet" /> <div id="app"> <b-form-textarea ref="my-textarea" v-model="text" placeholder="Certification text" rows="5" max-rows="6"> </b-form-textarea> <b-btn variant="primary" size="lg" @click="addName"> Add Name </b-btn> </div>来引用它

我在哪里为kafka connect sink自定义分区程序编写代码？

1 个答案: