Question

我正在尝试测试Brooklin在kafka群集之间的镜像数据。我正在关注维基https://github.com/linkedin/brooklin/wiki/mirroring-kafka-clusters

与wiki不同，我正在尝试在2个不同群集之间设置镜像。我能够启动Brooklin进程和数据流，但是无法镜像消息。 Brooklin在源kafka群集ATM上运行。我正在尝试反映主题“测试”

布鲁克林的server.properties是

############################# Server Basics #############################

brooklin.server.coordinator.cluster=brooklin-cluster
brooklin.server.coordinator.zkAddress=localhost:2181
brooklin.server.httpPort=32311
brooklin.server.connectorNames=file,test,kafkaMirroringConnector
brooklin.server.transportProviderNames=kafkaTransportProvider
brooklin.server.csvMetricsDir=/tmp/brooklin-example/


########################### Transport provider configs ######################

brooklin.server.transportProvider.kafkaTransportProvider.factoryClassName=com.linkedin.datastream.kafka.KafkaTransportProviderAdminFactory
brooklin.server.transportProvider.kafkaTransportProvider.bootstrap.servers=kafka-dest:9092
brooklin.server.transportProvider.kafkaTransportProvider.zookeeper.connect=kafka-dest:2181
brooklin.server.transportProvider.kafkaTransportProvider.client.id=datastream-producer

########################### File connector Configs ######################

brooklin.server.connector.file.factoryClassName=com.linkedin.datastream.connectors.file.FileConnectorFactory
brooklin.server.connector.file.assignmentStrategyFactory=com.linkedin.datastream.server.assignment.BroadcastStrategyFactory
brooklin.server.connector.file.strategy.maxTasks=1

########################### Test event producing connector Configs ######################

brooklin.server.connector.test.factoryClassName=com.linkedin.datastream.connectors.TestEventProducingConnectorFactory
brooklin.server.connector.test.assignmentStrategyFactory=com.linkedin.datastream.server.assignment.LoadbalancingStrategyFactory
brooklin.server.connector.test.strategy.TasksPerDatastream = 4

########################### Kafka Mirroring connector Configs ######################

brooklin.server.connector.kafkaMirroringConnector.factoryClassName=com.linkedin.datastream.connectors.kafka.mirrormaker.KafkaMirrorMakerConnectorFactory
brooklin.server.connector.kafkaMirroringConnector.assignmentStrategyFactory=com.linkedin.datastream.server.assignment.BroadcastStrategyFactory

然后我尝试启动以下数据流；

bin/brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-mirroring-stream -s "kafka://localhost:9092/test" -c kafkaMirroringConnector -t kafkaTransportProvider -m '{"owner":"root","system.reuseExistingDestination":"false"}' 2>/dev/null

尝试检查数据流；

bin/brooklin-rest-client.sh -o READALL -u http://localhost:32311/ 2>/dev/null
[2020-10-14 05:55:45,087] INFO Creating RestClient for http://localhost:32311/ with {}, count=1 (com.linkedin.datastream.DatastreamRestClientFactory)
[2020-10-14 05:55:45,113] INFO The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 '  (com.linkedin.r2.transport.http.client.HttpClientFactory)
[2020-10-14 05:55:45,215] INFO DatastreamRestClient created with retryPeriodMs=6000 retryTimeoutMs=90000 (com.linkedin.datastream.DatastreamRestClient)
[2020-10-14 05:55:45,502] INFO getAllDatastreams took 272 ms (com.linkedin.datastream.DatastreamRestClient)
{
  "name" : "first-mirroring-stream",
  "connectorName" : "kafkaMirroringConnector",
  "transportProviderName" : "kafkaTransportProvider",
  "source" : {
    "connectionString" : "kafka://localhost:9092/test"
  },
  "Status" : "READY",
  "destination" : {
    "connectionString" : "kafka://kafka-dest:9092/*"
  },
  "metadata" : {
    "datastreamUUID" : "df081002-fc7b-4f3a-b1ce-016e879d4b29",
    "group.id" : "first-mirroring-stream",
    "owner" : "root",
    "system.IsConnectorManagedDestination" : "true",
    "system.creation.ms" : "1602665999603",
    "system.destination.KafkaBrokers" : "kafka-dest:9092",
    "system.reuseExistingDestination" : "false",
    "system.taskPrefix" : "first-mirroring-stream"
  }
}

运行此命令后，我尝试在源上进行生产，并在目标上进行消耗，但没有任何镜像。

有人知道我想念什么/我做错了什么吗？

谢谢！

Answer 1

这是我的问题-我在为镜像配置的主题名称中有错字。

使用Brooklin设置kafka镜像

1 个答案: