Question

我有一个Kafka Consumer（内置Scala），可以从Kafka中提取最新记录。消费者看起来像这样：

val consumerProperties = new Properties()
consumerProperties.put("bootstrap.servers", "localhost:9092")
consumerProperties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
consumerProperties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
consumerProperties.put("group.id", "something")
consumerProperties.put("auto.offset.reset", "latest")

val consumer = new KafkaConsumer[String, String](consumerProperties)
consumer.subscribe(java.util.Collections.singletonList("topic"))

现在，我想为它编写一个集成测试。测试卡夫卡消费者有什么办法或任何最佳实践吗？

Answer 1

您需要以编程方式启动zookeeper和kafka以进行集成测试。

1.1启动zookeeper（ZooKeeperServer）

def startZooKeeper(zooKeeperPort: Int, zkLogsDir: Directory): ServerCnxnFactory = {
    val tickTime = 2000

    val zkServer = new ZooKeeperServer(zkLogsDir.toFile.jfile, zkLogsDir.toFile.jfile, tickTime)

    val factory = ServerCnxnFactory.createFactory
    factory.configure(new InetSocketAddress("0.0.0.0", zooKeeperPort), 1024)
    factory.startup(zkServer)

    factory
}

1.2启动kafka（KafkaServer）

case class StreamConfig(streamTcpPort: Int = 9092,
                    streamStateTcpPort :Int = 2181,
                    stream: String,
                    numOfPartition: Int = 1,
                    nodes: Map[String, String] = Map.empty)

def startKafkaBroker(config: StreamConfig,
                   kafkaLogDir: Directory): KafkaServer = {

  val syncServiceAddress = s"localhost:${config.streamStateTcpPort}"

  val properties: Properties = new Properties
  properties.setProperty("zookeeper.connect", syncServiceAddress)
  properties.setProperty("broker.id", "0")
  properties.setProperty("host.name", "localhost")
  properties.setProperty("advertised.host.name", "localhost")
  properties.setProperty("port", config.streamTcpPort.toString)
  properties.setProperty("auto.create.topics.enable", "true")
  properties.setProperty("log.dir", kafkaLogDir.toAbsolute.path)
  properties.setProperty("log.flush.interval.messages", 1.toString)
  properties.setProperty("log.cleaner.dedupe.buffer.size", "1048577")

  config.nodes.foreach {
    case (key, value) => properties.setProperty(key, value)
  }

  val broker = new KafkaServer(new KafkaConfig(properties))
  broker.startup()

  println(s"KafkaStream Broker started at ${properties.get("host.name")}:${properties.get("port")} at ${kafkaLogDir.toFile}")
  broker

}

使用KafkaProducer
然后与消费者一起消费以进行测试并验证其有效

您可以使用scalatest-eventstream CTShapeProperties方法为您启动Zookeeper和Kafka。

还有startBroker，它会在测试后清理你的kafka。

例如。

destroyBroker

如何测试卡夫卡消费者

1 个答案: