无法使用Alpakka使用来自Kafka Consumer的消息

时间：2019-10-26 17:30:30

标签： scala apache-kafka akka akka-stream alpakka

我正在尝试使用alpakka消耗来自kafka的消息。我没有从akka actor那里得到消费者已经停止的任何错误，但是它没有使用任何消息。下面是我的代码

val consumerSettings = ConsumerSettings(GatewaySettings.DataPlatformKafkaConsumer.kafkaConsumer, new StringDeserializer, new StringDeserializer)
      .withBootstrapServers(bootstrapServers)
      .withGroupId(groupId)
      .withProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")
      .withProperty(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "5000")
      .withProperty("security.protocol", "SASL_SSL")
      .withProperty("ssl.truststore.type", "jks")
      .withProperty("sasl.mechanism", "PLAIN")
      .withProperty("ssl.truststore.location", "deployment/local/ssl/cacerts")
      .withProperty("ssl.truststore.password", "changeit")
      .withProperty("sasl.jaas.config", config)
Consumer
      .plainSource(consumerSettings, Subscriptions.topics("file-upload-alerts"))
      .map(msg => {
       println(msg.value())
      })
      .runWith(Sink.ignore)

我为akka和kafka添加了调试语句，下面是我的调试语句

2019-10-26 12:03:10,397 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-21 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:03:10,567 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-21 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:10,568 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-21 Requesting messages, requestId: 2, partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:13,684 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:03:13,929 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:13,930 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Requesting messages, requestId: 3, partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:20,084 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-23 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:03:23,248 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:23,248 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Requesting messages, requestId: 4, partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:47,375 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-23 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:03:47,549 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-23 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:03:47,549 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-23 Requesting messages, requestId: 5, partitions: Set(file-upload-alerts-0)
2019-10-26 12:07:03,189 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:07:03,891 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:07:03,891 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Requesting messages, requestId: 6, partitions: Set(file-upload-alerts-0)
2019-10-26 12:07:25,142 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:07:25,381 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:07:25,381 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-27 Requesting messages, requestId: 7, partitions: Set(file-upload-alerts-0)
2019-10-26 12:07:28,539 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-26 Revoked partitions: Set(file-upload-alerts-0). All partitions: Set()
2019-10-26 12:07:28,780 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-26 Assigned partitions: Set(file-upload-alerts-0). All partitions: Set(file-upload-alerts-0)
2019-10-26 12:07:28,781 DEBUG   axe-gateway akka.kafka.internal.SingleSourceLogic SingleSourceLogic(akka://gateway-client) gateway-client-akka.actor.default-dispatcher-26 Requesting messages, requestId: 8, partitions: Set(file)

有人可以帮我吗？

1 个答案:

答案 0 :(得分：1)

您的map(msg => println(msg.value()))会将Source的元素转换为Unit s。要捕获并打印msg.value()，请尝试：

Consumer
  .plainSource(consumerSettings, Subscriptions.topics("file-upload-alerts"))
  .map(msg => msg.value())         // or .map(_.value)
  .runWith(Sink.foreach(println))  // or .runForeach(println)

另外，考虑使用onComplete检查是否成功并终止Actor系统：

val consumeMsgs: Future[Done] = Consumer
  ...

consumeMsgs onComplete {
  case Success(_) => println("Done"); system.terminate()
  case Failure(e) => println(e.getMessage); system.terminate()
}
相关问题
最新问题