Question

我正在使用akka-streams-kafka从kafka主题创建一个流消费者。使用广播来提供从kafka主题到Web套接字客户端的事件。

我发现了以下三种创建流源的方法。题：我的目标是服务数百/数千个websocket客户端（其中一些客户端可能是缓慢的消费者）。哪种方法更好地扩展？

欣赏任何想法？广播将速率降低到最慢的消费者 BUFFER_SIZE = 100000

Source.ActorRef（源演员不支持背压选项）

val kafkaSourceActorWithBroadcast = {

  val (sourceActorRef, kafkaSource) = Source.actorRef[String](BUFFER_SIZE, OverflowStrategy.fail)
                                      .toMat(BroadcastHub.sink(bufferSize = 256))(Keep.both).run

  Consumer.plainSource(consumerSettings, 
             Subscriptions.topics(KAFKA_TOPIC))
     .runForeach(record => sourceActorRef ! Util.toJson(record.value()))

kafkaSource
}

Source.queue

val kafkaSourceQueueWithBroadcast = {

  val (futureQueue, kafkaQueueSource) = Source.queue[String](BUFFER_SIZE, OverflowStrategy.backpressure)
                                        .toMat(BroadcastHub.sink(bufferSize = 256))(Keep.both).run

Consumer.plainSource(consumerSettings, Subscriptions.topics(KAFKA_TOPIC))
        .runForeach(record => futureQueue.offer(Util.toJson(record.value())))

kafkaQueueSource
}

缓冲

val kafkaSourceWithBuffer = Consumer.plainSource(consumerSettings, Subscriptions.topics(KAFKA_TOPIC))
                              .map(record => Util.toJson(record.value()))
                              .buffer(BUFFER_SIZE, OverflowStrategy.backpressure)
                              .toMat(BroadcastHub.sink(bufferSize = 256))(Keep.right).run

完整性的Websocket路由代码：

val streamRoute =
    path("stream") {
      handleWebSocketMessages(websocketFlow)
    }


def websocketFlow(where: String): Flow[Message, Message, NotUsed] = {

  Flow[Message]
   .collect {
    case TextMessage.Strict(msg) => Future.successful(msg)
    case TextMessage.Streamed(stream) =>
      stream.runFold("")(_ + _).flatMap(msg => Future.successful(msg))
  }
  .mapAsync(parallelism = PARALLELISM)(identity)
  .via(logicStreamFlow)
  .map { msg: String => TextMessage.Strict(msg) }
}


private def logicStreamFlow: Flow[String, String, NotUsed] = 
         Flow.fromSinkAndSource(Sink.ignore, kafkaSourceActorWithBroadcast)

Akka流Source.actorRef vs Source.queue vs buffer，哪一个使用？

0 个答案: