Question

我有一个非常简单的Akka Streams流，它使用alpakka从Kafka读取msg，对msg进行一些操作并将其索引到Elasticsearch。

我正在使用CommitableSource，因此我处于至少一次战略。我仅在对ES的索引成功时才提交偏移，如果失败，我将再次读取该消息，因为形成了最新的已知偏移。

 val decider: Supervision.Decider = {
    case _:Throwable =>  Supervision.Restart
    case _           => Supervision.Restart
  }

  val config: Config = context.system.settings.config.getConfig("akka.kafka.consumer")

  val flow: Flow[CommittableMessage[String, String], Done, NotUsed] =
    Flow[CommittableMessage[String,String]].
      map(msg => Event(msg.committableOffset,Success(Json.parse(msg.record.value()))))
    .mapAsync(10) { event => indexEvent(event.json.get).map(f=> event.copy(json = f))}
      .mapAsync(10)(f => {
    f.json match {
      case Success(_)=> f.committableOffset.commitScaladsl()
      case Failure(ex) => throw new StreamFailedException(ex.getMessage,ex)
    }
      })

  val r: Flow[CommittableMessage[String, String], Done, NotUsed] = RestartFlow.onFailuresWithBackoff(
    minBackoff = 3.seconds,
    maxBackoff = 3.seconds,
    randomFactor = 0.2, // adds 20% "noise" to vary the intervals slightly
    maxRestarts = 20 // limits the amount of restarts to 20
  )(() => {
    println("Creating flow")
    flow
  })

  val consumerSettings: ConsumerSettings[String, String] =
    ConsumerSettings(config, new StringDeserializer, new StringDeserializer)
      .withBootstrapServers("localhost:9092")
      .withGroupId("group1")
      .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")

  val restartSource: Source[CommittableMessage[String, String], NotUsed] = RestartSource.withBackoff(
    minBackoff = 3.seconds,
    maxBackoff = 30.seconds,
    randomFactor = 0.2, // adds 20% "noise" to vary the intervals slightly
    maxRestarts = 20 // limits the amount of restarts to 20
  ) {() =>
    Consumer.committableSource(consumerSettings, Subscriptions.topics("test"))
  }


  implicit val mat: ActorMaterializer = ActorMaterializer(ActorMaterializerSettings(context.system).withSupervisionStrategy(decider))



  restartSource
    .via(flow)
    .toMat(Sink.ignore)(Keep.both).run()

我想实现的是重新启动整个流程Source-> Flow-> Sink。如果出于某种原因我无法在Elastic中索引消息。

我尝试了以下操作：

Supervision.Decider-好像重新创建了流程，但没有邮件是从卡夫卡提取的，显然是因为它记得偏移量。
RestartSource-看起来不像以太，因为异常发生在流程阶段。
RestartFlow-也无济于事，因为它仅重新启动Flow，但是我需要从上次成功的偏移量重新启动Source。

有什么优雅的方法吗？

Answer 1

您可以组合可重新启动的源，流和宿。没有人阻止您对图形的每个部分执行可重新启动的源/流/接收器

更新：

代码示例

val sourceFactory = () => Source(1 to 10).via(Flow.fromFunction(x => { println("problematic flow"); x }))
RestartSource.withBackoff(4.seconds, 4.seconds, 0.2)(sourceFactory)

万一发生阶段故障，Akka Streams会重新创建流

1 个答案: