Question

给定xs: Seq[Source[A, Mat]]以及将各个物化器合并为一个物料的功能，是否可以将xs合并为一个聚合源，该物料源实现为聚合Mat？

考虑这个实际例子：

由N类N Source代表Source[A, Consumer.Control] Kafka主题，我希望将它们合并（合并）到同一个Source[A, Consumer.Control]中，以便生成的Consumer.Control处理所有原始控件。

对于2个来源，这是微不足道的。有这个：

class ConsumerControlBoth(c1: Consumer.Control,
                          c2: Consumer.Control)
                         (implicit ec: ExecutionContext) extends Consumer.Control {
  override def stop(): Future[Done] = {
    (c1.stop().zip(c2.stop())).map(_ => Done)
  }

  override def shutdown(): Future[Done] = {
    (c1.shutdown().zip(c2.shutdown())).map(_ => Done)
  }

  override def isShutdown: Future[Done] = {
    (c1.isShutdown.zip(c2.isShutdown)).map(_ => Done)
  }
}

我可以这样做：

Source.combineMat(s0, s1)(Merge(_))(new ConsumerControlBoth(_, _))

与

一起去是很诱人的

sources.foldLeft(sources.head) { case (acc, src) =>
  Source.combineMat(acc, src)(Merge(_))(new ConsumerControlBoth(_, _))
}

然而，我担心由于每个combineMat试图从其2个输入中均匀地分配元素提取，这可能导致最终的分布不均匀：来自最后一个源的提取元素将具有概率1/2，从倒数第二个提取 - 1/4等。

另一方面，存在用于组合源而没有考虑物化值的vararg方法，例如， Source.combine，其类型为Source[A, NotUsed]。我无法弄清楚如何使用它来组合物化价值。

我在这个假设中是对的还是结果在这个意义上是统一的？如何在通用案例中正确执行此操作？

UPD 即可。我刚刚提出这个问题（只是一个POC，没有健全性检查等）：

def merge[A, M](xs: List[Source[A, M]])
                (implicit
                 mat: Materializer,
                 M: Monoid[M]): Source[A, M] = {
  require(xs.lengthCompare(2) >= 0, "works for sources > 2")

  val seed: (M, List[Source[A, NotUsed]]) = (M.empty, List.empty[Source[A, NotUsed]])
  val (mat, sourcesRev) = xs.foldLeft(seed) { case ((ma, sa), s) =>
    val (mMat, mSrc) = s.preMaterialize()
    (M.combine(ma, mMat), mSrc :: sa)
  }

  val sources: List[Source[A, NotUsed]] = sourcesRev.reverse

  Source
    .combine(sources(0), sources(1), sources.drop(2): _*)(Merge(_))
    .mapMaterializedValue(_ => mat)
}

看起来没有上面提到的缺点，但我不确定我喜欢它。有什么意见吗？

Answer 1

可以组合任意数量的源（物化值的类型应该是相同的），如下所示：

import scalaz.{Ordering => _, _}    

def mergeWithPicker[A](originSources: Seq[Source[Partition, A]])(implicit monoid: Monoid[A], ord: Ordering[Partition]): Source[Partition, A] =
    merge(originSources, picker[A])

 def mergeWithSorter[A](originSources: Seq[Source[Partition, A]])(implicit monoid: Monoid[A], ord: Ordering[Partition]): Source[Partition, A] =
    merge(originSources, sorter[A])

private def merge[A](originSources: Seq[Source[Partition, A]], f: (Source[Partition, A], Source[Partition, A]) => Source[Partition, A])(implicit monoid: Monoid[A]): Source[Partition, A] = originSources match {
    case Nil     =>
      Source.empty[Partition].mapMaterializedValue(_ => monoid.zero)

    case sources =>
      @tailrec
      def reducePairs(sources: Seq[Source[Partition, A]]): Source[Partition, A] =
        sources match {
          case Seq(s) =>
            s

          case _      =>
            reducePairs(sources.grouped(2).map {
              case Seq(a)    => a
              case Seq(a, b) => f(a, b)
            }.toSeq)
        }

      reducePairs(sources)
    }

  private def picker[A](s1: Source[Partition, A], s2: Source[Partition, A])(implicit monoid: Monoid[A], ord: Ordering[Partition]): Source[Partition, A] =
    combineSources(new PartitionPicker[Partition], s1, s2)(monoid.append(_, _))

  private def sorter[A](s1: Source[Partition, A], s2: Source[Partition, A])(implicit monoid: Monoid[A], ord: Ordering[Partition]): Source[Partition, A] =
    combineSources(new MergeSorted[Partition], s1, s2)(monoid.append(_, _))

  private def combineSources[A, MatIn0, MatIn1, Mat](combinator: GraphStage[FanInShape2[A, A, A]], s0: Source[A, MatIn0], s1: Source[A, MatIn1])(combineMat: (MatIn0, MatIn1) => Mat): Source[A, Mat] =
    Source.fromGraph(GraphDSL.create(s0, s1)(combineMat) { implicit builder => (s0, s1) =>
      val merge = builder.add(combinator)
      s0 ~> merge.in0
      s1 ~> merge.in1
      SourceShape(merge.out)
    })

稍后您可以提供隐式Monoid，它描述了如何合并实体化值：

import akka.NotUsed

import scalaz.Monoid

object Monoids {

  implicit final val notUsedMonoid: Monoid[NotUsed] = new Monoid[NotUsed] {
    def zero: NotUsed = NotUsed

    def append(f1: NotUsed, f2: => NotUsed): NotUsed = f1
  }

  implicit def setMonoid[A]: Monoid[Set[A]] = new Monoid[Set[A]] {
    override def zero: Set[A] = Set.empty

    override def append(f1: Set[A], f2: => Set[A]): Set[A] = f1 ++ f2
  }

}

您可能也有兴趣观看此问题： https://github.com/akka/akka/issues/24369

将任意数量的源与物化值组合在一起

1 个答案: