像scala Future中的操作一样执行DAG

时间:2016-04-01 06:48:08

标签: scala future directed-acyclic-graphs

我正在研究用例,我必须使用scala Future执行相互依赖的操作(定义为有向无环图)。基本上每个操作(比如DAG的节点)将在Future中执行,并且一旦当前节点Future完成,将触发后续的依赖节点(它们也应该在Future中)。这将持续到每个节点都已完成处理或其中一个节点失败。到目前为止,我有(最小代码):

def run(node: Node, result: Result): Unit = {
  val f: Future[(Node, Result)] = Future {
    // process current Node 
    ...
  }

  f onComplete {
    case Success(x) =>
      val n = x._1 // Current Node
      val r = x._2 // Result of current Node
      if (!n.isLeaf()) {
        n.children.foreach { z =>
          run(z, r)
        }
      } 
    case Failure(e) => throw e
  }
}

这是解决此问题的正确方法(在回调中调用另一个未来)吗?一旦其中一个节点处理失败,我就无法以正确的方式停止其他运行的未来。

使用Future组合可以解决这个问题吗?如果是这样,我怎么能实现呢?

谢谢,
普拉

1 个答案:

答案 0 :(得分:1)

以下是一种更具功能性的方法:我们可以使用Unit / run作为评估Future / Future的结果,而不是使用import scala.concurrent.{Await, Future} import scala.concurrent.ExecutionContext.Implicits.global import scala.concurrent.duration._ import scala.util.Try case class Node[T](value: T, children: List[Node[T]]) object DagFuture extends App { def run[A, B](node: Node[A], result: B)(nodeEval: (Node[A], B) => B)(aggregator: List[B] => B): Future[B] = { val nodeResult: Future[B] = Future(nodeEval(node, result)) val allResults: Future[List[B]] = nodeResult.flatMap(r => Future.sequence(nodeResult :: node.children.map(x => run(x, r)(nodeEval)(aggregator)))) val finalResult: Future[B] = allResults.map(cl => aggregator(cl)) finalResult } val debugSum = (l: List[Int]) => { println(s"aggregating: $l") l.sum } def debugNodeEval(f: (Node[Int], Int) => Int)(n: Node[Int], r: Int) = { val eval = Try { f(n, r) } println(s"node: $n, result: $r, eval: $eval") eval.get } val debugNodeEvalDefault = debugNodeEval((n, r) => n.value + r) _ val singleNodeDag = Node(1, Nil) val multiNodeDag = Node(1, List(Node(20, Nil), Node(300, Nil))) println("\nSINGLE NODE DAG EXAMPLE:") val singleNodeFuture = run(singleNodeDag, 0)(debugNodeEvalDefault)(debugSum) val singleNodeResult = Await.result(singleNodeFuture, 5 seconds) println(s"Single node result: $singleNodeResult") println("\nDAG PATH LENGTH EXAMPLE:") val pathLengthFuture = run(multiNodeDag, 0)(debugNodeEvalDefault)(debugSum) val pathLengthResult = Await.result(pathLengthFuture, 5 seconds) println(s"Path length: $pathLengthResult") println("\nFAILED DAG ROOT NODE EXAMPLE:") val failedRootNodeFuture = run(multiNodeDag, 0)(debugNodeEval((n, r) => throw new Exception))(debugSum) val failedRootNodePromise = Await.ready(failedRootNodeFuture, 5 seconds) println(s"Failed root node: ${failedRootNodePromise.value}") println("\nFAILED DAG CHILD NODE EXAMPLE:") val failedChildNodeFuture = run(multiNodeDag, 0)(debugNodeEval((n, r) => if (n.value == 300) throw new Exception else n.value + r))(debugSum) val failedChildNodePromise = Await.ready(failedChildNodeFuture, 5 seconds) println(s"Failed child node: ${failedChildNodePromise.value}") } 。通常,您希望在功能上使用SINGLE NODE DAG EXAMPLE: node: Node(1,List()), result: 0, eval: Success(1) aggregating: List(1) Single node result: 1 DAG PATH LENGTH EXAMPLE: node: Node(1,List(Node(20,List()), Node(300,List()))), result: 0, eval: Success(1) node: Node(20,List()), result: 1, eval: Success(21) node: Node(300,List()), result: 1, eval: Success(301) aggregating: List(301) aggregating: List(21) aggregating: List(1, 21, 301) Path length: 323 FAILED DAG ROOT NODE EXAMPLE: node: Node(1,List(Node(20,List()), Node(300,List()))), result: 0, eval: Failure(java.lang.Exception) Failed root node: Some(Failure(java.lang.Exception)) FAILED DAG CHILD NODE EXAMPLE: node: Node(1,List(Node(20,List()), Node(300,List()))), result: 0, eval: Success(1) node: Node(20,List()), result: 1, eval: Success(21) aggregating: List(21) node: Node(300,List()), result: 1, eval: Failure(java.lang.Exception) Failed child node: Some(Failure(java.lang.Exception)) 的结果,而不是副作用。

我添加了类型注释和描述性变量名称,以便更容易理解。我还添加了几个案例来说明它将如何失败。当发生故障时,您还可以选择恢复而不是将所有内容都失败。但是,对于这个问题,如果子计算依赖于父值,那么失败可能更合理。

def run[A, B](node: Node[A], result: B)(nodeEval: (Node[A], B) => B)(aggregator: Traversable[B] => B): Future[B] = {
    val nodeResult = Future(nodeEval(node, result))
    val allResults = nodeResult flatMap { r => Future.sequence(nodeResult :: node.children.map { x => run(x, r)(nodeEval)(aggregator) }) }
    allResults map aggregator
  }

打印出来:

Future.flatMap(result => Future.sequence(children ...))

TL; DR

Future

松散地说它只是flatMap。当父Future完成时,其结果将在sequence中传递给子计算。如果父Future失败,则整个计算也会失败。 FutureFuture列表的结果合并为一个<img src='folder/'.$steamprofile['avatarfull'].'' alt="user-img" class="img-circle"> 。孩子char是其子女的父母,递归等等。因此适用相同的故障模式。