Question

我们如何使用Monix任务从一系列任务中编写单个输出流（可以并行运行）：例如，我有N个任务可以并行运行，并从HTTP call/server,得到一些响应，并且我将响应写回到文件流中（因为那里限制了内存），但是我发现stream closed error,想知道是否有一种方法可以使用Monix任务或任何其他API来实现这一目标

java.io.IOException: Stream closed

示例代码段：

 import java.io.{FileOutputStream, OutputStream, OutputStreamWriter}

    import monix.eval.Task
    import monix.execution.ExecutionModel.AlwaysAsyncExecution
    import monix.execution.Scheduler

    import scala.util.Random

    object TaskTest extends App {

      implicit val scheduler = Scheduler(
        AlwaysAsyncExecution
      )
      val list = List(1 to 100)

      val filePath = System.currentTimeMillis() + "_test.txt"
      val outputStreamWriter = new OutputStreamWriter(new FileOutputStream(filePath))
      val futureTask = autoClose(outputStreamWriter) {
        outputStream =>
          val totalCount = SeqOfTasks(outputStream)
          totalCount.map(_.sum)
      }

      futureTask.runToFuture.onComplete {
        case scala.util.Success(value) =>
          println(s"fetched a total of $value")

        case scala.util.Failure(exception) =>

          println("there is some error occurred ")
          println(exception)
      }

//close this stream when all tasks completed.
      def autoClose[A <: AutoCloseable, B](resource: A)(code: A ⇒ B): B = {
        try code(resource)
        finally resource.close()
      }


      def SeqOfTasks(outPutStream: OutputStreamWriter): Task[Seq[Int]] = {

        val list = List(0 to 10: _*)
        Task.wanderUnordered(list)(l => makeHttpCall(outPutStream, l))
      }

      // each taks will fetch some data and write to single out put stream that is shared by other tasks.
      def makeHttpCall(outputStream: OutputStreamWriter, value: Int): Task[Int] = {

        Task {
          val result = getDummyValue(value)
          val bytes = result.mkString("\t") + "\n"
          outputStream.write(bytes)
          //return some value
          result.size
        }
      }

      //return a dummy response for http
      def getDummyValue(n: Int): Seq[Int] = {
        println(s"dummy value : $n")
        Seq.fill(n)(Random.nextInt)
      }

Answer 1

该问题是由于import re regex = r"(.+),.*" test_str = " .CEVAXM6_NO_AXI4(CEVAXM6_NO_AXI4), //This part is random" matches = re.finditer(regex, test_str, re.MULTILINE) for matchNum, match in enumerate(matches, start=1): print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group())) for groupNum in range(0, len(match.groups())): groupNum = groupNum + 1 print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))处于惰性和异步状态，而Task却没有。

解决方案是使autoClose任务感知：

autoClose

注意：Oleg Pyzhcov上的Monix Gitter chat会为此回答

Monix从并行任务写入单个输出流

1 个答案: