如何从未知数量的迭代器对象创建akka流源

时间:2018-11-14 12:35:03

标签: scala akka scala-collections akka-stream

我正在尝试使用令牌扫描将cassandra表流式传输到akka流,该流将与来自外部源的数据结合在一起。

我的全表扫描逻辑如下:

def verySlowFullTableScan(callback: Product => Unit): Int = {
    var currentToken = -9223372036854775808L
    var maxToken = currentToken
    var maxTokenReached = false
    var countRetrieved = 0
    while (!maxTokenReached) {
      val resultSet: ResultSet = session.execute(statements.fullTableScan(currentToken, 10000))
      val rows = resultSet.iterator()
      if (!rows.hasNext) {
        maxTokenReached = true
      } else {
        while (rows.hasNext) {
          val row = rows.next()
          val product: Option[Product] = parseRow(row)
          // product should have been parsed correctly
          if (product.isDefined) {
            countRetrieved = countRetrieved + 1
            callback(product.get)
          } else {
            logger.warn(s"product isn't defined for $row")
          }
          if (!rows.hasNext) {
            currentToken = row.getLong("system.token(sku)")
            if (currentToken > maxToken) {
              maxToken = currentToken
              currentToken = currentToken + 1
            }
          }
        }
      }
    }
countRetrieved
  }

此函数假设每次迭代都有一个回调函数(显然很慢)。

我该如何将这种逻辑转换为akka流兼容源。我认为类似的事情会起作用:

val flow = Flow[Product].mapAsyncUnordered(...) { product => Future
  {do something with product}}.toMat(Sink.ignore)(Keep.right)

Source.fromIterator(verySlowFullTableScanThatActsAsAnIterator).runWithFlow(flow)

0 个答案:

没有答案