我正在尝试使用令牌扫描将cassandra表流式传输到akka流,该流将与来自外部源的数据结合在一起。
我的全表扫描逻辑如下:
def verySlowFullTableScan(callback: Product => Unit): Int = {
var currentToken = -9223372036854775808L
var maxToken = currentToken
var maxTokenReached = false
var countRetrieved = 0
while (!maxTokenReached) {
val resultSet: ResultSet = session.execute(statements.fullTableScan(currentToken, 10000))
val rows = resultSet.iterator()
if (!rows.hasNext) {
maxTokenReached = true
} else {
while (rows.hasNext) {
val row = rows.next()
val product: Option[Product] = parseRow(row)
// product should have been parsed correctly
if (product.isDefined) {
countRetrieved = countRetrieved + 1
callback(product.get)
} else {
logger.warn(s"product isn't defined for $row")
}
if (!rows.hasNext) {
currentToken = row.getLong("system.token(sku)")
if (currentToken > maxToken) {
maxToken = currentToken
currentToken = currentToken + 1
}
}
}
}
}
countRetrieved
}
此函数假设每次迭代都有一个回调函数(显然很慢)。
我该如何将这种逻辑转换为akka流兼容源。我认为类似的事情会起作用:
val flow = Flow[Product].mapAsyncUnordered(...) { product => Future
{do something with product}}.toMat(Sink.ignore)(Keep.right)
Source.fromIterator(verySlowFullTableScanThatActsAsAnIterator).runWithFlow(flow)