Question

我正在尝试使用骆驼在只读文件系统中的端点上找到可用的最新文件并使用它。由于我有一个基于akka流的应用程序，因此我决定使用streamz library

这是我当前的解决方案，也是我看到的问题。我确定我可以在这里找到更好的解决方案，但我自己无法解决。

我当前的解决方案涉及在只读端点上设置一条路由，该路由会在所有匹配的文件上进行汇总，并且父路径为关联键并选择最新的路由。我在端点设置中使用sortBy以确保最后修改的文件进入最新的交换。我在这里所做的是将这个输出写入另一个文件端点（本地副本）然后在此端点上进行receive方法设置。我想避免这种额外的文件复制，也许将汇总的交换直接传递给receive方法？

object FileConsumerStarter extends App {

  import FeedConsumer._

  val endpoint = """file:poller/src/main/resources/file-poller?"""
  val fileRegex = "file.*\\.csv"
  setupSource(endpoint, fileRegex)
      .to(Sink.foreach(a => println(a)))
      .run()

}

import java.util.UUID
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import org.apache.camel.{Exchange}
import org.apache.camel.processor.aggregate.UseLatestAggregationStrategy
import org.apache.camel.builder.RouteBuilder
import streamz.camel.StreamContext
import streamz.camel.akka.scaladsl.receive

trait FeedConsumer {

  implicit val actorSystem = ActorSystem("FeedConsumer")
  implicit val actorMaterializer = ActorMaterializer()
  implicit val streamContext = StreamContext()
  streamContext.camelContext.start()

  val defaultEndpointSettings = Seq(
    "autoCreate=false", 
    "charset=utf-8",
    "delay=10000",
    "noop=true",
    "sortBy=file:modified",
    "eagerMaxMessagesPerPoll=false",
    "readLock=none"
  )

  def setupSource(camelFileEndpoint: String, camelFileEndpointFileRegex: String, endpointSettings:Seq[String] = defaultEndpointSettings) = {

    val endpointString = (camelFileEndpoint.last match {
      case '?' => camelFileEndpoint
      case _ => camelFileEndpoint + '?'
    }
      ) + (s"include=$camelFileEndpointFileRegex" +: endpointSettings).mkString("&")

    val someUniqId = UUID.randomUUID()

    val rb = new RouteBuilder() {
      override def configure(): Unit =
        from(endpointString)
          .aggregate(header(Exchange.FILE_PARENT), new UseLatestAggregationStrategy())
            .completionTimeout(1000L)
            .ignoreInvalidCorrelationKeys()
          .to(s"file:///tmp/file_manager/${someUniqId}")
    }

    streamContext.camelContext.addRoutes(rb)

    val settings = Seq(
      "delete=true",
      pickOneFile,
      "readLock=markerFile",
      "delay=2000"
    ).mkString("&")

    receive[String](s"file:///tmp/gcm_file_manager/${someUniqId}?${settings}")

  }

object FeedConsumer extends FeedConsumer

如何使用Streamz DSL在Apache骆驼中消费聚合交易

0 个答案: