Question

所以我将文件从本地文件夹移动到sftp位置。文件可能也可能是500,000。我有编写进程的seda，其中多个编写者正在编写文件，但移动文件已成为瓶颈。以下是我的骆驼路线：

from(props.splitFolderUri)
            .routeId('move-files')
            .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moving this split file to sftp location ')
            .to(props.remoteFolderUri)
            .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moved this split file to mailbox')

我想让这个路由多线程，以便多个线程继续移动文件。我尝试使用如下的线程（10）

rom(props.splitFolderUri)
        .threads(10)
        .routeId('move-files')
        .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moving this split file to sftp location ')
        .to(props.remoteFolderUri)
        .log(LoggingLevel.INFO, 'file : ${exchange.getIn().headers[\'CamelFileName\']} :: moved this split file to mailbox')

但是，我重命名文件操作失败了。经过一些研究后，我发现这是出现的情况：这是场景：

Thread#1 retrieves file1, moves to process folder
Thread#2 retrieves same file: file1 at the same time. file1 is deleted
Thread#2 cannot find file1 in source directory, rename fails.
Thread#1 fails due to deleted file by Thread#2

因此，我尝试了noop = true，但这对我来说不是一个好选择，因为容器中的文件应该保留，或者我在容器内部的内存非常有限。那么，对于同样的问题，什么是好的解决方案。我是骆驼的新手所以不能确切指出实现这一点的最佳方法。

更新：我甚至尝试了以下方式：

from(props.splitFolderUri)
   .to("seda:processSplitFiles")

from("seda:processSplitFiles?concurrentConsumers=10")
    .to(log...)
    .to(props.remoteFolderUri)
    ...

但我继续收到以下错误：

    {"@timestamp":"2018-05-27T22:25:48.220-05:00","@version":1,"message":"Error during commit. Exchange[ID-8c85902bd64c-60121-1527477152921-0-90793]. Caused by:w [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename file: GenericFile[/apps/splitFiles/spl/stressTesting.txt_3843_sc_3843_t_10000] to: GenericFile[/apps/splitFiles/spl/.camel/stressTesting.txt_3843_sc_3843_t_10000]]","logger_name":"org.apache.camel.component.file.GenericFileOnCompletion","thread_name":"Camel (camel-1) thread #58 - seda://moveSplitFiles","level":"WARN","level_value":30000,"stack_trace":"org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file: GenericFile[/apps/splitFiles/spl/stressTesting.txt_3843_sc_3843_t_10000] to: GenericFile[/apps/splitFiles/spl/.camel/stressTesting.txt_3843_sc_3843_t_10000]
 org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
  org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
   org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
    org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83) org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
     org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
      org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
       org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
        org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:654)
         org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:609)
          org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
           org.apache.camel.processor.Pipeline.process(Pipeline.java:109)
            org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
             org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:298)
              org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:207)
               org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:154)
                java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  java.lang.Thread.run(Thread.java:748)
                  ","HOSTNAME":"8c85902bd64c"}

请让我知道可以为此做些什么。

Answer 1

（免责声明：不是专家，而是学习......）

我有一个非常类似的场景，通过使用单个线程来选择文件和一个seda路由来实际并行处理它们。

这样的事情：

expect

Answer 2

您可以使用camel聚合器以线程安全的方式工作。请在下面找到聚合器链接：

http://camel.apache.org/aggregator.html

Camel路由使用多个线程将文件从本地文件夹移动到sftp文件夹

2 个答案: