Question

使用具有更好性能的Apache Camel和Aggregator生成大文件，如下文所述：Large Files with Apache Camel

我的身体有33352行..并且使用completionSize="1000"和completionTimeout="2500"最终文件错过了最后352行

<camel:split streaming="true">
  <camel:simple>${body}</camel:simple>
  <camel:marshal>
    <camel:csv quote='"' quoteDisabled="false" headerDisabled="true" />
  </camel:marshal>
  <camel:aggregate strategyRef="setfepCsvStringBodyAggregator" completionSize="1000" completionTimeout="2500">
    <camel:correlationExpression>
      <constant>true</constant>
    </camel:correlationExpression>
    <to uri="file:{{setfep_dir_inprogress}}/?fileName={{setfep_filename_clientes}}.txt&amp;fileExist=Append" />
  </camel:aggregate>
</camel:split>

最终文件缺少33000行352。

[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000

如果我使用completionSize="1000"和completionInterval="2500"我的最终文件有33155个小时，缺少197行。

[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=566
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=43
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=401
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=768
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[Camel thread #0 - AggregateTimeoutChecker] [INFO ]  CamelLogger.log - Complete by=interval rows=377
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000
[main] [INFO ]  CamelLogger.log - Complete by=size rows=1000

如何解决这个问题？

Apache Camel 2.19.0

Answer 1

陷阱！

我作为单元测试运行...上下文在完成completionInterval或completionTimeout

之前结束

当我在我的路线中延迟时，我可以看到总共写了33352行。

<camel:delay>
  <constant>5000</constant>
</camel:delay>

但是在生产中没有必要，上下文会继续存在，或者我们可以使用forceCompletionOnStop选项：

<camel:aggregate strategyRef="setfepCsvStringBodyAggregator" forceCompletionOnStop="true" completionSize="1000" completionInterval="4000">

使用Apache Camel + Aggregator的大型CSV

1 个答案: