文件入站通道适配器弹簧集成，用于将多个文件聚合到一个主文件中进行作业处理

时间：2015-02-28 22:21:24

标签： spring-integration

我编写了一个代码，将多个文件合并为一个主文件。 问题在于 int-transformer 我在哪里获取一个文件，尽管我在文件入站通道适配器的复合过滤器中聚合了文件列表。复合过滤器中的文件列表大小是正确的，但在Transformer bean中，文件列表大小始终为1，并且过滤器未获得正确的列表大小聚合文件。

这是我的配置：

<!-- Auto Wiring -->
<context:component-scan base-package="com.nt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger"
    level="DEBUG" />
<int:wire-tap channel="logger" />

<!-- Aggregating the processed Output for OSS processing -->

<int:channel id="networkData" />
<int:channel id="requests" />

<int-file:inbound-channel-adapter id="pollProcessedNetworkData"
    directory="file:${processing.files.directory}" filter="compositeProcessedFileFilter"
    channel="networkData">
    <int:poller default="true" cron="*/20 * * * * *" />

</int-file:inbound-channel-adapter>

<bean id="compositeProcessedFileFilter"
    class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine" />

<int:transformer id="aggregateNetworkData"
    input-channel="networkData" output-channel="requests">
    <bean id="networkData" class="com.nt.na21.nam.integration.helper.CSVFileAggregator">
    </bean>
</int:transformer>

CompositeFileListFilterForBaseLine： 

public class CompositeFileListFilterForBaseLine implements FileListFilter<File> {

    private final static Logger LOG = Logger
            .getLogger(CompositeFileListFilterForBaseLine.class);

    @Override
    public List<File> filterFiles(File[] files) {
        List<File> filteredFile = new ArrayList<File>();
        int index;
        String fetchedFileName = null;
        String fileCreatedDate = null;
        String todayDate = DateHelper.toddMM(new Date());
        LOG.debug("Date - dd-MM: " + todayDate);

        for (File f : files) {
            fetchedFileName = StringUtils.removeEnd(f.getName(), ".csv");
            index = fetchedFileName.indexOf("_");

            // Add plus one to index to skip underscore
            fileCreatedDate = fetchedFileName.substring(index + 1);
            // Format the created file date
            fileCreatedDate = DateHelper.formatFileNameDateForAggregation(fileCreatedDate);
            LOG.debug("file created date: " + fileCreatedDate + " today Date: "
                    + todayDate);
            if (fileCreatedDate.equalsIgnoreCase(todayDate)) {
                filteredFile.add(f);
                LOG.debug("File added to List of File: " + f.getAbsolutePath());
            }
        }
        LOG.debug("SIZE: " + filteredFile.size());
        LOG.debug("filterFiles method end.");
        return filteredFile;
    }

}

CSVFileAggregator

的类文件 
public class CSVFileAggregator {

    private final static Logger LOG = Logger.getLogger(CSVFileAggregator.class);

    private int snePostion;

    protected String masterFileSourcePath=null;

    public File handleAggregateFiles(List<File> files) throws IOException {
        LOG.debug("materFileSourcePath: " + masterFileSourcePath);
        LinkedHashSet<String> allAttributes = null;
        Map<String, LinkedHashSet<String>> allAttrBase = null;
        Map<String, LinkedHashSet<String>> allAttrDelta = null;
        LOG.info("Aggregator releasing [" + files.size() + "] files");
    }
}

日志输出： 

INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - NetFileAggregator context initialized. Polling input folder...
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - Input directory is: D:\Projects\csv\processing
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - Date - dd-MM: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file1_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file2_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - **SIZE: 2**
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - filterFiles method end.
DEBUG: org.springframework.integration.file.FileReadingMessageSource - Added to queue: [csv\processing\file1_base_0103.csv, csv\processing\file2_base_0103.csv]
INFO : org.springframework.integration.file.FileReadingMessageSource - Created message: [GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]]
DEBUG: org.springframework.integration.endpoint.SourcePollingChannelAdapter - Poll resulted in Message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'networkData', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - csv\processing\file2_base_0103.csv
DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.transformer.MessageTransformingHandler - org.springframework.integration.transformer.MessageTransformingHandler@606f8b2b received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: com.nt.na21.nam.integration.helper.CSVFileAggregator - materFileSourcePath: null
INFO : com.nt.na21.nam.integration.helper.CSVFileAggregator - **Aggregator releasing [1] files**

有人可以帮助我确定过滤器的问题，同样不收集转换吗？

提前致谢。

问题在于int：aggregator，因为我不确定如何调用。我之前在设计中已经使用了它，但它根本没有被执行。感谢您的快速回复。

对于这个问题，我编写了一个FileScaner实用程序，它将扫描Folder里面的所有文件，聚合工作正常。

请使用聚合器查找配置并且不起作用，因此我将设计拆分为两个轮询器，首先生成所有CSV文件，然后再收集它并聚合它。

<!-- Auto Wiring -->
<context:component-scan base-package="com.bt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger" level="DEBUG" />
<int:wire-tap channel = "logger" />

<int:channel id="fileInputChannel" datatype="java.io.File" />
<int:channel id="error" />
<int:channel id="requestsCSVInput" />

<int-file:inbound-channel-adapter id="pollNetworkFile"
    directory="file:${input.files.directory}" channel="fileInputChannel"
    filter="compositeFileFilter" prevent-duplicates="true">
    <int:poller default="true" cron="*/20 * * * * *"
        error-channel="error" />
</int-file:inbound-channel-adapter>

<bean id="compositeFileFilter"
    class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForTodayFiles" />

<int:transformer id="transformInputZipCSVFileIntoCSV"
    input-channel="fileInputChannel" output-channel="requestsCSVInput">
    <bean id="transformZipFile"
        class="com.nt.na21.nam.integration.file.net.NetRecordFileTransformation" />
</int:transformer>

<int:router ref="docTypeRouter" input-channel="requestsCSVInput"
    method="resolveObjectTypeChannel">
</int:router>

<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />

<bean id="docTypeRouter"
    class="com.nt.na21.nam.integration.file.net.DocumentTypeMessageRouter" />

<int:service-activator ref="vLanMessageHandler" output-channel="newContentItemNotification" input-channel="Vlan" method="handleFile" />

<bean id="vLanMessageHandler" class="com.nt.na21.nam.integration.file.handler.VLanRecordsHandler" />

<int:service-activator ref="VlanShaperMessageHandler" output-channel="newContentItemNotification" input-channel="VlanShaper" method="handleFile" />

<bean id="VlanShaperMessageHandler" class="com.nt.na21.nam.integration.file.handler.VlanShaperRecordsHandler" />

<int:service-activator ref="PweMessageHandler" output-channel="newContentItemNotification" input-channel="TdmPwe" method="handleFile" />

<bean id="PweMessageHandler" class="com.nt.na21.nam.integration.file.handler.PseudoWireRecordsHandler" />


<int:channel id="newContentItemNotification" />

<!-- Adding for aggregating the records in one place for OSS output -->

 <int:aggregator input-channel="newContentItemNotification" method="aggregate"
    ref="netRecordsResultAggregator" output-channel="net-records-aggregated-reply"
    message-store="netRecordsResultMessageStore"
    send-partial-result-on-expiry="true">
 </int:aggregator>

 <int:channel id="net-records-aggregated-reply" />
 <bean id="netRecordsResultAggregator" class="com.nt.na21.nam.integration.aggregator.NetRecordsResultAggregator" />

 <!-- Define a store for our network records results and set up a reaper that will
    periodically expire those results. -->
<bean id="netRecordsResultMessageStore" class="org.springframework.integration.store.SimpleMessageStore" />


 <int-file:outbound-channel-adapter id="filesOut"
                               directory="file:${output.files.directory}" 
                               delete-source-files="true">

 </int-file:outbound-channel-adapter>

代码工作正常，直到路由到下面的所有频道：

<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />

我正在尝试从包含CSV数据的上述频道的进程返回LinkedHashSet，我需要聚合所有合并

LinkedHashSet vAllAttributes to get the master output CSV file.
List<String> masterList = new ArrayList<String>(vAllAttributes);
Collections.sort(masterList);

1 个答案:

答案 0 :(得分：2)

好吧，看起来你误解了一点<int-file:inbound-channel-adapter>行为。它的性质是one file per message生成channel。它并不依赖于FileListFilter的逻辑。就像：

  1. FileReadingMessageSource使用DirectoryScanner将文件从提供的目录中检索到内部toBeReceived Queue

  2. 由于我们扫描目录中的文件，DirectoryScanner的设计看起来像List<File> listFiles(File directory)。我想这会让你误入歧途。

  3. 之后，filter将应用于原始文件列表，并仅返回适当的文件。

  4. 它们存储在toBeReceived Queue。

  5. 只有在此之后，FileReadingMessageSource才会轮询队列中的项目以构建输出通道的消息。

    6. 要实现aggregation要求，您最好在<aggregator>和<int-file:inbound-channel-adapter>之间使用<int:transformer>。

    您可以使用<poller>标记<int-file:inbound-channel-adapter>的{​​{1}}，以便在单个计划任务期间真正轮询所有文件。但无论如何，消息将与max-messages-per-poll="-1"返回文件一样多。

    之后你必须接受filter的一些技巧：

    1. <aggregator> - 允许您将文件消息合并到单个correlationKey，以便为更远的MessageGroup发布单个消息。由于我们没有来自<transformer>的任何上下文，但我们知道所有邮件都是由单个轮询任务提供的，并且已经安排了线程（您不会在<int-file:inbound-channel-adapter>上使用task-executor {1}}），因此我们可以简单地将<poller>用作：

      correlationKey

    2. 但这还不够，因为无论如何我们应该以某种方式产生单个消息。很遗憾，我们不知道文件的数量（但您可以通过自定义correlation-strategy-expression="T(Thread).currentThread().id" 中的ThreadLocal来完成此操作）以允许FileListFilter返回ReleaseStrategy总计阶段。因此，我们永远不会有正常的true。但我们可以group completion聚合器中未完成的群组使用forceRelease上的MessageGroupStoreReaper或group-timeout。

    3. 除了上一条款，您还应在<aggregator>上提供这些选项：

      <aggegator>

      4. 这就是全部。没有理由提供任何自定义send-partial-result-on-expiry="true" expire-groups-upon-completion="true" （aggregation function / ref或method），因为默认情况下只需使用expression生成一条消息来自群组中所有邮件的List。这适合您的payloads。虽然您可以为聚合函数避免使用CSVFileAggregator和此<transformer>。

      希望我明白

相关问题
最新问题