Question

我正在管道的开头使用Spring Cloud Stream File Source应用程序构建数据管道。我需要一些帮助来解决一些缺少的功能

“我的文件”源应用程序（基于org.springframework.cloud.stream.app:spring-cloud-starter-stream-source-file）运行得很好，除了缺少一些需要帮助的功能。我需要

要在轮询并发送消息后删除文件
轮询子目录

关于项目1，我读到文件源应用程序中不存在删除功能（在sftp源上可用）。每次重新启动应用程序时，都会重新选择过去处理过的文件，是否可以使处理过的文件的历史记录永久化？有简单的选择吗？

Answer 1

要满足这些要求，您肯定需要修改上述“文件源”项目的代码：https://docs.spring.io/spring-cloud-stream-app-starters/docs/Einstein.BUILD-SNAPSHOT/reference/htmlsingle/#_patching_pre_built_applications

我建议分叉该项目并按原样从GitHub对其进行轮询，因为您将要修改该项目的现有代码。然后您按照提到的文档中的说明如何构建与SCDF环境兼容的目标特定于粘合剂的工件。

现在讨论问题：

要轮询同一文件模式的子目录，您需要在RecursiveDirectoryScanner上配置Files.inboundAdapter()：

/**
 * Specify a custom scanner.
 * @param scanner the scanner.
 * @return the spec.
 * @see FileReadingMessageSource#setScanner(DirectoryScanner)
 */
public FileInboundChannelAdapterSpec scanner(DirectoryScanner scanner) {

请注意，必须在此filters上配置所有DirectoryScanner。否则会有警告：

    // Check that the filter and locker options are _NOT_ set if an external scanner has been set.
    // The external scanner is responsible for the filter and locker options in that case.
    Assert.state(!(this.scannerExplicitlySet && (this.filter != null || this.locker != null)),
            () -> "When using an external scanner the 'filter' and 'locker' options should not be used. " +
                    "Instead, set these options on the external DirectoryScanner: " + this.scanner);

要跟踪文件，最好考虑基于FileSystemPersistentAcceptOnceFileListFilter实现https://docs.spring.io/spring-integration/reference/html/#metadata-store的外部持久性存储来拥有ConcurrentMetadataStore。必须使用它代替preventDuplicates()，因为FileSystemPersistentAcceptOnceFileListFilter还要确保对我们来说仅一次逻辑。

在发送后删除文件可能不是一种情况，因为您可以按原样发送File，并且该文件必须在另一端可用。

此外，您可以在ChannelInterceptor中添加source.output()并实现其postSend()来执行((File) message.getPayload()).delete()，这将在消息成功发送到活页夹目的地。

Answer 2

@EnableBinding(Source.class)
@Import(TriggerConfiguration.class)
@EnableConfigurationProperties({FileSourceProperties.class, FileConsumerProperties.class,
    TriggerPropertiesMaxMessagesDefaultUnlimited.class})
public class FileSourceConfiguration {


@Autowired
@Qualifier("defaultPoller")
PollerMetadata defaultPoller;
@Autowired
Source source;
@Autowired
private FileSourceProperties properties;
@Autowired
private FileConsumerProperties fileConsumerProperties;

private Boolean alwaysAcceptDirectories = false;
private Boolean deletePostSend;
private Boolean movePostSend;
private String movePostSendSuffix;

@Bean
public IntegrationFlow fileSourceFlow() {
    FileInboundChannelAdapterSpec messageSourceSpec = Files.inboundAdapter(new File(this.properties.getDirectory()));

    RecursiveDirectoryScanner recursiveDirectoryScanner = new RecursiveDirectoryScanner();
    messageSourceSpec.scanner(recursiveDirectoryScanner);
    FileVisitOption[] fileVisitOption = new FileVisitOption[1];
    recursiveDirectoryScanner.setFilter(initializeFileListFilter());

    initializePostSendAction();

    IntegrationFlowBuilder flowBuilder = IntegrationFlows
            .from(messageSourceSpec,
                    new Consumer<SourcePollingChannelAdapterSpec>() {

                        @Override
                        public void accept(SourcePollingChannelAdapterSpec sourcePollingChannelAdapterSpec) {
                            sourcePollingChannelAdapterSpec
                                    .poller(defaultPoller);
                        }

                    });

    ChannelInterceptor channelInterceptor = new ChannelInterceptor() {
        @Override
        public void postSend(Message<?> message, MessageChannel channel, boolean sent) {
            if (sent) {
                File fileOriginalFile = (File) message.getHeaders().get("file_originalFile");
                if (fileOriginalFile != null) {
                    if (movePostSend) {
                        fileOriginalFile.renameTo(new File(fileOriginalFile + movePostSendSuffix));
                    } else if (deletePostSend) {
                        fileOriginalFile.delete();
                    }
                }
            }
        }

        //Override more interceptor methods to capture some logs here
    };
    MessageChannel messageChannel = source.output();
    ((DirectChannel) messageChannel).addInterceptor(channelInterceptor);
    return FileUtils.enhanceFlowForReadingMode(flowBuilder, this.fileConsumerProperties)
            .channel(messageChannel)
            .get();
}

private void initializePostSendAction() {
    deletePostSend = this.properties.isDeletePostSend();
    movePostSend = this.properties.isMovePostSend();
    movePostSendSuffix = this.properties.getMovePostSendSuffix();

    if (deletePostSend && movePostSend) {
        String errorMessage = "The 'delete-file-post-send' and 'move-file-post-send' attributes are mutually exclusive";
        throw new IllegalArgumentException(errorMessage);
    }

    if (movePostSend && (movePostSendSuffix == null || movePostSendSuffix.trim().length() == 0)) {
        String errorMessage = "The 'move-post-send-suffix' is required when 'move-file-post-send' is set to true.";
        throw new IllegalArgumentException(errorMessage);
    }

    //Add additional validation to ensure the user didn't configure a file move that will result in cyclic processing of file
}

private FileListFilter<File> initializeFileListFilter() {

    final List<FileListFilter<File>> filtersNeeded = new ArrayList<FileListFilter<File>>();

    if (this.properties.getFilenamePattern() != null && this.properties.getFilenameRegex() != null) {
        String errorMessage = "The 'filename-pattern' and 'filename-regex' attributes are mutually exclusive.";
        throw new IllegalArgumentException(errorMessage);
    }

    if (StringUtils.hasText(this.properties.getFilenamePattern())) {
        SimplePatternFileListFilter patternFilter = new SimplePatternFileListFilter(this.properties.getFilenamePattern());
        if (this.alwaysAcceptDirectories != null) {
            patternFilter.setAlwaysAcceptDirectories(this.alwaysAcceptDirectories);
        }
        filtersNeeded.add(patternFilter);
    } else if (this.properties.getFilenameRegex() != null) {
        RegexPatternFileListFilter regexFilter = new RegexPatternFileListFilter(this.properties.getFilenameRegex());
        if (this.alwaysAcceptDirectories != null) {
            regexFilter.setAlwaysAcceptDirectories(this.alwaysAcceptDirectories);
        }
        filtersNeeded.add(regexFilter);
    }

    FileListFilter<File> createdFilter = null;

    if (!Boolean.FALSE.equals(this.properties.isIgnoreHiddenFiles())) {
        filtersNeeded.add(new IgnoreHiddenFileListFilter());
    }

    if (Boolean.TRUE.equals(this.properties.isPreventDuplicates())) {
        filtersNeeded.add(new AcceptOnceFileListFilter<File>());
    }

    if (filtersNeeded.size() == 1) {
        createdFilter = filtersNeeded.get(0);
    } else {
        createdFilter = new CompositeFileListFilter<File>(filtersNeeded);
    }

    return createdFilter;
}

}

春季云流文件源应用程序-子目录

2 个答案: