Question

我正在使用Spring Integration RecursiveDirectoryScanner递归扫描目录，以处理将被放置在配置目录（/ home / test）下的传入文件。

我经常遇到以下错误：

    ERROR org.springframework.integration.handler.LoggingHandler - java.lang.IllegalArgumentException: java.nio.file.FileSystemException: /home/test: Too many open files
        at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:89)
        at org.springframework.integration.file.FileReadingMessageSource.scanInputDirectory(FileReadingMessageSource.java:387)
        at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:361)
        at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:90)
        at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:134)
        at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:224)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Caused by: java.nio.file.FileSystemException: /home/test: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
        at java.nio.file.Files.newDirectoryStream(Files.java:457)
        at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
        at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
        at java.nio.file.Files.walk(Files.java:3574)
        at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:73)

我的Spring Integration流程如下：

XML配置

 <task:executor id="pollerPool"
        pool-size="${pollerThreadPoolSize}"
        queue-capacity="${pollerThreadQueueCapacity}" rejection-policy="ABORT" />

    <task:executor id="fileHandlerPool"
        pool-size="${fileHandlerPoolSize}"
        queue-capacity="${fileHandlerPoolThreadQueueCapacity}" rejection-policy="CALLER_RUNS" />


     <bean id="iFilter" class="org.springframework.integration.file.filters.ChainFileListFilter">
        <constructor-arg>
             <list>
                  <bean id="lastModifiedFileListFilter" class="org.springframework.integration.file.filters.LastModifiedFileListFilter">
                     <property name="age" value="120" />
                  </bean>
                 <ref bean="acceptOnceFileListFilter"/>

                 <bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
                    <constructor-arg value="^.*\.(txt|csv|xls|xlsx|asc)$"/>
                 </bean>

            </list>
        </constructor-arg>
      </bean>

     <bean id="acceptOnceFileListFilter" name="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" primary="true" />

     <bean id="recursiveDirectoryScanner" class="org.springframework.integration.file.RecursiveDirectoryScanner">
        <property name="filter" ref="iFilter" />
        <property name="locker" ref="nioFileLocker" />
     </bean>

    <bean id="nioFileLocker" class="org.springframework.integration.file.locking.NioFileLocker" />

    <int-file:inbound-channel-adapter
        id="fileSource" channel="fileReceivedChannel" auto-startup="true"
        directory="file:${polling.directory}" 
        scanner="recursiveDirectoryScanner"  >
        <int:poller task-executor="pollerPool"
            fixed-rate="${pollerFixedRate}"
            receive-timeout="${pollerReceiveTimeout}">
        </int:poller>
    </int-file:inbound-channel-adapter>

动态参数如下：

polling.directory = / home / test pollerThreadPoolSize = 1 pollerThreadQueueCapacity = 10 pollerFixedRate = 5000 pollerReceiveTimeout = 5000 fileHandlerPoolSize = 2 fileHandlerPoolThreadQueueCapacity = 100

编辑：

我确实在选择文件时出现的服务激活器中解锁文件。我从文件中获取了一些信息并将其解锁。

@Autowired
  NioFileLocker nioFileLocker;

   protected void doTransform(Message<?> message) throws Exception {
  MessageBuilder<File> payload = (MessageBuilder<File>) message.getPayload();
    File inFile = payload.getPayload();
   try {
      nioFileLocker.unlock(inFile);
    } catch (Exception e) {
      LOGGER.error("file not unlock");
    }
 }

配置是否存在问题？如何确保不再出现该异常？

谢谢。

Answer 1

我建议您在没有NioFileLocker的情况下测试您的解决方案。看起来好像不是用它来解锁文件，但是lock(File fileToLock)确实在OS中保留了一些文件标记。

另一方面，文件柜在UNIX系统上无法可靠运行。它仍然允许访问文件。至少是为了阅读。

为了更好地进行独占文件访问，我建议将FileSystemPersistentAcceptOnceFileListFilter与外部MetadataStore一起使用，而不要在内存AcceptOnceFileListFilter中使用。这样，您的应用程序只有一个实例可以访问该文件，并且根本不会再对其进行处理。

Spring Integration RecursiveDirectoryScanner提供太多文件打开异常

1 个答案: