Question

下面的代码显示我在文件夹中一次读取一个文件。我想一次读取10个文件，通过METHOD_A运行并写入另一个文件夹。文件夹中有5000个文件。一次读取文件1非常慢。我想加快速度。

我正在使用Java Spring Boot。有什么建议我怎么办？

for (int i = 0; i < files.length; i++){
     Object obj = parser.parse(new FileReader(files[i]));
     JSONObject obj1 = METHOD_A(obj);

        try{
            PrintWriter writer = new PrintWriter(...);
            writer.println(obj1);
            writer.close();
        } catch (IOException e) {
            // do something
    }
}

提前谢谢。

Answer 1

是的，你可以！，例子;

    ExecutorService executorService = Executors.newFixedThreadPool(/*Number of Threads*/);


    for (int i = 0; i < /* Number of Requests */; i++) {
        WorkerThread wt = context.getBean(WorkerThread.class, String.valueOf(i));
        tasks.add(wt);
    }

    List<Future<String>> futures = executorService.invokeAll(tasks);

    for (Future<String> future : futures) {
        String result = future.get(10000, TimeUnit.MILLISECONDS);
    }

    executorService.shutdown();

Answer 2

根据您的要求，我已为您添加以下代码。试试这个，看看是否适合你。请记住，您的服务器应具有足够数量的处理器以及并发进程。如果你仍然混淆，请在最后看到一个示例程序： -

    private int numberOfFileProcessed = 0;
    private int numberOfThreadAlive = 0 ;
    private int numberOfThreadAlive = 0;
    private int numberOfThreadToBeAllowed = 10;//Change this value to control number of thread to be run concurrently

    for (int i = 0; i < files.length; i++){
     Object obj = parser.parse(new FileReader(files[i]));
     JSONObject obj1 = METHOD_A(obj);
     try{
            Thread t = new Thread(new ReadFiles(obj1));
            t.start();
            numberOfThreadAlive++;
        }catch (Exception e) {
        //do something  
       }

      while(numberOfThreadAlive > numberOfThreadToBeAllowed){//This while loop will control number of thread to be not more than 10
        try{Thread.sleep(100);}catch(Exception e){}//Release the processor    
      }
  }

  private final synchronized void jobCompleted(){       
            numberOfFileProcessed++;
            numberOfThreadAlive--;      
  }

  while(numberOfFileProcessed < files.length){
    //wait till last thread complete it's task
    //I am not using thread.join() for performance 
    try{Thread.sleep(100);}catch(Exception e){}//Release the processor
  }



private class ReadFiles implements Runnable  {
        JSONObject jobj;
        public ReadFiles(JSONObject obj) {
            jobj = obj;
        }
        @SuppressWarnings("unchecked")
        public void run() {
          try{      
            PrintWriter writer = new PrintWriter(...);
            writer.println(jobj);
            writer.close();
            jobCompleted();
           } catch (IOException e) {
            // do something
          }
        }

    }

以下是可用于理解的测试文件

        package com.test.threadtest;

    public final class ThreadTest {
        private int numberOfFileProcessed = 0;
        private int numberOfThreadAlive = 0 ;   
        int numberOfThreadToBeAllowed = 10;

        public void processFiles(){

            for (int i = 0; i < 50; i++){
                try{
                      Thread t = new Thread(new ReadFiles(i));
                      t.start();
                      numberOfThreadAlive++;
                 }catch (Exception e) {
                    //do something  
                 }

                  while(numberOfThreadAlive > numberOfThreadToBeAllowed){//This while loop will control number of thread to be not more than 10
                    try{Thread.sleep(100);}catch(Exception e){}//Release the processor
                    System.out.println("Reached maximum");
                  }
            }  
              while(numberOfFileProcessed < 50){
                //wait till last thread complete it's task
                //I am not using thread.join() for performance 
                  System.out.println("Number of file processed :" + numberOfFileProcessed);
                try{Thread.sleep(100);}catch(Exception e){}
              }
        }

        private final synchronized void jobCompleted(){     
                numberOfFileProcessed++;
                System.out.println("numberOfFileProcessed :" + numberOfFileProcessed);
                numberOfThreadAlive--;      
        }

        public static void main(String[] args) {
            // TODO Auto-generated method stub
            ThreadTest test = new ThreadTest();
            test.processFiles();
            System.out.println("Exit from the process");
            System.exit(0);
        }

        private class ReadFiles implements Runnable  {
            int i;
            public ReadFiles(int val) {
                i = val;
            }
            @SuppressWarnings("unchecked")
            public void run() {
              try{      
                System.out.println("I am Thread : " + i);
                Thread.sleep(1000);
                jobCompleted();
               } catch (Exception e) {
                // do something
              }
            }

        }
    }

Answer 3

对于CPU限制的工作，应该考虑管理线程和任务，这不是您的情况。对于输入/输出，应改为使用非阻塞IO，例如AsynchronousFileChannel。 RxIo简化了AFC的使用，您可以通过AsyncFiles API来实现自己的目标，例如：

AsyncFiles
      .readAll(files[i])
      .thenApply(parser::parse)
      .thenApply(obj -> obj.toString().getBytes())
      .thenCompose(bytes -> AsyncFiles.writeBytes(..., bytes))
      .thenAccept(index ->     /* invoked on completion */)
      .exceptionally(excep -> /* invoked on error*/

通常，要管理异步IO，应使用非阻塞IO，这样可以避免显式管理线程池和阻塞线程。既然接受的答案建议正在阻止线程获取结果，那么您将在newFixedThreadPool(size)中配置多少个线程来实现并发？

也许，线程池大小可以等于正在读取的文件数。您可能需要10个？

但是随后在WorkerThread提议中，您在读取和解析时阻塞了线程。我不知道parser的类型，但是由于您将其设置为阻塞FileReader，因此parser.parse(…)肯定会阻塞。最后，您将在writer.println()上再次屏蔽。

下图显示了与“ 同步，两个控制线程”相对应的已接受答案提议与与“ 异步”相对应的我的提议之间的区别

eloquent javascript

在Spring Boot中异步读取10个文件

3 个答案: