在Java中并行或异步下载多个文件

时间:2015-10-12 07:12:50

标签: java multithreading file asynchronous java-6

这里我试图一个接一个地下载多个文件:

环境 - Java 1.6

public List<Attachment> download(List<Attachment> attachments)
{
  for(Attachment attachment : attachments) {
    attachment.setDownStatus("Failed");
    String destLocation = "C:\Users\attachments";
    try {
        String attUrl = attachment.getUrl();
        String fileName = attachment.getFileName();            
        URL url = new URL(attUrl);
        File fileLocation = new File(destLoc, fileName);
        FileUtils.copyURLToFile(url, fileLocation);
        if(fileLocation.exists()) {
           attachment.setDownStatus("Completed");
         }
       } catch(Exception e) {
          attachment.setDownStatus("Failed");
       } finally {
          attachment.setDestLocation(destLocation);
       }
   }
  return attachments;
}

我正在从提供的网址(http://cdn.octafinance.com/wp-content/uploads/2015/07/google-hummingbird.jpg)下载文件。

FileUtils.copyURLToFile(url, fileLocation);

上面的代码完美地完成了下载工作,没有任何问题。

我的问题:
如果附件列表更多,则需要更多时间,因此我希望将其设置为异步或并行进程,而不是按顺序下载。

3 个答案:

答案 0 :(得分:4)

将Java 8 Streams与Timestamp initialization

结合使用
public List<Attachment> download(List<Attachment> attachments){

    ForkJoinPool forkJoinPool = new ForkJoinPool(attachments.size());

    return forkJoinPool.submit(() -> processAttachments(attachments).get();

}

private List<Attachment> processAttachments(List<Attachment> attachments) {
    return attachments.stream().parallel().map(attachment -> processSingleAttachment(attachment).collect(Collectors.toList());
}

private Attachment processSingleAttachment(Attachment attachment){
     //business logic to download single attachment
    .
    .
}

答案 1 :(得分:3)

public List<Attachment> download(List<Attachment> attachments)
{
  ExecutorService executorService = Executors.newCachedThreadPool();
  for(final Attachment attachment : attachments){
    executorService.submit(new Runnable() {

        @Override
        public void run() {
          try{
            String attUrl = attachment.getUrl();
            String fileName = attachment.getFileName();
            String destLocation = "C:\Users\attachments";
            URL url = new URL(attUrl);
            String fileLocation = new File(destLoc, fileName);
            FileUtils.copyURLToFile(url, fileLocation);
            if(fileLocation.exists()) {
              attachment.setDownStatus("Completed");
            }
          }
          catch(Exception e){
            attachment.setDownStatus("Failed");
          }
        }
    });
 }
 executorService.shutdown();
 return attachments;
}

答案 2 :(得分:1)

实际上,仔细观察之后,鲍里斯的代码有问题,有时候肯定不会设置一些东西。这是一个更好的版本,修复了:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>jQuery UI Datepicker - Default functionality</title>
  <link rel="stylesheet" href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css">
  <script src="//code.jquery.com/jquery-1.10.2.js"></script>
  <script src="//code.jquery.com/ui/1.11.4/jquery-ui.js"></script>
  <script>
  $(function() {
    $( "#datepicker" ).datepicker();
  });
  </script>
</head>
<body>
 
<p>Date: <input type="text" id="datepicker"></p>
 
 
</body>
</html>

但是,考虑到public List<Attachment> download(List<Attachment> attachments) { ExecutorService executorService = Executors.newCachedThreadPool(); List<Future<Attachment>> futures = new ArrayList<Future<Attachment>>(); for (final Attachment attachment : attachments) { futures.add(executorService.submit(new Callable<Attachment>() { @Override public Attachment call() throws Exception { return doDownload(attachment); } })); } for (Future<Attachment> future: futures) { try { future.get(); } catch (Exception ex) { // Do something } } return attachments; } private Attachment doDownload(Attachment attachment) throws Exception { attachment.setDownStatus("Failed"); attachment.setDestLocation("C:\\Users\\attachments"); String attUrl = attachment.getUrl(); String fileName = attachment.getFileName(); URL url = new URL(attUrl); File fileLocation = new File(attachment.getDestLocation(), fileName); FileUtils.copyURLToFile(url, fileLocation); if (fileLocation.exists()) { attachment.setDownStatus("Completed"); } return attachment; } 的结构以及如何使用它,这绝对不是最佳选择。我没有解决这个问题:我只回答了问题。