Question

我有一个字符串列表，对于列表中的每个项目（基本上是一个文件名），我必须进行一些处理。我有按顺序进行的流程，但是我想将其转换为CompleatableFuture，在其中我必须更新许多列表。我该怎么办？

这是我的原始代码

Path fileLocation = findLocationForFiles(fileDTSuffix);
Map<String, Map<String, String>> csvMap = getCsvToMap("filename", fileLocation);
List<String> filesMissingCSV = new ArrayList<>();
List<String> expiredAssets = new ArrayList<>();

int filesUploaded = 0;
int filesProcessed = 0;
//TODO : Convert below to use CompletableFuture to perform the actions in thread

for (String fileName : filesToProcessList) {
    if(filesProcessed > 100){
        System.out.println("Processed 100 files - calling cleanupTempDir ..... ");
        cleanupTempDir(fileLocation);
        filesProcessed = 0;
    }
    Path path = Paths.get(fileName);

    File file = new File(fileName);
    String baseFileName = file.getName();
    String translatedNameFromVW = translateFilenameFromVW(baseFileName);
    String translatedNameFromVWSansConfig = translateFilenameFromVWSansConfig(baseFileName);
    String translatedNameToVW = translateFilenameToVW(baseFileName);
    boolean isFilenameEqFromVW = baseFileName.equalsIgnoreCase(translatedNameFromVW);
    boolean isFilenameEqToVW = baseFileName.equalsIgnoreCase(translatedNameToVW);
    Map<String, String> csvFileInfo = getCSVMapAttributes(csvMap, baseFileName, translatedNameFromVW, translatedNameFromVWSansConfig, translatedNameToVW);

    if (null == csvFileInfo) {
        filesMissingCSV.add(fileName + " | " + file.getName());
    } else {
        if (!isAssetExpired(csvFileInfo)) {
            AssetMaster am = generateJSON(fileName, csvFileInfo);
            addToCassandra(path, am, translatedNameToVW);
            printJson(am, fileLocation);
            filesUploaded++;
            filesProcessed++;
        } else {
            expiredAssets.add(fileName + " | " + file.getName());
        }
    }
}

我正在尝试将其转换为使用CompletableFutures，如下所示

List<CompletableFuture> asyncThreadsList = new ArrayList();

for (String fileName : filesToProcessList) {
    asyncThreadsList.add(CompletableFuture.runAsync(() -> {
        try {
            processEachFile(csvMap, fileName, fileLocation);
        } catch (IOException ex) {
            Logger.getLogger(UploadThreaded.class.getName()).log(Level.SEVERE, null, ex);
        }
    }));
}
CompletableFuture.allOf(asyncThreadsList.toArray(new CompletableFuture[0]))
   .thenRunAsync(() -> System.out.println("Ended doing things"));

我对processEachFile(csvMap, fileName, fileLocation)的定义如下

private static CompletableFuture<Void> processEachFile(Map<String, Map<String, String>> csvMap, String fileName, Path fileLocation) throws IOException {

    if (filesProcessed > 100) {
        System.out.println("Processed 100 files - calling cleanupTempDir ..... ");
        cleanupTempDir(fileLocation);
        filesProcessed = 0;
    }
    Path path = Paths.get(fileName);

    File file = new File(fileName);
    String baseFileName = file.getName();
    String translatedNameFromVW = translateFilenameFromVW(baseFileName);
    String translatedNameFromVWSansConfig = translateFilenameFromVWSansConfig(baseFileName);
    String translatedNameToVW = translateFilenameToVW(baseFileName);
    boolean isFilenameEqFromVW = baseFileName.equalsIgnoreCase(translatedNameFromVW);
    boolean isFilenameEqToVW = baseFileName.equalsIgnoreCase(translatedNameToVW);
    Map<String, String> csvFileInfo = getCSVMapAttributes(csvMap, baseFileName, translatedNameFromVW, translatedNameFromVWSansConfig, translatedNameToVW);

    if (null == csvFileInfo) {
        filesMissingCSV.add(fileName + " | " + file.getName());
    } else {
        if (!isAssetExpired(csvFileInfo)) {
            AssetMaster am = generateJSON(fileName, csvFileInfo);
            addToCassandra(path, am, translatedNameToVW);
            printJson(am, fileLocation);
            filesUploaded++;
            filesProcessed++;
        } else {
            expiredAssets.add(fileName + " | " + file.getName());
        }
    }
    return null;
}

我的问题是我需要合并每次运行中的这些列表：filesMissingCSV，expiredAssets，还需要更新每次运行中的计数filesUploaded和filesProcessed以获得最终结果价值观。而且我很难确定应该如何实现。

是否可以做到？应该如何实现？

谢谢

Answer 1

@adbdkb评论了对于目录，我实际上有一个外部循环，该循环用于每个目录中的文件。使用与此处所示相同的机制，我也可以为外循环创建CompletableFutures列表吗？

因此，我决定提供一个简单的解决方案，以解决主要OP的问题。

首先，我创建了一个类来处理每个文件的结果。

class ProcessDetail {
  private int fileUploaded;
  private int fileProcessed;
  private List<String> expireFile = new ArrayList<>();
  private List<String> missingFile = new ArrayList<>();
}

也有一些方法可以简化任务。第一种方法是processEachDir。我只是传递一个字符串列表作为代表目录文件的参数。此方法返回List<CompletableFuture<ProcessDetail>>。

private static List<CompletableFuture<ProcessDetail>> processEachDir(List<String> dir) {
    System.out.println(dir + " time: " + LocalDateTime.now());
    sleep(1);
    return dir.stream()
            .map(fileName -> processEachFile(fileName))
            .collect(Collectors.toList());
}

第二种方法是processEachFile，它获取代表文件名的字符串参数。此方法返回CompletableFuture<ProcessDetail>

private static CompletableFuture<ProcessDetail> processEachFile(String fileName) {
    System.out.println(fileName + " time: " + LocalDateTime.now());

    return CompletableFuture.supplyAsync(() -> {
        ProcessDetail processDetail = new ProcessDetail();
        if (csvFileInfo()) {
            processDetail.getMissingFile().add(fileName);
        } else if (!isAssetExpired()) {
            processDetail.setFileProcessed(1);
            processDetail.setFileUploaded(1);
        } else {
            processDetail.getExpireFile().add(fileName);
        }
        return processDetail;
    });
}

现在该完成任务了：为每个目录下面的代码创建CompletableFuture<ProcessDetail>的列表，并处理文件和其他内容。并将其结果放入以目录名称为键的map中。 processDetails.put(dir.getKey(), result);

directories.entrySet().forEach(dir -> {
        List<CompletableFuture<ProcessDetail>> list = processEachDir(dir.getValue());
        CompletableFuture.allOf(list.toArray(new CompletableFuture<?>[0]))
                .thenAccept(nothing -> {
                    ProcessDetail result = list.stream()
                            .map(CompletableFuture::join).reduce(new ProcessDetail(), merge);
                    processDetails.put(dir.getKey(), result);

                });
});

DEMO

循环转换处理，将许多列表更新为CompletableFuture

1 个答案: