CompletableFuture.join() to block list of CompletableFutures

时间:2017-12-18 05:45:35

标签: java multithreading java-8 future completable-future

I have an embarrassingly-parallel problem and in my design, I have a couple of worker nodes doing the calculation. Each node exposes its logic using REST API.

My master node distributes the logic to all the nodes using HTTP call to the RESTAPI and waits (barrier) till all nodes return its result.When all the nodes return the result, I gather them and calculate the final result.

My code is as follows, I use CompletableFuture.join() to block the code until all the nodes return their logic. I have an executor which is configured with a pool of 50 threads,but looking at the response times, instead of calling nodes in parallel it seems that each node gets executed in sequence.

Am I using join() correctly in the case ?

//Do the logic in each node parallely
Map<CompletableFuture<NodeResultsResponse>,String> allRes = new HashMap<>();
System.out.println("*** Cluster Logic on "+nodeContext.getIp());

for(String ip : clusterContext.getIps()) {
  System.out.println("***  calling on node IP "+ip);
  //this will make an REST call to node IP
  CompletableFuture<NodeResultsResponse> res = helperScanOnNode(ip, jobId);
  allRes.put(res,ip);
}

//Wait(block) till all nodes return result
CompletableFuture.allOf(allRes.keySet().toArray(new CompletableFuture[allRes.size()])).join();

List<NodeDMatch>> allResults = new HashMap<>();

//aggregate results from each node
for(CompletableFuture<NodeResultsResponse> eachResult : allRes.keySet()) {
  try {
    NodeResultsResponse r = eachResult.get();
    allResults.add(r.getDResult);
  } catch (Exception e) {
    //
  }
}
calculateResult(allResults);

the method which calls each node in a separate thread."executor" is defined with a threadpool with 10 max threads.

private CompletableFuture<NodeResultsResponse> helperScanOnNode(String ip, String jobId) {

        CompletableFuture<NodeResultsResponse> cfr = CompletableFuture.supplyAsync(()->{
                    String url = CommonUtils.getURL(ip)+"/node/scan/"+jobId;
                    try {

                        ResponseEntity<NodeResultsResponse> res = restTemplate.exchange
                                (url, HttpMethod.POST, new HttpEntity<NodeResultsResponse>(createHeaders()), NodeResultsResponse.class);
                        return res.getBody();
                    } catch (Exception e) {
                        e.printStackTrace();;
                        return null;
                    }

                },
                executor);
        return cfr;

}

0 个答案:

没有答案