Question

我目前正处于一个项目中，我必须对休息服务执行多个并发的http请求，这些服务会返回JSON响应。这是一个批处理操作，任何时候请求的数量可以从几个hunderd到几个thousend。

这就是为什么我认为拥有一个异步http客户端以便我可以拥有并发请求的原因，这可以大大加快这个过程。我首先尝试了ning的async-http-client。也许我做错了什么，因为这对我来说有点慢。 1000个请求大约需要10秒。

之后我尝试了Apache的实现，在1000个请求的大约4秒内更快。但我似乎无法获得稳定的要求。大多数时候我会得到一个包含1000个回复的List（就像我期望的那样），但有时候我只是缺少一些回复，比如1或2。

这是我目前的代码：

public class AsyncServiceTest {
    public AsyncServiceTest(String serviceURI) {
        this.httpClient = HttpAsyncClients.custom().setMaxConnPerRoute(100).setMaxConnTotal(20)
                .setDefaultRequestConfig(RequestConfig.custom().build()).build();

        this.objectMapper = new ObjectMapper();
        this.serviceURI = serviceURI;
    }

    private List<Object> getResults(List<String> queryStrings) throws Exception {
        try {
            httpClient.start();

            final List<HttpGet> requests = new ArrayList<>(addresses.size());
            for (String str : queryStrings) {
                requests.add(new HttpGet(buildUri(str))); // In this method we build the absolute request uri.
            }

            final CountDownLatch latch = new CountDownLatch(requests.size());
            final List<Object> responses = new ArrayList<>(requests.size());
            final List<String> stringResponses = new ArrayList<>(requests.size());

            for (final HttpGet request : requests) {
                httpClient.execute(request, new FutureCallback<HttpResponse>() {
                    @Override
                    public void completed(HttpResponse response) {
                        try {
                            stringResponses.add(IOUtils.toString(response.getEntity().getContent(), "UTF-8"));
                            latch.countDown();
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    }

                    @Override
                    public void failed(Exception e) {
                        latch.countDown();
                    }

                    @Override
                    public void cancelled() {
                        latch.countDown();
                    }
                });
            }

            latch.await();

            for (String r : stringResponses) {
                responses.add(mapToLocation(r)); // Mapping some Strings to JSON in this method.
            }

            return responses;
        } finally {
            httpClient.close();
        }
    }
}

所以，从本质上讲，我想知道我的代码（可能）是否有问题，还是只是因为库的工作方式？因为CountDownLatch始终为零。或者是否有人指向正确的方向（可能是另一个库）？

Answer 1

在我的代码中，它似乎是一个并发问题（感谢@vanOekel）。答案是用ArrayList<E>替换Vector<E>，这实际上是线程安全的。示例代码：

public class AsyncServiceTest {
    public AsyncServiceTest(String serviceURI) {
        this.httpClient = HttpAsyncClients.custom().setMaxConnPerRoute(100).setMaxConnTotal(20)
                .setDefaultRequestConfig(RequestConfig.custom().build()).build();

        this.objectMapper = new ObjectMapper();
        this.serviceURI = serviceURI;
    }

    private List<Object> getResults(List<String> queryStrings) throws Exception {
        try {
            httpClient.start();

            final CountDownLatch latch = new CountDownLatch(queryStrings.size());
            final Vector<Object> responses = new Vector<>(queryStrings.size());

            for (String str : queryStrings) {
                // buildUri: In this method we build the absolute request uri.
                httpClient.execute(new HttpGet(buildUri(str)), new FutureCallback<HttpResponse>() {
                    @Override
                    public void completed(HttpResponse response) {
                        try {
                            // mapToLocation: Mapping some Strings to JSON in this method.
                            responses.add(mapToLocation(IOUtils.toString(response.getEntity().getContent(), "UTF-8")));
                            latch.countDown();
                        } catch (IOException e) {
                            failed(e);
                        }
                    }

                    @Override
                    public void failed(Exception e) {
                        logger.error(e.getLocalizedMessage(), e);
                        latch.countDown();
                    }

                    @Override
                    public void cancelled() {
                        logger.error("Request cancelled.");
                        latch.countDown();
                    }
                });
            }

            latch.await();
            return responses;
        } finally {
            httpClient.close();
        }
    }
}

感谢所有有用的回复。如果有人对上述代码的优化有任何建议，我将很高兴听到。

Apache的HttpAsyncClient与多个请求不可靠

1 个答案: