Question

我正在使用PoolingHttpClientConnectionManager版httpclient中的4.5.3来获取URL数组。我按照Apache httpcomponents的tutorial第2.4节中描述的示例进行操作。但是，我在使用以下代码时遇到了堆内存不足的问题：

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
        .setConnectionManager(cm)
        .build();

// URIs to perform GETs on
String[] urisToGet = {
    "http://www.domain1.com/a",
    "http://www.domain1.com/b",
    "http://www.domain1.com/c",
    "http://www.domain1.com/d",
    // thousands more
    // ....
};

// create a thread for each URI
GetThread[] threads = new GetThread[urisToGet.length];
for (int i = 0; i < threads.length; i++) {
    HttpGet httpget = new HttpGet(urisToGet[i]);
    threads[i] = new GetThread(httpClient, httpget);
}

// start the threads
for (int j = 0; j < threads.length; j++) {
    threads[j].start();
}

// join the threads
for (int j = 0; j < threads.length; j++) {
    threads[j].join();
}

作为参考，GetThread如下：

static class GetThread extends Thread {

    private final CloseableHttpClient httpClient;
    private final HttpContext context;
    private final HttpGet httpget;

    public GetThread(CloseableHttpClient httpClient, HttpGet httpget) {
        this.httpClient = httpClient;
        this.context = HttpClientContext.create();
        this.httpget = httpget;
    }

    @Override
    public void run() {
        try {
            CloseableHttpResponse response = httpClient.execute(
                    httpget, context);
            try {
                HttpEntity entity = response.getEntity();
            } finally {
                response.close();
            }
        } catch (ClientProtocolException ex) {
            // Handle protocol errors
        } catch (IOException ex) {
            // Handle I/O errors
        }
    }

}

我尝试将连接管理器传递到每个线程中，并在每个线程的finally部分中关闭过期的连接：

this.connectionManager.closeExpiredConnections();
this.connectionManager.closeIdleConnections(30, TimeUnit.SECONDS);

但是，这不能解决问题。

也许是因为我创建了太多线程吗？我需要改用ThreadPoolExecutor吗？

如何使用PoolingHttpClientConnectionManager来管理数千个GET请求？

0 个答案: