Question

我正在使用apache commons 4.x HTTPClient对URI进行HEAD请求，以获取该链接的最终post 302 URL位置。例如：http://bit.ly/test1231真的指向cnn.com或其他东西。什么是使用HttpClient在服务器中实现这一目标的最佳和最有效的方法，该服务器可以运行数月而不会泄漏？现在我遇到的问题是每隔x分钟所有线程冻结，同时尝试从池中拉出连接并且它们都会超时。

我计划让100个工作线程进行提取，所以我使用的是Threaded连接管理器。

UPDATE这是我用来获取httpClient对象的代码

HttpParams httpParams = new BasicHttpParams();

HttpConnectionParams.setConnectionTimeout(httpParams, 5000);

HttpConnectionParams.setSoTimeout(httpParams, 5000);

ConnManagerParams.setMaxTotalConnections(httpParams, 5000);

HttpProtocolParams.setVersion(httpParams, HttpVersion.HTTP_1_1);



ConnManagerParams.setMaxConnectionsPerRoute(httpParams, new ConnPerRoute() {

   @Override

   public int getMaxForRoute(HttpRoute route) {

     return 35;

   }

 });

emptyCookieStore = new CookieStore() {

    @Override

    public void addCookie(Cookie cookie) {



    }

    ArrayList<Cookie> emptyList = new ArrayList<Cookie>();



    @Override

    public List<Cookie> getCookies() {

      return emptyList;

    }

    @Override

    public boolean clearExpired(Date date) {

      return false;

    }



    @Override

    public void clear() {

    }

  };



  // set request params

  httpParams.setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY);

  httpParams.setParameter("http.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)");

  httpParams.setParameter("http.language.Accept-Language", "en-us");

  httpParams.setParameter("http.protocol.content-charset", "UTF-8");

  httpParams.setParameter("Accept", "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");

  httpParams.setParameter("Cache-Control", "max-age=0");

  SchemeRegistry schemeRegistry = new SchemeRegistry();

  schemeRegistry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));

  schemeRegistry.register(new Scheme("https", PlainSocketFactory.getSocketFactory(), 443));

  final ClientConnectionManager cm = new ThreadSafeClientConnManager(httpParams,schemeRegistry);



  DefaultHttpClient httpClient = new DefaultHttpClient(cm, httpParams);

  httpClient.getParams().setParameter("http.conn-manager.timeout", 120000L);

  httpClient.getParams().setParameter("http.protocol.wait-for-continue", 10000L);

  httpClient.getParams().setParameter("http.tcp.nodelay", true);

Answer 1

很可能你有太多的工作线程竞争很少的连接。请确保将每个路由限制的最大连接数设置为合理的值（默认情况下，根据HTTP规范的要求将限制设置为两个并发连接）

Apache HTTPClient HEAD请求的最高性能？

1 个答案: