Java程序在循环中发出HttpGet请求时卡住了

时间:2014-09-11 05:22:21

标签: java network-programming httpclient web-crawler

我必须在循环中点击公共函数(downloadDataQUANDL)大约1500次才能下载数据。 但是我的程序在运行大约50-100个网络呼叫后,被随机卡在一个网址上。 不确定为什么它会被卡住,因为定义了最大重试次数。因此,在耗尽所有尝试后,程序应跳过并移至下一个URL。

有什么想法来解决这个问题吗?请帮忙

public List<Record> downloadDataQUANDL(String symbol){
        List<Record> records = new ArrayList<Record>();
        Calendar fromdate = Calendar.getInstance();
        fromdate.add(Calendar.DATE, -365);
        Date todate = new Date();
        String url = getFeederURLQuandl(symbol, new Date(fromdate.getTimeInMillis()), todate);
        InputStreamReader inputStreamReader = null;
        BufferedReader bufferedReader = null;
        try{
            HttpResponse response = retryQuandl(url);
            if(response.getStatusLine().getStatusCode() == 200){
                inputStreamReader = new InputStreamReader(response.getEntity().getContent());
                bufferedReader = new BufferedReader(inputStreamReader);
                String line;
                while ((line = bufferedReader.readLine()) != null) {
                    if (!Character.isDigit(line.charAt(0))) {
                        continue;
                    }

                    String[] values = line.split(",");
                    Date date = (Date) new SimpleDateFormat("yyyy-MM-dd").parse(values[0]);

                    records.add(new Record(date, Double.parseDouble(values[1]),
                        Double.parseDouble(values[2]), Double.parseDouble(values[3]),
                        Double.parseDouble(values[5]), Double.parseDouble(values[6])));
                }

            }else{
                System.out.println("Data fetch failed for " + symbol);
                return null;
            }
        }catch(Exception e){
            System.out.println("Data fetch failed for " + symbol);
            return null;
        }
        return records;
    }


   private HttpResponse retryQuandl(String url){
        RequestConfig requestConfig =           RequestConfig.custom().setSocketTimeout(30*1000).setConnectTimeout(30*1000).build();
        HttpClient client = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).build();
        HttpGet request = new HttpGet(url);
        System.out.println(url);
        int responsecode=0;
int nooftries = 1;
        HttpResponse response=null;
        while(responsecode != 200 && nooftries <= 5){
            try{
                response = client.execute(request);
                responsecode = response.getStatusLine().getStatusCode();
            }catch(Exception e){}
            try {
                Thread.sleep(nooftries * 1000);
            } catch (InterruptedException e) {}
            nooftries++;
        }

        return response;
    }

0 个答案:

没有答案