Question

我使用以下代码来获取纽约时报页面的html，不幸的是，这将返回null。我曾尝试过其他网站（美国有线电视新闻网，卫报等）并且工作正常。我正在使用Google App Engine中的URLFetchService。

这是代码段。请告诉我我做错了什么？

//url = https://www.nytimes.com/2017/05/02/us/politics/health-care-paul-ryan-fred-upton-congress.html

private String extractFromUrl(String url, boolean forced) throws java.io.IOException, org.xml.sax.SAXException,
                      de.l3s.boilerpipe.BoilerpipeProcessingException  {

    Future<HTTPResponse> urlFuture = getMultiResponse(url);

    HTTPResponse urlResponse = null;
    try {
        urlResponse = urlFuture.get(); // Returns null here
    } catch ( InterruptedException ie ) {
        ie.printStackTrace();
    } catch ( ExecutionException ee ) {
        ee.printStackTrace();
    }

    String urlResponseString = new String(urlResponse.getContent());
    return urlResponseString;
}

public Future<HTTPResponse> getMultiResponse(String website) {
    URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
    URL url = null;
    try {
        url = new URL(website);
    } catch (MalformedURLException e) {
        e.printStackTrace();
    }

    FetchOptions fetchOptions = FetchOptions.Builder.followRedirects();
    HTTPRequest request = new HTTPRequest(url, HTTPMethod.GET, fetchOptions);
    Future<HTTPResponse> futureResponse = fetcher.fetchAsync(request);
    return futureResponse;
}

我得到的例外是：

java.util.concurrent.ExecutionException: java.io.IOException: Could not fetch URL: https://www.nytimes.com/2017/05/02/us/politics/health-care-paul-ryan-fred-upton-congress.html, error: Received exception executing http method GET against URL https://www.nytimes.com/2017/05/02/us/politics/health-care-paul-ryan-fred-upton-congress.html: null
[INFO]  at com.google.appengine.api.utils.FutureWrapper.setExceptionResult(FutureWrapper.java:66)
[INFO]  at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
[INFO]  at main.java.com.myapp.app.MyServlet.extractFromUrl(MyServlet.java:10)

Answer 1

查看curl的详细输出，您可以看到网站尝试设置cookie并重定向您，以防cookie不被接受。

看来，在放弃之前，时间会重定向你7次 -

$ curl --verbose -L "https://www.nytimes.com/2017/05/02/us/politics/health-care-paul-ryan-fred-upton-congress.html" 2>&1 | grep 303 | wc -l
7

似乎UrlFetch的最大重定向数为5 [0]。

为了成功抓取www.nytimes.com，您必须禁用以下重定向并自行处理Cookie逻辑。这里有一些灵感[1]和[2]

[0] https://groups.google.com/forum/#!topic/google-appengine/F2dX3LqOrhY

[1] https://groups.google.com/d/msg/google-appengine-java/pE0xak7LRxg/M__U-SM3YMMJ

[2] https://stackoverflow.com/a/13588616/7947020

尝试获取纽约时报页面时，使用GAE的URLFetchService返回null

1 个答案: