Question

考虑以下网址http://www.google.com/url?rct=j&sa=t&url=http://www.ksat.com/news/father-of-woman-killed-in-memorial-day-floods-testifies-for-better-flood-warnings&ct=ga&cd=CAIyHWU3NmVhMGQ0NWQ3MmRmY2I6Y29tOmVuOlVTOlJM&usg=AFQjCNE_8XwECqkmyPIMzcSxCDh2hP16wQ。当我将此url传递给JSOUP时，html内容不准确。但是当我在浏览器中打开此网址时，它会重新切换到http://www.ksat.com/news/father-of-woman-killed-in-memorial-day-floods-testifies-for-better-flood-warnings。

然后，我将此url传递给jsoup，现在我获得了确切的html内容。

如何从第一个url ??

中获取确切的html内容

我尝试了很多选项

        Response response = Jsoup.connect(url).followRedirects(true).timeout(timeOut*1000).userAgent(userAgent).execute();
        int status = response.statusCode();
        if (status == HttpURLConnection.HTTP_MOVED_TEMP || status == HttpURLConnection.HTTP_MOVED_PERM || status == HttpURLConnection.HTTP_SEE_OTHER) {
            redirectUrl = response.header("location");
            response = Jsoup.connect(redirectUrl).followRedirects(false).timeout(timeOut*1000).userAgent(userAgent).execute();
        }
        Document doc=response.parse();

我尝试了很多user agents，.referrer("http://google.com")选项等。我目前正在使用jsoup版本1.8.3。

Answer 1

Google会返回一个带有JavaScript / META重定向的html页面：

<script>window.googleJavaScriptRedirect=1</script><script>var n={navigateTo:function(b,a,d){if(b!=a&&b.google){if(b.google.r){b.google.r=0;b.location.href=d;a.location.replace("about:blank");}}else{a.location.replace(d);}}};n.navigateTo(window.parent,window,"http://www.ksat.com/news/father-of-woman-killed-in-memorial-day-floods-testifies-for-better-flood-warnings");
</script><noscript><META http-equiv="refresh" content="0;URL='http://www.ksat.com/news/father-of-woman-killed-in-memorial-day-floods-testifies-for-better-flood-warnings'"></noscript>

这与HTTP重定向标头不同，因为Jsoup不解释JavaScript你运气不好。

但是，你当然可以解析这个以获得真正的链接。当然，这可以在不访问Google的情况下实现，因为该链接是原始网址中参数的一部分。

JSOUP从重定向链接获取html内容

1 个答案: