Jsoup.parse移动网址

Question

我是Jsoup的新手，我正在尝试使用Jsoup.parse（）下载移动网站。以下代码适用于普通网址但不适用于移动设备。这有什么问题？

代码：

 private static Document downloadDocument(String url, String referer, int timeout) {
    if (url.isEmpty() || url == null) {
        return null;
    }
    if (referer.isEmpty() || referer == null) {
        //default to google.
        referer = "http://www.google.com";
    }
    Document document;
    try {
        document = Jsoup.parse(new URL(url), timeout);
    } catch (IOException e) {
        //TODO - Remove System.out.println - Memory Issue.
        System.out.println("Sorry, unable to download document");
        return null;
    }
    return document;
}

堆栈跟踪如下：

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=http://m.careerbuilder.com/
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
    at org.jsoup.helper.HttpConnection.get(HttpConnection.java:167)
    at org.jsoup.Jsoup.parse(Jsoup.java:183)
    ...

Answer 1

您要解析的网站会检查用户代理，但不接受默认的网站（ Java / jdk_version ）。所以你应该使用“假的”用户代理，如下所示：

Document html = Jsoup.connect("http://m.careerbuilder.com").userAgent("Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36").get();
System.out.println(html);

其中 Mozilla / 5.0（Windows NT 6.2; Win64; x64）AppleWebKit / 537.36（KHTML，与Gecko一样）Chrome / 32.0.1667.0 Safari / 537.36 是Chrome 32.0.1667.0的用户代理< / p>