Question

我正在尝试从https://www.groupon.pl/deals/ga-hotel-alpin-17网站阅读标题（这是此特定网站特有的问题）

address = "https://www.groupon.pl/deals/ga-hotel-alpin-17";
URL url = new URL(address);
URLConnection httpcon = url.openConnection();
httpcon.setConnectTimeout(5000);
httpcon.setReadTimeout(5000);
httpcon.addRequestProperty("User-Agent", "Mozilla/4.0");
response = httpcon.getInputStream();
Scanner scanner = new Scanner(response);
String responseBody = scanner.useDelimiter("\\A").next();
String title = responseBody.substring(responseBody.toUpperCase().indexOf("<TITLE>") + 7, responseBody.toUpperCase().indexOf("</TITLE>"));

我得到403或SocketTimeoutException：

java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)

获取此网站没有问题，例如使用简单的wget命令。

我怀疑服务器不希望被Java查询但是为什么设置用户代理没有帮助？还有什么可以做假装真正的浏览器行为吗？有什么想法吗？

Answer 1

我找到了answear，下面的程序有效！成功的关键是准确地观察浏览器发送和使用它们的请求标头。缺少＆＃34;接受编码＆＃34;头

import sun.misc.IOUtils;
import java.io.*;
import java.net.URL;
import java.net.URLConnection;
import java.util.Scanner;

public class Program {


    public static void main(String[] args) throws IOException {
        System.out.println("Hello World!");

        String address = "https://www.groupon.pl/deals/ga-hotel-alpin-17";
        URL url = new URL(address);
        URLConnection httpcon = url.openConnection();
        httpcon.setConnectTimeout(5000);
        httpcon.setReadTimeout(5000);

//        httpcon.addRequestProperty("Host", "www.groupon.pl");
        httpcon.addRequestProperty("User-Agent", "Mozilla/5.0 (X11; Fedora; Lin… Gecko/20100101 Firefox/54.0");
//        httpcon.addRequestProperty("Accept", "text/html,application/xhtml+x…lication/xml;q=0.9,*/*;q=0.8");
//        httpcon.addRequestProperty("Accept-Language", "en-US,en;q=0.5");
        httpcon.addRequestProperty("Accept-Encoding", "utf-8");
//        httpcon.addRequestProperty("DNT", "1");
//        httpcon.addRequestProperty("Connection", "keep-alive");
//        httpcon.addRequestProperty("Upgrade-Insecure-Requests", "1");

        InputStream response = httpcon.getInputStream();



        Scanner scanner = new Scanner(response);
        String responseBody = scanner.useDelimiter("\\A").next();
        String title = responseBody.substring(responseBody.toUpperCase().indexOf("<TITLE>") + 7, responseBody.toUpperCase().indexOf("</TITLE>"));

        System.out.println("End!" + title);
    }
}

评论标题不是必需的。

干杯！卢卡斯

PS。很遗憾，我为这个问题得到-2 ......

java SocketTimeoutException

1 个答案: