Question

我正在尝试使用以下代码获取网页的来源：

public static String getFile(String sUrl) throws ClientProtocolException, IOException {
    DefaultHttpClient httpclient = new DefaultHttpClient();
    StringBuilder b = new StringBuilder();

    // Prepare a request object
    HttpGet httpget = new HttpGet(sUrl);

    // Execute the request
    HttpResponse response = httpclient.execute(httpget);

    // Examine the response status
    System.out.println(response.getStatusLine());

    //status code should be 200
    if (response.getStatusLine().getStatusCode() != 200) {
        return null; 
    }

    // Get hold of the response entity
    HttpEntity entity = response.getEntity();

    // If the response does not enclose an entity, there is no need
    // to worry about connection release
    if (entity != null) {
        InputStream instream = entity.getContent();

        try {
            BufferedReader reader = new BufferedReader(new InputStreamReader(instream));
            // do something useful with the response
            String s = reader.readLine();

            while (s != null) {
                b.append(s);
                b.append("\n");
                s = reader.readLine();
            }

        } catch (IOException ex) {
            // In case of an IOException the connection will be released
            // back to the connection manager automatically
            throw ex;

        } catch (RuntimeException ex) {
            // In case of an unexpected exception you may want to abort
            // the HTTP request in order to shut down the underlying
            // connection and release it back to the connection manager.
            httpget.abort();
            throw ex;

        } finally {
            // Closing the input stream will trigger connection release
            instream.close();
        }

        // When HttpClient instance is no longer needed,
        // shut down the connection manager to ensure
        // immediate deallocation of all system resources
        httpclient.getConnectionManager().shutdown();
    }

    return b.toString();
}

它工作正常，但某些符号如＆amp; nbsp， - ，单引号等未正确复制。我尝试将页面源text/html类型保存到amazon s3中，并通过访问s3 server中保存的页面来显示它。

我上面提到的符号显示为�。对此有什么解决方案吗？

Answer 1

您需要确保使用页面编码来阅读内容，否则将使用您的系统默认编码（显然不是您所见的正确编码）：

 BufferedReader reader = new BufferedReader(
      new InputStreamReader(instream, entity.getContentEncoding()));

Answer 2

首先需要指定InputStreamReader使用的编码。您的构造函数版本采用系统上的默认编码。

编码可以在标题中传递。它默认为ISO-8859-1但是（Latin-1）但实际上是Windows-1252（Windows Latin-1）。

String charset = "Windows-1252"; // Can be used as default.
String enc = entity.getContentEncoding(); // Or from Content-Type.
if (enc != null) {
    charset = enc;
}

BufferedReader reader = new BufferedReader(
                     new InputStreamReader(instream, charset));

对于HTML实体，apache有：

String s = ...
s = StringEscapeUtils.unescapeHTML4(s);

获取页面源时未正确复制符号

2 个答案: