网站内容阅读和识别字符集

时间:2014-05-13 09:53:34

标签: c# asp.net vb.net http webrequest

  Protected Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim Imagelink As String = ""
        Dim Text As String = TextBox1.Text
        Dim request As HttpWebRequest = DirectCast(HttpWebRequest.Create(Text), HttpWebRequest)
        request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
        Dim respons As HttpWebResponse
        respons = DirectCast(request.GetResponse(), HttpWebResponse)
        Dim enc As Encoding
        Try
            enc = Encoding.GetEncoding(respons.CharacterSet)
        Catch ex As Exception
            enc = Encoding.GetEncoding("ISO-8859-1")
        End Try

        Dim reader As New StreamReader(respons.GetResponseStream(), enc)
        Dim sr As String = reader.ReadToEnd()

        Dim Pattern As String = "<img([^s]|s[^r]|sr[^c]|src[^=]|src=[^'""])*src=['""](?<SRC>[^'""]*)['""]"
        Dim m As MatchCollection = Regex.Matches(sr, Pattern)
        For Each mm As Match In m
            Dim link_ As String = mm.Groups("SRC").Value
            ' Dim x_ As String = link_.Substring(0, 7)

            If link_.Substring(0, 7) = "http://" Then
                Response.Write(mm.Groups("SRC").Value + "" + "<br>")
                Imagelink = link_
            End If


        Next
        Dim image_ As New Image
        image_.Attributes("src") = Imagelink
        PlaceHolder1.Controls.Add(image_)
    End Sub

这是我用来向网页发送请求以获取其内容并从网页中提取图像链接的代码。但是在某些网页中,它不返回它返回“”的网页标题中的字符集。但是,当我尝试使用默认编码进行解析时,它也没有提供正确的内容?以前任何人遇到过这种情况真的很令人沮丧吗?如果有人能指出我正确的方向,如何克服这个或预测使用什么样的编码,那将是非常感谢。

在响应标头中没有提供字符集的网站示例

Image site link

1 个答案:

答案 0 :(得分:0)

在这种情况下,字符编码在标题中给出:Content-Type: text/html; charset=utf-8