vb.net阅读Craigslist数据

时间:2016-04-03 00:07:29

标签: vb.net craigslist

您好我正在阅读一些关于如何从阅读网站源代码的网站获取数据的文章。据说我创建了一个例程来获取纽约和匹兹堡的计算机演出的Craigslist数据,如下所示。出于某种原因(可能是我在这方面缺乏知识)我只为匹兹堡和纽约分别收到25条记录。是否有可能获得今天的整个列表,甚至可能是数据范围

Friend Shared Function GetCraigslist() As DataTable

    Dim dt As New DataTable
    dt.Columns.Add("description")
    dt.Columns.Add("Title")
    dt.Columns.Add("link")

    Dim XMLDoc As System.Xml.XmlDocument = New System.Xml.XmlDocument
    Dim list As New List(Of String)
    list.Add("https://pittsburgh.craigslist.org/cpg/index.rss")
    list.Add("http://newyork.craigslist.org/cpg/index.rss")

    For Each link In list

        XMLDoc.Load(link)
        Dim nTable As NameTable = XMLDoc.NameTable
        Dim nsManager As XmlNamespaceManager = New XmlNamespaceManager(nTable)

        nsManager.AddNamespace("rdf", "http://www.w3.org/1999/02/22-rdf-syntax-ns#")
        nsManager.AddNamespace("item", "http://purl.org/rss/1.0/")
        nsManager.AddNamespace("dc", "http://purl.org/dc/elements/1.1/")
        Dim XMLItems As System.Xml.XmlNodeList = XMLDoc.DocumentElement.SelectNodes("//rdf:RDF/item:item", nsManager)

        XMLDoc.Save("C:\development\data.xml")
        For Each currentItem As Xml.XmlNode In XMLItems
            Dim R As DataRow = dt.NewRow
            R("Title") &= currentItem.SelectSingleNode("item:title", nsManager).InnerText
            R("description") &= currentItem.SelectSingleNode("item:description", nsManager).InnerText
            R("link") &= currentItem.SelectSingleNode("item:link", nsManager).InnerText
            dt.Rows.Add(R)
        Next
    Next

    Return dt

End Function

0 个答案:

没有答案