等待请求完成后再继续

时间:2019-12-23 12:13:58

标签: vb.net web-scraping web-crawler html-agility-pack

我想出了当前设置的问题。在运行当前代码时,它说ProductListPage为null,并且在删除错误后不会继续处理。如何添加需要等待页面加载并找到//div[@class='productContain padb6']//div[@class='large-4 medium-4 columns']/a的命令?

这是我当前的代码:

Dim htmlDoc As HtmlDocument = New HtmlWeb().Load("https://example.com")
Dim ProductListPage As HtmlNodeCollection = htmlDoc.DocumentNode.SelectNodes("//div[@class='productContain padb6']//div[@class='large-4 medium-4 columns']/a")
For Each src As HtmlNode In ProductListPage
    htmlDoc = New HtmlWeb().Load(src.Attributes("href").Value)
    Dim LinkTester As HtmlNodeCollection = htmlDoc.DocumentNode.SelectNodes("//div[@class='row padt6 padb4']//a")
    If LinkTester IsNot Nothing Then
        For Each dllink In LinkTester
            Dim LinkURL As String = dllink.Attributes("href").Value
            Console.WriteLine(LinkURL)

            Dim ExtractFilename As String = LinkURL.Substring(LinkURL.LastIndexOf("/"))
            Dim DLClient As New WebClient
            DLClient.DownloadFileAsync(New Uri(LinkURL), ".\" & ExtractFilename)
        Next
    End If
Next

0 个答案:

没有答案