Question

我有一个我不熟悉的新项目。一个任务是我需要浏览一些网站来收集一些数据。一个样本网站就是：https://www.hudhomestore.com/Home/Index.aspx

我已经阅读并观看了关于＆＃34;收集＆＃34;的教程。来自网页的数据，例如：

但我的问题是我们通常如何设置偏好，以及＆＃34;搜索＆＃34;根据我们的偏好，然后使用上面的链接在我的代码中加载结果？

修改

根据我的选择设置搜索条件是正确的。但是，搜索的总计数（如果我手动为MI状态）是223，但我执行下面的代码，tdNodeCollection只有121.你能告诉我我哪里出错吗？

    HtmlWeb web = new HtmlWeb();
    HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument();

    string zipCode = "", city = "", county = "", street = "", sState = "MI", fromPrice = "0", toPrice = "0", fcaseNumber = "",
           bed = "0", bath = "0", buyerType = "0", Status = "0", indoorAmenities = "", outdoorAmenities = "", housingType = "",
           stories = "", parking = "", propertyAge = "", sLanguage = "ENGLISH";

    var doc = await (Task.Factory.StartNew(() => web.Load("https://www.hudhomestore.com/Listing/PropertySearchResult.aspx?" +
        "zipCode=" + zipCode + "&city=" + city + "&country=" + county + "&street=" + street + "&sState=" + sState +
        "&fromPrice=" + fromPrice + "&toPrice=" + toPrice +
        "&fcaseNumber=" + fcaseNumber + "&bed=" + bed + "&bath=" + bath +
        "&buyerType=" + buyerType + "&Status=" + Status + "&indoorAmenities=" + indoorAmenities +
        "&outdoorAmenities=" + outdoorAmenities + "&housingType=" + housingType + "&stories=" + stories +
        "&parking=" + parking + "&propertyAge=" + propertyAge + "&sLanguage=" + sLanguage)));

    HtmlNodeCollection tdNodeCollection = doc
                             .DocumentNode
                             .SelectNodes("//*[@id=\"dgPropertyList\"]//tr//td");

Answer 1

您可以为此目的使用HTMLAgilityPack。我已经制作了一个小的测试代码，并根据您可以设置的搜索条件对您要废弃的第二页进行测试。

        HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument();
        HtmlWeb web = new HtmlWeb();
        //string InitialUrl = "https://www.hudhomestore.com/Home/Index.aspx";
        //Here you need to set the values of these variable to whatever user inputs
        //after setting these values, add them to initial URL
        string zipCode = "", city = "", county = "", street = "", sState = "AK", fromPrice = "0", toPrice = "0", fcaseNumber = "",
               bed = "0", bath = "0", buyerType = "0", Status = "0", indoorAmenities = "", outdoorAmenities = "", housingType = "",
               stories = "", parking = "", propertyAge = "", sLanguage = "ENGLISH";
        HtmlAgilityPack.HtmlDocument document = web.Load("https://www.hudhomestore.com/Listing/PropertySearchResult.aspx?" +
            "zipCode=" + zipCode + "&city=" + city + "&country=" + county + "&street=" + street + "&sState=" + sState + 
            "&fromPrice=" + fromPrice + "&toPrice=" + toPrice +
            "&fcaseNumber=" + fcaseNumber + "&bed=" + bed + "&bath=" + bath + 
            "&buyerType=" + buyerType + "&Status=" + Status + "&indoorAmenities=" + indoorAmenities + 
            "&outdoorAmenities=" +outdoorAmenities + "&housingType=" + housingType + "&stories=" + stories + 
            "&parking=" + parking + "&propertyAge=" + propertyAge + "&sLanguage=" + sLanguage);
        HtmlNodeCollection tdNodeCollection = document
                                 .DocumentNode
                                 .SelectNodes("//*[@id=\"dgPropertyList\"]//tr//td");

再次计算它们并查看您的表达式，td's内tr内id="dgPropertyList"正好为td 接下来，手动检查td，并从foreach (HtmlAgilityPack.HtmlNode node in tdNodeCollection) { //Do you say you want to access to <h2>, <p> here? //You can do: HtmlNode h2Node = node.SelectSingleNode("./h2"); //That will get the first <h2> node HtmlNodeCollection allH2Nodes = node.SelectNodes(".//h2"); //That will search in depth too //And you can also take a look at the children, without using XPath (like in a tree): HtmlNode h2Node_ = node.ChildNodes["h2"]; }跟踪您需要的内容并获取该数据。

Selenium webdriver

我已经测试了代码，它可以工作并解析整个文档以达到所需的表格。它将获取div中该表中的所有行。所以，你可以进一步挖掘这些行，找到你的td并得到你需要的东西。

另一种选择可能是使用<td align='center' *ngFor="let items of QuestionMaster.Ratings let i=index">  <input type="radio" [value]="items.rateno" name="{{item.question1}}" ngModel required /> </td>，Get your hands on Selenium

如果您不希望浏览器可见并且仍希望使用类似Selenium的功能，那么您可以使用PhantomJS

希望它有所帮助。

如何通过指定搜索条件从网站提取数据？

1 个答案: