Question

我正在网页上进行抓取练习，并希望在我的网页上显示一些内容，而我却陷入了查找后代的代码中。以下是网页HTML

<ul class="results">
    <li class="gts" data-webm-section="OAG-AD-14960184">
        <a class="item-link-container" href="/bikes/details/2016-Indian-Chieftain-Dark-Horse-MY17/OAG-AD-14960184/?cr=0&amp;gts=OAG-AD-14960184&amp;gtsviewtype=TopSpot&amp;gtssaleid=OAG-AD-14960184&amp;psq=%28%28Service%3D%5BBikesales%5D%26State%3D%5BNSW%5D%29%26%28%28%28%28SiloType%3D%5BBrand%20new%20bikes%20available%5D%7CSiloType%3D%5BBrand%20new%20bikes%20in%20stock%5D%29%7CSiloType%3D%5BDealer%20used%20bikes%5D%29%7CSiloType%3D%5BDemo%20%26%20near%20new%20bikes%5D%29%7CSiloType%3D%5BPrivate%20used%20bikes%5D%29%29&amp;pso=0&amp;pss=Premium">
            <header>
                <h3><span class=></span>Heading</h3>
                <div class="spotlight flag non-textual">Spotlight</div>
            </header>
            <div class="primary panel">
                <ul class="photos" data-js-lazy-load-length="3" style="width:8350px">
                    <li>
                        <img src="http//" height="221" width="334" alt="2016 Indian Chieftain Dark Horse MY17" />
                    </li>
                </ul>
                <div class="image-nav previous" data-webm-clickvalue="previous-image">
                    <span class="arrow"></span>
                    <span class="background"></span>
                </div>
                <div class="image-nav next" data-webm-clickvalue="next-image">
                    <span class="arrow"></span>
                    <span class="background"></span>
                </div>
                <div class="image-nav-count">
                    <span class="current">1</span> of 24
                </div>
            </div>
            <div class="secondary panel">
                <span class="price">$29,995*</span>
                <div data-fancybox-href="/mvcajax/bikes/PriceGuide/" class="pricing-message light-box-iframe">
                    Ride Away No More To Pay
                </div>
                <div class="features">
                    <ul>
                        <li class="ui-category">
                            <i></i>Cruiser
                        </li>
                        <li class="engine-size">
                            <i></i>1,811 cc
                        </li>
                        <li class="odometer">
                            <i></i>2,552 km
                        </li>
                    </ul>
                    <div class="bike-facts non-textual"></div>
                </div>
            </div>
            <p class="description">**NO REGRETS - 7 Day Money Back guarantee** PLUS 12 months Warranty &amp; Roadside Assist.Conditions App...</p>

        </a>
    </li>
</ul>

在上面的html我想要自行车的图像，价格和类别

以下是我的代码

 public async Task<ActionResult> Webcrawl()
    {
        string URL = "https://www.bikesales.com.au/bikes/new-south-wales/";
        List<bikes> bikelist = new List<bikes>();
        using (var client = new HttpClient())
        {
            var html = await client.GetStringAsync(URL);

            HtmlDocument Doc = new HtmlDocument();
            Doc.LoadHtml(html);

            var ProductsHtml = Doc.DocumentNode.Descendants("ul").Where(node => node.GetAttributeValue("class", "").Equals("results")).ToList();

            var ProductsList = ProductsHtml[0].Descendants("li").Where(node => node.GetAttributeValue("class", "").Equals("gts")).ToList();
            foreach (var list in ProductsList)
            {
                var PriceNode = list.SelectSingleNode("//div[@class='secondary panel']");
                var bike = new bikes
                {
                    Name = list.Descendants("h3").FirstOrDefault().InnerText,
                    Title = list.Descendants("p").FirstOrDefault().InnerText,
                    Price = PriceNode.Descendants("span").FirstOrDefault().InnerText,
                    Image = list.SelectNodes("//div[@class='primary panel']/ul[1]/li[1]/img").FirstOrDefault().ChildAttributes("src").FirstOrDefault().Value,
                    Type = list.SelectNodes("//div[@class='secondary panel']/div[2]/ul[1]/li[1]").FirstOrDefault().InnerText.Trim('\r', '\n', '\t'),
                    Engine = list.SelectNodes("//div[@class='secondary panel']/div[2]/ul[1]/li[2]").FirstOrDefault().InnerText.Trim('\r', '\n', '\t'),
                    Odometer = list.SelectNodes("//div[@class='secondary panel']/div[2]/ul[1]/li[3]").FirstOrDefault().InnerText.Trim('\r', '\n', '\t'),
                };
                bikelist.Add(bike);
            }
            return View(bikelist);
        }
    }

当我运行上面的代码时，我只获得列表中的第一个元素，除了名称，即相同的图像，相同的类型和相同的价格。

请纠正我在代码中的错误。在此先感谢。

Answer 1

要从特定元素中搜索，您需要在xpath中使用点。否则，从文档的根目录执行搜索。此外，我再次写了一些这些xpath，因为在某些情况下它们没有返回数据。

var products = Doc.DocumentNode.SelectNodes("//ul[@class='results']/li[@class='gts']");
foreach (HtmlNode product in products)
{
    var PriceNode = product.SelectSingleNode(".//div[@class='secondary panel']");
    var bike = new bikes
    {
        Name = product.SelectSingleNode(".//li[@class='ui-category']").InnerText,
        Title = product.SelectSingleNode(".//header/h3").InnerText,
        Price = product.SelectSingleNode(".//span[@class='price']").InnerText,
        Image = product.SelectSingleNode(".//ul[@class='photos']//img").Attributes["src"].Value,
        Type = product.SelectSingleNode(".//li[@class='ui-category']").InnerText.Trim('\r', '\n', '\t'),
        Engine = product.SelectSingleNode(".//li[@class='engine-size']").InnerText.Trim('\r', '\n', '\t'),
        Odometer = product.SelectSingleNode(".//li[@class='odometer']").InnerText.Trim('\r', '\n', '\t'),
    };
    bikelist.Add(bike);
}

它生成：

Touring
2018 Indian Roadmaster Elite
$49,995*
Touring
1,811 cc
0 km

Cruiser
2018 Indian Scout
$19,995*
Cruiser
1,133 cc
301 km

Naked
2018 Suzuki GSX-S1000
$15,690*
Naked
999 cc
0 km

Touring
2017 Indian Roadmaster
$37,995*
Touring
1,811 cc
6,577 km

Super Sport
2018 Suzuki GSX-R600
$13,790*
Super Sport
599 cc
0 km

Cruiser
2018 Indian Scout
$18,995*
Cruiser
1,133 cc
1,901 km

...and so on

HTML Agility Pack找到Descendants

1 个答案: