我为教育目的而抓取的网站具有分页功能。
我的代码完美地抓取了第一页...
但是我怎么刮
?page=2
?page=3
?page=4
?page=5
还有???
应该指出的是,我一直在寻找解决方案,但似乎找不到能够完全回答我需要知道的事情的东西。
当前代码:
// @nuget: HtmlAgilityPack
using System;
using System.Data;
using System.Data.SqlClient;
using System.Net;
using HtmlAgilityPack;
public class Program
{
public static void Main()
{
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12
| SecurityProtocolType.Ssl3;
HtmlWeb web = new HtmlWeb();
HtmlDocument html = web.Load("https://www.g2crowd.com/products/google-analytics/reviews");
// var divNodes = html.DocumentNode.SelectNodes("//div[@class='mb-2 border-bottom']");
var divNodes = html.DocumentNode.SelectNodes(@"//div[@itemprop='reviewBody']");
if (divNodes != null)
{
foreach (var tag in divNodes)
{
string review = tag.InnerText;
review = review.Replace("What do you like best?", "What do you like best?\n");
review = review.Replace("What do you dislike?", "\nWhat do you dislike?\n");
review = review.Replace("Recommendations to others considering the product", "\n\nRecommendations to others considering the product\n");
review = review.Replace("What business problems are you solving with the product? What benefits have you realized?", "\n\nWhat business problems are you solving with the product? What benefits have you realized?\n");
Console.WriteLine(review);
Console.WriteLine("\n------------------------------- Review found. Adding to Database -------------------------------\n");
review = review.Replace("'", "");
review = review.Replace("\n", "<br />");
}
}
}
}
答案 0 :(得分:0)
下一个链接如下:
//link[@rel=next]
只要继续关注它,直到它不再存在。