Question

如何屏蔽废弃javascript生成的html页面？我尝试过这样的事情：

        HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
        HtmlWeb hw = new HtmlWeb();
        doc = hw.Load("http://stats.nba.com/scores.html?gameDate=04/11/2014");
        HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div[@id='scoreboards']");

然而，这并不起作用，因为内容是由javascript生成的。有没有办法在javascript生成html后刮取页面？也许使用其他敏捷包装？

Answer 1

使用webbrowser控件获取js或ajax

所包含的内容和元素

private void LoadHtmlWithBrowser(String url)
{
    webBrowser1.ScriptErrorsSuppressed = true;
    webBrowser1.Navigate(url);

    waitTillLoad(this.webBrowser1);

    HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    var documentAsIHtmlDocument3 = (mshtml.IHTMLDocument3)webBrowser1.Document.DomDocument; 
    StringReader sr = new StringReader(documentAsIHtmlDocument3.documentElement.outerHTML); 
    doc.Load(sr);
}

private void waitTillLoad(WebBrowser webBrControl)
{
    WebBrowserReadyState loadStatus;
    int waittime = 100000;
    int counter = 0;
    while (true)
    {
        loadStatus = webBrControl.ReadyState;
        Application.DoEvents();
        if ((counter > waittime) || (loadStatus == WebBrowserReadyState.Uninitialized) || (loadStatus == WebBrowserReadyState.Loading) || (loadStatus == WebBrowserReadyState.Interactive))
        {
            break;
        }
        counter++;
    }

    counter = 0;
    while (true)
    {
        loadStatus = webBrControl.ReadyState;
        Application.DoEvents();
        if (loadStatus == WebBrowserReadyState.Complete && webBrControl.IsBusy != true)
        {
            break;
        }
        counter++;
    }
}

Answer 2

您可以使用web浏览器对象，该对象是一个c＃对象，其作用类似于浏览器并运行javascript代码，并在收到响应后使用敏捷包解析它。

MSDN - Web Browser

我如何屏幕抓取由javascript生成的HTML页面

2 个答案: