Question

我正在编写一个脚本，用于从网站的页面源中提取所有评论。

for addr in links:
    driver.get(addr)
    print(addr)
    for comments in soup.findAll(text=lambda text: isinstance(text, Comment)):
        comments.extract()
        print(comments)

为什么它只这样提取CDATA：

//<![CDATA[
Feedback.Bootstrap.InitializeFeedback({page:true},"epf",true,false,false,false,false);;
//]]>
//<![CDATA[
function addRemoveListenersOnAll(){var t=_ge("b_content"),n,i,r;t&&(n=_d.createElement("STYLE"),n.id=styleIdString,n.innerText="#b_results h2>a {padding: 16px 40px 0 6px;margin: -16px -40px 0 -6px;}",_d.head&&_d.head.appendChild(n),i=t.getElementsByClassName("b_ad"),i&&AddRemoveListener(i),r=t.getElementsByClassName("b_algo"),r&&AddRemoveListener(r))}function AddRemoveListener(n){for(var t,i,u=n.length,r=0;r<u;r++)if(t=n[r].getElementsByTagName("CITE"),t&&t.length>0)for(i=0;i<t.length;i++)sj_be(t[i],"click",algo_c)}function mouseMoveAfterTouchHandler(){sj_ue(document,"mousemove",mouseMoveAfterTouchHandler);var n=_d.getElementById(styleIdString);n&&n.parentNode&&n.parentNode.removeChild(n);sj_log("CI.TTC","mouse","started");sj_ue(document,"mousemove",mouseMoveAfterTouchHandler)}function touchStartHandlerAll(n){n.pointerType==="touch"&&(addRemoveListenersOnAll(),sj_log("CI.TTC","touch","started"),sj_ue(document,"pointerdown",touchStartHandlerAll),document.addEventListener("mousemove",mouseMoveAfterTouchHandler))}var styleIdString="ttcDynStyle",algo_c=function(n){function i(n){var t=n.getElementsByTagName("a"),i;t&&t.length>0&&(i=t[0],si_ct(i),sj_log("CI.TTC","click","touch"),_w.open(i.href,"_self"))}n=sj_ev(n);var t=sj_et(n);if(t){if(t.tagName=="A")return!0;while(t&&!(t.className.indexOf("b_algo")>=0||t.className.indexOf("sb_add")>=0)){if(t.tagName=="BODY")return;t=t.parentNode}}return t?(i(t),!0):(sj_sp(n),!1)};document.addEventListener("pointerdown",touchStartHandlerAll);Feedback.Bootstrap.InitializeFeedback({page:true},"sb_feedback",1,0,0);;
//]]>

但它不会提取此类常规注释：

<!--div class="s-bk-lf"><div class="acc-title" >Следите за новостями и акциями нашего проекта!!!</div> </div><br-->
<!--LiveInternet counter-->
<!--img src="/img/ng6.png"width="150" height=""  hspace="100" vspace="80" align="left" -->

如何提取常规注释而不是仅提取CDATA？

Answer 1

我发现它是在脚本的开头查看漂亮汤解析的html（来自bing搜索结果。

我通过在循环中添加beautifulsoup来解决它。

for addr in links:
    print(addr)
    driver.get(addr)
    html = driver.page_source
    soup = BeautifulSoup(html, "html.parser")
    for comments in soup.findAll(text=lambda text: isinstance(text, Comment)):
        comments.extract()
        print(comments)

它不干净，因为我只是从脚本的前面复制了html和汤，但是确实有效。

为什么Beautiful Soup仅提取CDATA而没有常规注释？

1 个答案: