Question

我终于发现BS4不再像BS3那样使用“标记按摩”了。但我仍然需要一种类似的方法来处理不需要的document.write。您将在BS3中执行以下操作，但如何在BS4中执行此操作？

# Javascript code in ths page generates HTML markup  
# that isn't parsed correctly by BeautifulSoup.
# To avoid this problem, all document.write fragments are removed
my_massage = copy(BeautifulSoup.MARKUP_MASSAGE)
my_massage.append((re.compile(u"document.write(.+);"), lambda match: ""))
my_massage.append((re.compile(u'alt=".+">'), lambda match: ">"))

另外，由于BS4 BeautifulSoup consctuctor不再支持markupmassage参数，我的程序应该在哪里处理document.write问题？我假设这是问题所在，因为我只是试图打印出表格标记，而且当我运行风车时我遇到一个线程异常。

这就是我的代码：

#!/usr/bin/env python
# Generated by the windmill services transformer
#from windmill.authoring import WindmillTestClient
from bs4 import BeautifulSoup

import re, urlparse
from copy import copy
from windmill.authoring import setup_module, WindmillTestClient
from windmill.conf import global_settings
import sys


global_settings.START_CHROME = True # This makes it use Firefox
setup_module(sys.modules[__name__])


def get_table_info(client):
        """
    Parse HTML page and extract featured image name and link
    """
    # Get Javascript updated HTML page
    client.waits.forElement(xpath=u"//table[@id='trades']",
                        timeout=40000)
    response = client.commands.getPageText()
    assert response['status']
    assert response['result']

    # Create soup from HTML page and get desired information
    soup = BeautifulSoup(response['result'])

    table_info = soup.select("#trades")
    return table_info


def test_scrape():
    """
    Scrape site
    """

    # Open main gallery page
    client = WindmillTestClient(__name__)
    client.open(url='http://www.zulutrade.com/trader/128391')


    table_info = {}
    table_info = get_table_info(client)


    print table_info



test_scrape()

Answer 1

您不需要告诉BeautifulSoup如何按摩标记 - 您可以在将其提供给BeautifulSoup构造函数之前自行修改它：

html = response['result']
html = re.sub(r'document.write(.+);', '', html)
html = re.sub(r'alt=".+">', '>', html)
soup = BeautifulSoup(html)

文件刮使用beautifulsoup4.2和windmill，如何消除“document.write（）;”从标记传递给BeautifulSoup构造函数

1 个答案: