Question

运行此代码时，输出的某些单词会被拆分。就像“容忍”一词被分为“容忍”一样。我看了看html源，似乎就是页面创建的方式。

在许多其他地方，单词也被拆分。在写文本之前如何重新组合它们？

import requests, codecs
from bs4 import BeautifulSoup
from bs4.element import Comment

path='C:\\Users\\jason\\Google Drive\\python\\'

def tag_visible(element):
    if element.parent.name in ['sup']:
        return False
    if isinstance(element, Comment):
        return False
    return True

ticker = 'TSLA'
quarter = '18Q2'    
mark1= 'ITEM 1A'
mark2= 'UNREGISTERED SALES'
url_new='https://www.sec.gov/Archives/edgar/data/1318605/000156459018019254/tsla-10q_20180630.htm'

def get_text(url,mark1,mark2):
    html = requests.get(url) 
    soup = BeautifulSoup(html.text, 'html.parser')

    for hr in soup.select('hr'):
        hr.find_previous('p').extract()

    texts = soup.findAll(text=True)

    visible_texts = filter(tag_visible, texts) 
    text=u" ".join(t.strip() for t in visible_texts)

    return text[text.find(mark1): text.find(mark2)]

text = get_text(url_new,mark1,mark2)

file=codecs.open(path + "test.txt", 'w', encoding='utf8')
file.write (text)
file.close()

Answer 1

您正在处理使用Microsoft Word格式化的 HTML 。不要提取文本并尝试在没有上下文的情况下进行处理。

您要处理的部分用<a name="...">标记清楚地描述，让我们开始选择带有<a name="ITEM_1A_RISK_FACTORS">标记的所有元素，一直到但不包括<a name="ITEM2_UNREGISTERED_SALES">标记：

def sec_section(soup, item_name):
    """iterate over SEC document paragraphs for the section named item_name

    Item name must be a link target, starting with ITEM
    """

    # ask BS4 to find the section
    elem = soup.select_one('a[name={}]'.format(item_name))
    # scan up to the parent text element
    # html.parser does not support <text> but lxml does
    while elem.parent is not soup and elem.parent.name != 'text':
        elem = elem.parent

    yield elem
    # now we can yield all next siblings until we find one that
    # is also contains a[name^=ITEM] element:
    for elem in elem.next_siblings:
        if not isinstance(elem, str) and elem.select_one('a[name^=ITEM]'):
            return
        yield elem

此函数为我们提供了HTML文档中<text>节点的所有子节点，这些子节点从包含特定链接目标的段落开始，一直到命名为ITEM的下一个链接目标

接下来，通常的Word清理任务是删除标签和style属性：

def clean_word(elem):
    if isinstance(elem, str):
        return elem
    # remove last-rendered break markers, non-rendering but messy
    for lastbrk in elem.select('a[name^=_AEIOULastRenderedPageBreakAEIOU]'):
        lastbrk.decompose()
    # remove font tags and styling from the document, leaving only the contents
    if 'style' in elem.attrs:
        del elem.attrs['style']
    for e in elem:  # recursively do the same for all child nodes
        clean_word(e)
    if elem.name == 'font':
        elem = elem.unwrap()
    return elem

Tag.unwrap() method最有用，因为文本几乎被标签任意分割。

现在，将文本干净地提取很简单：

for elem in sec_section(soup, 'ITEM_1A_RISK_FACTORS'):
    clean_word(elem)
    if not isinstance(elem, str):
        elem = elem.get_text(strip=True)
    print(elem)

这将在其余文本中输出：

•that the equipment and processes which we have selected for Model 3 production will be able to accurately manufacture high volumes of Model 3 vehicles within specified design tolerances and with high quality;

现在可以正确合并文本，不再需要重新组合。

整个部分仍在表格中，但是clean_word()现在将其清理为更加合理的内容：

<div align="left">
<table border="0" cellpadding="0" cellspacing="0">
<tr>
<td valign="top">
<p> </p></td>
<td valign="top">
<p>•</p></td>
<td valign="top">
<p>that the equipment and processes which we have selected for Model 3 production will be able to accurately manufacture high volumes of Model 3 vehicles within specified design tolerances and with high quality;</p></td></tr></table></div>

因此您可以在此处使用更智能的文本提取技术来进一步确保干净的文本转换；您可以将这样的项目符号表转换为*前缀，例如：

def convert_word_bullets(soup, text_bullet="*"):
    for table in soup.select('div[align=left] > table'):
        div = table.parent
        bullet = div.find(string='\u2022')
        if bullet is None:
            # not a bullet table, skip
            continue
        text_cell = bullet.find_next('td')
        div.clear()
        div.append(text_bullet + ' ')
        for i, elem in enumerate(text_cell.contents[:]):
            if i == 0 and elem == '\n':
                continue  # no need to include the first linebreak
            div.append(elem.extract())

此外，如果您运行的话，您可能也想跳过分页符（[page number]和<hr/>元素的组合）

for pagebrk in soup.select('p ~ hr[style^=page-break-after]'): 
    pagebrk.find_previous_sibling('p').decompose()
    pagebrk.decompose()

这比您自己的版本更明确，在该版本中，您删除所有<hr/>元素和前面的元素，无论它们实际上是否是同级兄弟。

同时执行两个操作清理Word HTML。与您的功能结合在一起，将成为：

def get_text(url, item_name):
    response = requests.get(url) 
    soup = BeautifulSoup(response.content, 'html.parser')

    for pagebrk in soup.select('p ~ hr[style^=page-break-after]'): 
        pagebrk.find_previous_sibling('p').decompose()
        pagebrk.decompose()

    convert_word_bullets(soup)
    cleaned_section = map(clean_word, sec_section(soup, item_name))

    return ''.join([
        elem.get_text(strip=True) if elem.name else elem
        for elem in cleaned_section])


text = get_text(url, 'ITEM_1A_RISK_FACTORS')
with open(os.path.join(path, 'test.txt'), 'w', encoding='utf8') as f:
    f.write(text)

Answer 2

此页面标记确实很糟糕。您将需要删除多余的标签以解决您的问题。幸运的是，Beautifulsoup可以完成繁重的工作。以下代码将删除所有font标签。

soup = BeautifulSoup(html.text, 'html.parser')
for font in soup.find_all('font'):
    font.unwrap()

beautifulsoup如何重新组合单词

2 个答案: