Question

我看到NLTK建议使用BeautifulSoup get_text()将HTML预处理为文本，以便进行后续的NLP分析。但它似乎并没有很好地发挥作用。在以下示例中，xyz和abc是一致的，但它们不应该是。是否有更好的预处理实用程序可以将HTML转换为NLP应用程序的文本？

$ cat main.py
#!/usr/bin/env python
# vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1:

html_doc = "<h2>xyz</h2><p>abc</p>"

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')

print soup.get_text()
$ ./main.py 
xyzabc

Answer 1

您只需要设置一个分隔符，它将用作子节点之间的分隔符：

In [1]: from bs4 import BeautifulSoup

In [2]: html_doc = "<h2>xyz</h2><p>abc</p>"

In [3]: soup = BeautifulSoup(html_doc, 'html.parser')

In [4]: soup.get_text(separator=" ")
Out[4]: 'xyz abc'

Answer 2

我建议您使用html2text工具。这是在命令行中进行的测试：

$ html2text --ignore-links https://content.cultureandempire.com/chapter1.html 

  * Culture & Empire
  *   * __Introduction
  * __**1.** Preface 
  * __**2.** Chapter 1 - Magic Machines 
  * __**3.** Chapter 2 - Spheres of Light 
  * __**4.** Chapter 3 - Faceless Societies 
  * __**5.** Chapter 4 - Freedom in Chains 
  * __**6.** Chapter 5 - Eyes of the Spider 
  * __**7.** Chapter 6 - Wealth of Nations 
  * __**8.** Chapter 7 - March of the Kaiju 
  * __**9.** Chapter 8 - The Reveal 
  * __**10.** Postface 
  * __**11.** Appendix 1 
  *   * Published with GitBook 

#  __Culture & Empire

# Chapter 1. Magic Machines

> Far away, in a different place, a civilization called Culture had taken
seed, and was growing. It owned little except a magic spell called Knowledge.

In this chapter, I'll examine how the Internet is changing our society. It's
happening quickly. The most significant changes have occurred during just the
last 10 years or so. More and more of our knowledge about the world and other
people is transmitted and stored digitally. What we know and who we know are
moving out of our minds and into databases. These changes scare many people,
whereas in fact they contain the potential to free us, empowering us to
improve society in ways that were never before possible.

## From Bricks to Bits

否则，您可以使用lxml.html.Element.text_content()或python's textract

NLP从HTML到文本的预处理

2 个答案: