Question

我希望在呈现HTML后，从网站上看到所有文本。我在Python中使用Scrapy框架。使用xpath('//body//text()')我能够获得它，但使用HTML标记，我只想要文本。对此有何解决方案？

Answer 1

最简单的选择是extract //body//text()和join找到的所有内容：

''.join(sel.select("//body//text()").extract()).strip()

其中sel是Selector个实例。

另一种选择是使用nltk的clean_html()：

>>> import nltk
>>> html = """
... <div class="post-text" itemprop="description">
... 
...         <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p>
... 
...     </div>"""
>>> nltk.clean_html(html)
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !"

另一种选择是使用BeautifulSoup的get_text()：

get_text()

如果您只想要文档或标记的文本部分，那么   可以使用get_text()方法。它返回文档中的所有文本   或标记下方，作为单个Unicode字符串。

>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(html)
>>> print soup.get_text().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !

另一种选择是使用lxml.html的text_content()：

.text_content()

返回元素的文本内容，包括   孩子的文本内容，没有标记。

>>> import lxml.html
>>> tree = lxml.html.fromstring(html)
>>> print tree.text_content().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !

Answer 2

你试过吗？

xpath('//body//text()').re('(\w+)')

OR

 xpath('//body//text()').extract()

Answer 3

xpath('//body//text()')并不总是将杓子驱动到您上次使用的标签中的节点中（在您的案例正文中）。如果您键入xpath('//body/node()/text()').extract()，您将看到html中的节点身体。您可以尝试xpath('//body/descendant::text()')。

如何从Scrapy网站获取所有纯文本？

3 个答案: