我正在尝试使用Scrapy从众多网站上抓取内容，而实际上只是想要主要内容文本。这意味着避免导航文本，页眉文本和页脚文本。我注意到的一件事是，很多时候页脚或页眉会在体内。因此，我需要找到一种方法来获取正文中所有孩子的文字，同时也忽略导航，页脚和页眉孩子的可能性。
以下内容似乎可以很好地抓住一切：
sel = Selector(response=response)
items = TextItem()
items['text'] = ' '.join(sel.xpath("//body//text()").extract())
但是会返回所有内容。
我一直在试图找到切下如下Xpath的导航仪的方法，但收效甚微。
' '.join(sel.xpath("//body//text()[not(descendant-or-self::nav) and not(descendant-or-self::header) and not(descendant-or-self::footer)]").extract())
是否只有Xpath可以做到这一点？还是我需要利用bs4等更复杂的逻辑和工具？
感谢您的帮助！
答案 0 :(得分：1)
我建议您首先从body节点中删除不需要的节点（例如，使用lxml）
from lxml import html
...
bad_tags = ['nav', 'header', 'footer']
t = html.fromstring(your_main_body_node)
for bad_tag in bad_tags:
for bad_node in t.xpath(f'//{bad_tag}'):
bad_node.getparent().remove(bad_node)
答案 1 :(得分：0)
您可以在XPath中使用适当的谓词[]（使用functions，axes ...来筛选文本节点。
例如，假设您要获取此article的文本（不包括标题，标题，升级链接（'Get your unlimited Newsweek trial'的文本）。
您可以编写如下内容：
//div[@class="article-body v_text paywall"]//text()[parent::p or parent::em or parent::a[@target]]
我们从非常具体的元素中获取文字。
为此article，您可以在Scrapy中使用以下代码（标题或标题将被提取）：
''.join(response.xpath('//div[@class="story-body__inner"]//text()[parent::p or parent::a]').getall())
输出：
'The US has said it will hold off an a threatened hike in tariffs on $7.5bn (£5.75bn) worth of European and UK goods that it imposed as punishment for subsidies for plane-maker Airbus.The move comes as the two sides wrestle to put to an end their 16-year trade battle over state aid for Airbus and American rival Boeing.The US last year raised border taxes on more than 100 items, including jumpers, single-malt whiskies and cheese.It has said the EU has not done enough."The EU and member states have not taken the actions necessary to come into compliance with WTO decisions," America\'s top trade official, Robert Lighthizer, said on Wednesday. "The United States, however, is committed to obtaining a long-term resolution to this dispute. The European Union cautiously welcomed the US decision not to increase the amount of goods subject to tariffs."The Commission acknowledges the decision of the US not to exacerbate the ongoing aircraft dispute by increasing tariffs on European products," an EU spokesperson said.Airbus last month said it would alter some deals responsible for the dispute, saying the changes, including increasing its interest rates on loans with France and Spain, eliminated "any justification" for the US border taxes.The move prompted EU officials to call for an end to "unjustified" tariffs. Many American businesses have also protested the duties, which raise prices for American buyers.On Wednesday Airbus spokesman Clay McConnell said in a statement the company "profoundly regrets that, despite Europe\'s recent actions to achieve full compliance, USTR [US Trade Representative] has decided to maintain tariffs on Airbus aircraft - especially at a time when aviation and other sectors are going through an unprecedented crisis."The US announced tariffs on $7.5bn worth of goods last year after the World Trade Organization ruled that state aid provided to Airbus to launch its A380 and A350 jets was illegal and authorised American retaliation. In February, the US raised the rates being charged on aircraft from 10% to 15%, leaving the 25% duty on other items unchanged.This summer, American officials again threatened to raise tariff rates or make new items subject to the import tax.The items threatened with new duties included salmon fillets, gin and olives.The US is required by law to review the tariffs periodically. On Wednesday it announced minor tweaks to the list, for example, removing sweet biscuits made in the UK and adding jams from France and Germany.Trade lawyer Jamieson Greer, former chief of staff to US Trade Representative Robert Lighthizer, told the BBC: "The reality is that this can all be solved if Airbus took some action to provide restitution".The European Union, which brought its own case challenging American subsidies for Boeing, has threatened to hit the US with tariffs of its own. It is waiting for the World Trade Organization to decide how big such a punishment might be. The US in May said it had eliminated the benefits in dispute. That WTO ruling is expected later this year. "In the absence of a settlement, the EU will be ready to fully avail itself of its own sanction rights," Trade Commissioner Phil Hogan said last month.The issue has also complicated trade talks between the US and the UK.UK Trade Secretary Liz Truss raised the matter in talks with Mr Lighthizer this month, as the two sides held a third round of negotiations.The secretary "was clear that the UK considers these tariffs to be unacceptable and continued to push for their immediate removal," the Department for International Trade said.'
仔细查看HTML代码，在其中查找模式，并相应地编写XPath表达式以选择所需的文本节点。使用ChroPath插件或浏览器中的devtools（F12键）来测试XPath表达式。