Question

以下是我尝试选择 2016的HTML。

<span id="titleYear">
  "("
  <a href="/year/2016/?ref_=tt_ov_inf">2016</a>
  ")"
</span>

这是XPath语句：//span[@id='titleYear']/a/text()

不幸的是，由于某种原因，该语句会选择<a href="/year/2016/?ref_=tt_ov_inf">2016</a>。

//span[@id='titleYear']/a/text()返回与//span[@id='titleYear']/a和//span[@id='titleYear']/a[text()]相同的结果。

为什么text()在此上下文中没有效果？

这是因为2016还没有被认定为文字节点吗？

值得注意的是，我使用Anaconda和Python 3.6.5以及Scrapy 1.5.0。

Python脚本

import scrapy

class IMDBcrawler(scrapy.Spider):
    name = 'imdb'
    def start_requests(self):
        pages = []
        count = 1
        limit = 10
        while (count <= limit):
            str_number = '%07d' % count
            pages.append('https://www.imdb.com/title/tt' + str_number)
            count += 1
        for url in pages:
            yield scrapy.Request(url=url, callback=self.parse)
    def parse(self, response):
        yield {
            'nom': response.xpath('//div[@class="title_wrapper"]/h1/text()').extract_first(),
            'ano': response.xpath('//span[@id="titleYear"]/a/text()').extract_first(),
        }

输出

[
  {
    "nom": "Chinese Opium Den\u00a0",
    "ano": "<a href=\"\/year\/1894\/?ref_=tt_ov_inf\">1894<\/a>"
  },
  {
    "nom": "Pauvre Pierrot\u00a0",
    "ano": "<a href=\"\/year\/1892\/?ref_=tt_ov_inf\">1892<\/a>"
  },
  {
    "nom": "Carmencita\u00a0",
    "ano": "<a href=\"\/year\/1894\/?ref_=tt_ov_inf\">1894<\/a>"
  },
  {
    "nom": "Un bon bock\u00a0",
    "ano": "<a href=\"\/year\/1892\/?ref_=tt_ov_inf\">1892<\/a>"
  },
  {
    "nom": "Blacksmith Scene\u00a0",
    "ano": "<a href=\"\/year\/1893\/?ref_=tt_ov_inf\">1893<\/a>"
  },
  {
    "nom": "Corbett and Courtney Before the Kinetograph\u00a0",
    "ano": "<a href=\"\/year\/1894\/?ref_=tt_ov_inf\">1894<\/a>"
  },
  {
    "nom": "Employees Leaving the Lumi\u00e8re Factory\u00a0",
    "ano": "<a href=\"\/year\/1895\/?ref_=tt_ov_inf\">1895<\/a>"
  },
  {
    "nom": "Miss Jerry\u00a0",
    "ano": "<a href=\"\/year\/1894\/?ref_=tt_ov_inf\">1894<\/a>"
  },
  {
    "nom": "Le clown et ses chiens\u00a0",
    "ano": "<a href=\"\/year\/1892\/?ref_=tt_ov_inf\">1892<\/a>"
  },
  {
    "nom": "Edison Kinetoscopic Record of a Sneeze\u00a0",
    "ano": "<a href=\"\/year\/1894\/?ref_=tt_ov_inf\">1894<\/a>"
  }
]

感谢。

Answer 1

不确定使用Scrapy的问题是什么，但是在请求的帮助下直接使用lxml，使用xpath的更简单的findtext可以正常工作：

import requests

from lxml import html

pages = []

for count in range(1, 10):
    str_num = '%07d' % count
    res = html.fromstring(requests.get('https://www.imdb.com/title/tt' + str_num).text)
    pages.append({'nom': res.findtext('.//div[@class="title_wrapper"]/h1'), 'ano': res.findtext('.//span[@id="titleYear"]/a')})

结果：

In [40]: pages
Out[40]:
[{'ano': '1894', 'nom': 'Carmencita\xa0'},
 {'ano': '1892', 'nom': 'Le clown et ses chiens\xa0'},
 {'ano': '1892', 'nom': 'Pauvre Pierrot\xa0'},
 {'ano': '1892', 'nom': 'Un bon bock\xa0'},
 {'ano': '1893', 'nom': 'Blacksmith Scene\xa0'},
 {'ano': '1894', 'nom': 'Chinese Opium Den\xa0'},
 {'ano': '1894', 'nom': 'Corbett and Courtney Before the Kinetograph\xa0'},
 {'ano': '1894', 'nom': 'Edison Kinetoscopic Record of a Sneeze\xa0'},
 {'ano': '1894', 'nom': 'Miss Jerry\xa0'}]

XPath语句未按预期解析

1 个答案: