Question

我在元素的兄弟节点上进行迭代。但是，我在使用xpath函数提取其中的数据时遇到了问题。我似乎总是得到一个空列表。

# Determine if the cast members are listed in a table or unordered list.
cast_siblings = response.xpath('//h2/span[starts-with(@id,"Cast")]/../following-sibling::*')
for sibling in cast_siblings:
    desc_str = "".join(sibling.xpath('./descendant::text()').extract())
    if "Trailer" in desc_str or "Film Festival" in desc_str or "Comment" in desc_str:
        break

    # Extract the name to get the type of the node.
    node_type = sibling.xpath('name()').extract_first()
    if node_type == 'ul':
        cast_list = sibling.xpath('./ul[1]/li')
        self.parse_list_into_cast_table(cast_list, _id, conn)
    elif node_type == 'table':
        cast_tables = sibling.xpath('./table')
        self.parse_table_into_cast_table(cast_tables, _id, conn)
    else:
        print "Unknown cast element type! " + _id

在上面的代码中，我继续浏览兄弟节点，直到我点击一个包含＆＃34; Trailer＆＃34; ＆＃34;电影节＆＃34;或＆＃34;评论＆＃34;在文中的某个地方。

否则，对于每个兄弟节点，我首先得到它的名字，以找出它是什么类型的节点 - 我对表和无序列表感兴趣。

然而，失败的部分是我打电话给sibling.xpath（＆＃39; ./ table＆＃39;）和/或sibling.xpath（＆＃39; ul [1] / li＆＃39;）。

结果对象始终为空列表。

我使用了scrapy shell并检查了从cast_siblings获得的选择器列表。每个选择器似乎都有效，当我提取它们时，我可以看到内部元素。我无法使用xpath找到它们。

这是我的控制台的摘录，显示空列表结果：

>>> a = response.xpath('//h2/span[starts-with(@id,"Cas")]/../following-sibling::*')
>>> a
[<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Maruy'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Azumi'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Prese'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p><b>Additional Cast Members:</b>\n</p>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<ul><li> <a href="/Yukari_Ito" title="Yu'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Trail'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p>\n<script type="text/javascript" src="'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="mediaplayer"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<script type="text/javascript">\n  jwplay'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Comme'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<br>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="comment-outer"><hr><div id="com'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div align="center"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table cellpadding="2" style="border:0px'>]
>>> a[0]
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>
>>> a[1]
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>
>>> a[1].xpath('./table')
[]

这很奇怪，因为上面例子中的[1]清楚地显示了一个表格。

Answer 1

好像您正在寻找已经是表的节点的表子节点：

node_type = sibling.xpath('name()').extract_first()
if node_type == 'ul':
    ...
elif node_type == 'table':
    cast_tables = sibling.xpath('./table') <---

在这里，您已经知道当前节点是表，并且您正在尝试查找其下的另一个表。所以在这种情况下它应该是：

cast_tables = sibling

值得注意的是，当您测试scrapy选择器时，结果包含根节点标记，即'//h1'将返回<h1>...</h1>

Scrapy遍历兄弟节点--xpath产生空列表

1 个答案: