Question

我在Windows Vista 64位上使用Python.org版本2.7 64位。我有一些Scrapy代码试图在这个链接上解析表格＃Wayne Rooney的比赛历史：＆＃34; ...

http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney 我到目前为止的代码是：

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "wiki"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]

    def parse(self, response):

        for row in response.selector.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]'):
        # Is this row contains goal symbols?
            list_of_goals = row.xpath('//span[@title="Goal"]')
            if list_of_goals:
                list = str(list_of_goals)
                print remove_tags(list).encode('utf-8')     

execute(['scrapy','crawl','wiki'])

这会返回表格中除目标数据之外的所有数据（它也没有返回助攻，但我还没有添加任何逻辑。这段代码是原始作品的开发我没有返回目标或助攻的代码：

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "goal"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]    

    def parse(self, response):
        titles = response.selector.xpath("normalize-space(//title)")
        for titles in titles:
            body = response.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]').extract()
            body2 = "".join(body)

            print remove_tags(body2).encode('utf-8')

execute(['scrapy','crawl','goal'])

HTML源代码中表明目标的声明是：

<span class="incident-wrapper"><span class="incidents-icon ui-icon goal" title="Goal"></span></span>

有谁能告诉我为什么我在顶部列出的代码没有返回用该逻辑得分的目标？是否与球图标用于表示目标而不是单词这一事实有关？

由于

Answer 1

在第一个版本中，您只获得<span class="incidents-icon ui-icon goal" title="Goal"></span>并且没有文字，因此您会得到空字符串 - 因为您remove_tags()。

添加字符串＆＃34; GOAL＆＃34;对于带有＆＃34;目标图标＆＃34;

的行

list_of_goals = row.xpath('//span[@title="Goal"]')

if list_of_goals:        
    list = str(list_of_goals)
    print remove_tags(list).encode('utf-8') + "GOAL" # <-- string

或（因为<span title="Goal">中没有文字）

list_of_goals = row.xpath('//span[@title="Goal"]')

if list_of_goals:
    print "GOAL" # <-- string

编辑：

我使用Scrapy 0.22.2创建了我的版本。

可能我们使用不同版本的Scrapy，因为你的一些功能确实对我不起作用。

我更喜欢css选择器，然后xpath - 它们对我来说更简单。

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.cmdline import execute


class MySpider(Spider):

    name = "goal"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]    

    def parse(self, response):
        sel = Selector(response)

        #titles = sel.xpath("normalize-space(//title)")
        #print 'titles:', titles.extract()[0]
        print
        print 'titles:', "".join( sel.css("title::text").extract() ).strip()
        print

        #rows = sel.xpath('//table[@id="player-fixture"]//tbody//tr')
        rows = sel.css('table#player-fixture tbody tr')

        for row in rows:
            #print 'date:', row.xpath('.//td[@class="date"]/text()').extract()
            #print 'result:', row.xpath('.//td[@class="result"]/a/text()').extract()
            print 'date:', "".join( row.css('.date::text').extract() ).strip()
            print 'result:', "".join( row.css('.result a::text').extract() ).strip()
            print 'team_home:', "".join( row.css('.team.home a::text').extract() ).strip()
            print 'team_away:', "".join( row.css('.team.away a::text').extract() ).strip()
            print 'info:', "".join( row.css('.info::text').extract() ).strip(), "".join( row.css('.info::attr(title)').extract() ).strip()
            print 'rating:', "".join( row.css('.rating::text').extract() ).strip()
            print 'incidents:', ", ".join( row.css('.incidents-icon::attr(title)').extract() ).strip()
            print '-'*40

#execute(['scrapy','crawl','goal'])
execute(['scrapy','runspider','main.py'])

和结果的一部分

titles: Wayne Rooney Match History | WhoScored.com

date: 17-08-2013
result: 1 : 4
team_home: Swansea
team_away: Manchester United
info: 28' Minutes played in this match
rating: 7.26
incidents: Assist, Assist
----------------------------------------
date: 26-08-2013
result: 0 : 0
team_home: Manchester United
team_away: Chelsea
info: 90' Minutes played in this match
rating: 7.03
incidents: 
----------------------------------------
date: 14-09-2013
result: 2 : 0
team_home: Manchester United
team_away: Crystal Palace
info: 90' Minutes played in this match
rating: 8.44
incidents: Man of the Match, Goal
----------------------------------------
date: 17-09-2013
result: 4 : 2
team_home: Manchester United
team_away: Bayer Leverkusen
info: 84' Minutes played in this match
rating: 9.18
incidents: Goal, Goal, Assist
----------------------------------------
date: 22-09-2013
result: 4 : 1
team_home: Manchester City
team_away: Manchester United
info: 90' Minutes played in this match
rating: 7.17
incidents: Goal, Yellow Card
----------------------------------------
date: 25-09-2013
result: 1 : 0
team_home: Manchester United
team_away: Liverpool
info: 90' Minutes played in this match
rating: 
incidents: Man of the Match, Assist
----------------------------------------

Xpath（）方法不使用Scrapy返回结果

1 个答案: