我想从网页中的多个表中抓取内容,HTML代码如下:
<div class="fixtures-table full-table-medium" id="fixtures-data">
<h2 class="table-header"> Date 1 </h2>
<table class="table-stats">
<tbody>
<tr class='preview' id='match-row-EFBO755307'>
<td class='details'>
<p>
<span class='team-home teams'>
<a href='random_team'>team 1</a>
</span>
<span class='team-away teams'>
<a href='random_team'>team 2</a>
</span>
</p>
</td>
</tr>
<tr class='preview' id='match-row-EFBO755307'>
<td class='match-details'>
<p>
<span class='team-home teams'>
<a href='random_team'>team 3</a>
</span>
<span class='team-away teams'>
<a href='random_team'>team 4</a>
</span>
</p>
</td>
</tr>
</tbody>
</table>
<h2 class="table-header"> Date 2 </h2>
<table class="table-stats">
<tbody>
<tr class='preview' id='match-row-EFBO755307'>
<td class='match-details'>
<p>
<span class='team-home teams'>
<a href='random_team'>team X</a>
</span>
<span class='team-away teams'>
<a href='random_team'>team Y</a>
</span>
</p>
</td>
</tr>
<tr class='preview' id='match-row-EFBO755307'>
<td class='match-details'>
<p>
<span class='team-home teams'>
<a href='random_team'>Team A</a>
</span>
<span class='team-away teams'>
<a href='random_team'>Team B</a>
</span>
</p>
</td>
</tr>
</tbody>
</table>
</div>
日期下有更多的比赛(9或2或1,取决于该日期比赛的比赛)和没有。表格是63(等于天数)
我想为每个日期提取球队之间的比赛,以及哪支球队在主场以及哪支球队离开。
我正在使用scrapy shell并尝试以下命令:
title = sel.xpath("//td[@class = 'match-details']")[0]
l_home = title.xpath("//span[@class = 'team-home teams']/a/text()").extract()
这打印了主队的名单,并打印了所有客队的名单,
l_Away = title.xpath("//span[@class = 'team-away teams']/a/text()").extract()
这给了我所有日期的清单:
sel.xpath("/html/body/div[3]/div/div/div/div[4]/div[2]/div/h2/text()").extract()
我想要的是所有日期获得一天比赛(以及哪支球队在主场和客场比赛)
我的items.py应该是这样的:
date = Field()
home_team = Field()
away_team2 = Field()
请帮我编写解析函数和项类。
提前致谢。
答案 0 :(得分:3)
以下是来自scrapy shell
的示例逻辑:
>>> for table in response.xpath('//table[@class="table-stats"]'):
... date = table.xpath('./preceding-sibling::h2[1]/text()').extract()[0]
... print date
... for match in table.xpath('.//tr[@class="preview" and @id]'):
... home_team = match.xpath('.//span[@class="team-home teams"]/a/text()').extract()[0]
... away_team = match.xpath('.//span[@class="team-away teams"]/a/text()').extract()[0]
... print home_team, away_team
...
Date 1
team 1 team 2
team 3 team 4
Date 2
team X team Y
Team A Team B
在parse()
方法中,您需要在内部循环中实例化Item
实例并yield
实例化它:
def parse(self, response):
for table in response.xpath('//table[@class="table-stats"]'):
date = table.xpath('./preceding-sibling::h2[1]/text()').extract()[0]
for match in table.xpath('.//tr[@class="preview" and @id]'):
home_team = match.xpath('.//span[@class="team-home teams"]/a/text()').extract()[0]
away_team = match.xpath('.//span[@class="team-away teams"]/a/text()').extract()[0]
item = MyItem()
item['date'] = date
item['home_team'] = home_team
item['away_team'] = away_team
yield item
其中Myitem
将是:
class MyItem(Item):
date = Field()
home_team = Field()
away_team = Field()