我正在制作一个项目,其中我使用scrapy从网站中删除项目,但问题是,该网站的前2页的xpath与其他页面的xpath不同。 结果我的蜘蛛只是抓住前两页的项目而只是简单地爬过其他页面。 我怎么能让我的蜘蛛也刮掉页面上的东西?
我也在这里包括我的蜘蛛,以便你可以在必要时看透我的蜘蛛。
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from project2.items import Project2Item
from scrapy.http import Request
class ProjectSpider(BaseSpider):
name = "project2spider"
allowed_domains = ["http://directory.thesun.co.uk/"]
current_page_no = 1
start_urls = [
'http://directory.thesun.co.uk/find/uk/computer-repair'
]
def get_next_url(self, fired_url):
if '/page/' in fired_url:
url, page_no = fired_url.rsplit('/page/', 1)
else:
if self.current_page_no != 1:
#end of scroll
return
self.current_page_no += 1
return "http://directory.thesun.co.uk/find/uk/computer-repair/page/%s" % self.current_page_no
# the parse procedure, and here is the codes which declares which field to scrap.
def parse(self, response):
fired_url = response.url
hxs = HtmlXPathSelector(response)
sites = hxs.select('//div[@class="abTbl "]')
for site in sites:
item = Project2Item()
item['Catogory'] = site.select('span[@class="icListBusType"]/text()').extract()
item['Bussiness_name'] = site.select('a/@title').extract()
item['Description'] = site.select('span[last()]/text()').extract()
item['Number'] = site.select('span[@class="searchInfoLabel"]/span/@id').extract()
item['Web_url'] = site.select('span[@class="searchInfoLabel"]/a/@href').extract()
item['adress_name'] = site.select('span[@class="searchInfoLabel"]/span/text()').extract()
item['Photo_name'] = site.select('img/@alt').extract()
item['Photo_path'] = site.select('img/@src').extract()
#items.append(item)
yield item
next_url = self.get_next_url(fired_url)
if next_url:
yield Request(next_url, self.parse, dont_filter=True)
我需要使用其他页面:sites = hxs.select('//div[@class="icListItem"]')
如何将它包含在我的蜘蛛中,以便它也可以从其他页面中删除项目..
目前正在抓取前两页,只是抓住其他页面。
答案 0 :(得分:0)
到目前为止你做了什么? 一种解决方案是在调用下一页时使用作为元数据传递的类索引参数。类似的东西:
def parse(self, response):
hxs = HtmlXPathSelector(response)
2nd_xpath = False
try:
if response.meta['index'] > 1:
2nd_xpath = True
index = response.meta['index']
except KeyError:
index = 0
sites = (hxs.select('//div[@class="icListItem"]') if 2nd_xpath
else hxs.select('//div[@class="abTbl "]'))
...
request = Request(next_url, self.parse, dont_filter=True)
request.meta['index'] = index + 1
yield request
该代码确实可以改进,但你明白了。