Python Scrapy多页抓取

时间:2019-03-22 11:35:01

标签: python scrapy

我正在尝试将+1添加到网址并抓取多个页面,但是我在+ x部分上总是遇到错误

import scrapy

#formatted here
class HeadphonesSpider(scrapy.Spider):

name = "headphones"

def start_requests(self):

    for x in range(5):
        x = str (x)
        url = 'https://www. domain .com/page=' + x

        yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        img_urls = response.css('div.job-title a::attr(href)').extract()

        with open('urls.txt', 'w') as f:
            for u in img_urls:
                f.write(u + "\n")

0 个答案:

没有答案