使用scrapy模拟POST的FormRequest不起作用

时间:2019-03-16 12:18:30

标签: python scrapy web-crawler

我正在尝试获取有关site上产品规格的数据。默认视图为价格,但我想进入规格视图。由于该按钮执行POST请求,因此我尝试使用FormRequest对其进行仿真,但是没有任何反应。

这是我正在使用的代码:

import scrapy
from scrapy import Spider
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
from Demo.items import DemoItem, Distributor

class ProductSpider(scrapy.Spider):
    name='nameCode'

    start_urls = ['https://octopart.com/electronic-parts/integrated-circuits-ics' ]

    def parse(self,response):
        token = response.xpath('//button[@name="serp-view"]/@value').extract_first()
        return FormRequest.from_response(response, url="https://octopart.com/electronic-parts/integrated-circuits-ics", formdata={'serp-view':token}, callback=self.scrape_pages)

    def scrape_pages(self,response):
        open_in_browser(response)        

输出:

[scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics?q=> (referer: https://octopart.com/electronic-parts/integrated-circuits-ics)
[scrapy.core.engine] INFO: Closing spider (finished)

0 个答案:

没有答案