我正在尝试获取有关site上产品规格的数据。默认视图为价格,但我想进入规格视图。由于该按钮执行POST请求,因此我尝试使用FormRequest对其进行仿真,但是没有任何反应。
这是我正在使用的代码:
import scrapy
from scrapy import Spider
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
from Demo.items import DemoItem, Distributor
class ProductSpider(scrapy.Spider):
name='nameCode'
start_urls = ['https://octopart.com/electronic-parts/integrated-circuits-ics' ]
def parse(self,response):
token = response.xpath('//button[@name="serp-view"]/@value').extract_first()
return FormRequest.from_response(response, url="https://octopart.com/electronic-parts/integrated-circuits-ics", formdata={'serp-view':token}, callback=self.scrape_pages)
def scrape_pages(self,response):
open_in_browser(response)
输出:
[scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://octopart.com/electronic-parts/integrated-circuits-ics?q=> (referer: https://octopart.com/electronic-parts/integrated-circuits-ics)
[scrapy.core.engine] INFO: Closing spider (finished)