Question

我正在尝试从“http://eastasiaeg.com/en/laptop-in-egypt”中删除产品列表使用Scrapy。

部分产品动态加载，并尝试构建Scrapy请求。但它有些问题。请帮助。

# -*- coding: utf-8 -*-
import scrapy

from v4.items import Product


class IntelEGEastasiaegComSpider(scrapy.Spider):
    name = "intel_eg_eastasiaeg_com_py"

    start_urls = [
            'http://eastasiaeg.com/en/laptop-in-egypt'
        ]

    def start_requests(self):
        request_body = {"categoryId":"3","manufacturerId":"0","vendorId":"0","priceRangeFilterModel7Spikes":{"CategoryId":"3","ManufacturerId":"0","VendorId":"0","SelectedPriceRange":{},"MinPrice":"2400","MaxPrice":"44625"},"specificationFiltersModel7Spikes":{"CategoryId":"3","ManufacturerId":"0","VendorId":"0","SpecificationFilterGroups":[{"Id":"27","FilterItems":[{"Id":"103","FilterItemState":"Unchecked"},{"Id":"104","FilterItemState":"Unchecked"},{"Id":"105","FilterItemState":"Unchecked"},{"Id":"110","FilterItemState":"Unchecked"}]},{"Id":"11","FilterItems":[{"Id":"302","FilterItemState":"Unchecked"},{"Id":"75","FilterItemState":"Unchecked"}]},{"Id":"6","FilterItems":[{"Id":"21","FilterItemState":"Unchecked"},{"Id":"24","FilterItemState":"Unchecked"},{"Id":"25","FilterItemState":"Unchecked"},{"Id":"26","FilterItemState":"Unchecked"}]},{"Id":"5","FilterItems":[{"Id":"1069","FilterItemState":"Unchecked"},{"Id":"1078","FilterItemState":"Unchecked"},{"Id":"1118","FilterItemState":"Unchecked"},{"Id":"1862","FilterItemState":"Unchecked"}]},{"Id":"2","FilterItems":[{"Id":"8","FilterItemState":"Unchecked"},{"Id":"10","FilterItemState":"Unchecked"},{"Id":"1451","FilterItemState":"Unchecked"},{"Id":"1119","FilterItemState":"Unchecked"}]},{"Id":"8","FilterItems":[{"Id":"61","FilterItemState":"Unchecked"},{"Id":"62","FilterItemState":"Unchecked"},{"Id":"63","FilterItemState":"Unchecked"}]},{"Id":"333","FilterItems":[{"Id":"2460","FilterItemState":"Unchecked"}]}]},"attributeFiltersModel7Spikes":"null","manufacturerFiltersModel7Spikes":{"CategoryId":"3","ManufacturerFilterItems":[{"Id":"2","FilterItemState":"Unchecked"},{"Id":"1","FilterItemState":"Unchecked"},{"Id":"3","FilterItemState":"Unchecked"},{"Id":"6","FilterItemState":"Unchecked"}]},"vendorFiltersModel7Spikes":"null","pageNumber":"2","orderby":"10","viewmode":"grid","pagesize":"null","queryString":"","shouldNotStartFromFirstPage":"true","onSaleFilterModel":"null","keyword":"","searchCategoryId":"0","searchManufacturerId":"0","priceFrom":"","priceTo":"","includeSubcategories":"False","searchInProductDescriptions":"False","advancedSearch":"False","isOnSearchPage":"False"}
        for body in request_body:
            request_body = body
            yield scrapy.Request('http://eastasiaeg.com/en/getFilteredProducts',
                                 method="POST",
                                 body=request_body,
                                 callback=self.parse,
                                 headers={'Content-type': 'application/json; charset=UTF-8'}, )

    def parse(self, response):
        print response.body

Answer 1

当您想要在其中包含表单数据的POST请求时，您应该使用compute()。

scrapy.FormRequest

你的方法也可行，但你的for循环在这里没有多大意义 def start_requests(self): form_data = {} # your formdata yield scrapy.FormRequest(url, formdata=form_data)遍历您for body in request_body:所在字典的键，您基本上只需要一个键就可以发出24个请求。
所以要使用request_body尝试：

scrapy.Request

P.S。 scrapy请求默认为def start_requests(self): form_data = {} # your formdata # Request only takes string as body so you need to # convert python dict to string request_body = json.dumps(form_data) yield scrapy.Request('http://eastasiaeg.com/en/getFilteredProducts', method="POST", body=request_body, headers={'Content-Type': 'application/json; charset=UTF-8'}, ) # Usually Content-Type matters here a lot.回调，因此您无需指定它。

使用POST方法进行Scrapy制作请求

1 个答案: