Question

我有一个参数元组，必须在scrapy中将其与我的请求一起传递，但是在scrapy.Request或response.follow中找不到任何带有参数的参数。我应该放在哪里？

params = (
            ('search_criteria[current_page]', '1'),
            ('search_criteria[excludeAggregations]', 'false'),
            ('search_criteria[filter_groups][0][filters][0][field]', 'status'),
            ('search_criteria[filter_groups][0][filters][0][value]', '1'),
            ('search_criteria[filter_groups][1][filters][0][field]', 'visibility'),
            ('search_criteria[filter_groups][1][filters][0][value]', '4'),
            ('search_criteria[filter_groups][2][filters][0][field]', 'category_ids'),
            ('search_criteria[filter_groups][2][filters][0][value]', '474'),
            ('search_criteria[page_size]', '20'),
            ('search_criteria[request_index]', 'product'),
            ('search_criteria[sort_orders][0][direction]', 'ASC'),
            ('search_criteria[sort_orders][0][field]', 'price'),
            ('tracker', 'a2zxlqwzj6-1567316733-1'),
        )
URL= "https://magento.ackermans.co.za/rest/default/V1/pepkor/searchV2"
scrapy.Request(URL,params=???)

Answer 1

可能是How to specify parameters on a Request using scrapy的副本。

对于您的特定情况，假设您使用的是python 3.6+，则下面的方法会起作用

params = (
            ('search_criteria[current_page]', '1'),
            ('search_criteria[excludeAggregations]', 'false'),
            ('search_criteria[filter_groups][0][filters][0][field]', 'status'),
            ('search_criteria[filter_groups][0][filters][0][value]', '1'),
            ('search_criteria[filter_groups][1][filters][0][field]', 'visibility'),
            ('search_criteria[filter_groups][1][filters][0][value]', '4'),
            ('search_criteria[filter_groups][2][filters][0][field]', 'category_ids'),
            ('search_criteria[filter_groups][2][filters][0][value]', '474'),
            ('search_criteria[page_size]', '20'),
            ('search_criteria[request_index]', 'product'),
            ('search_criteria[sort_orders][0][direction]', 'ASC'),
            ('search_criteria[sort_orders][0][field]', 'price'),
            ('tracker', 'a2zxlqwzj6-1567316733-1'),
        )

def to_url_params(params):
  param_string = ""
  for param in params:
    # iterate over params and append each into 
    param_string = f'{param_string}&{param[0]}={param[1]}'  
  # get rid of first & symbol
  return param_string[1:]

param_string = to_url_params(params)

URL= f"https://magento.ackermans.co.za/rest/default/V1/pepkor/searchV2?{param_string}"

Answer 2

如果您不想为此编写自定义函数，则可以将元组更改为字典，并使用urllib.parse（python3）中的urlencode：

from urllib.parse import urlencode

params = {
            'search_criteria[current_page]': '1',
            'search_criteria[excludeAggregations]': 'false',
            'search_criteria[filter_groups][0][filters][0][field]': 'status',
            'search_criteria[filter_groups][0][filters][0][value]': '1',
            'search_criteria[filter_groups][1][filters][0][field]': 'visibility',
            'search_criteria[filter_groups][1][filters][0][value]': '4',
            'search_criteria[filter_groups][2][filters][0][field]': 'category_ids',
            'search_criteria[filter_groups][2][filters][0][value]': '474',
            'search_criteria[page_size]': '20',
            'search_criteria[request_index]': 'product',
            'search_criteria[sort_orders][0][direction]': 'ASC',
            'search_criteria[sort_orders][0][field]': 'price',
            'tracker': 'a2zxlqwzj6-1567316733-1',
        }

encoded_params = urlencode(params)
url = f"https://magento.ackermans.co.za/rest/default/V1/pepkor/searchV2?{encoded_params}"

将请求网址的* params *放置在scrapy请求中的位置

2 个答案: