Question

我需要将嵌套的dict作为参数传递给get请求。

以下是它的工作方式

query = {%22channel%22:%22rent%22,%22page%22:2,%22pageSize%22:12,%22filters%22:{%22agencyIds%22:[%22CBPHMG%22]}}

以下是我在Scrapy日志中获得的内容：

%7B%22pageSize%22:%20300,%20%22page%22:%208,%20%22channel%22:%20%22rent%22,%20%22filters%22:%20%7B%22agencyIds%22:%20%22VDTUED%22%7D%7D

问题在于方形和花括号。

我现在所做的只是json.dumps(dict)并将其附加到网址。我还尝试使用反斜杠来防止更改符号。没有人。

 q = {"channel":"sold","page":1,"pageSize":300,"filters":{"agencyIds":["PRDNEW"]}}
 query = json.dumps(q)
 query = query.replace('"', '\\"')
 url = url + query

以下代码也适用于python3请求。

import requests

url = "https://services.realestate.com.au/services/listings/search"

querystring = {"query":"{\"channel\":\"buy\",\"page\":2,\"pageSize\":12,\"filters\":{\"agencyIds\":[\"CBPHMG\"]}}"}

headers = {'cache-control': 'no-cache'}

response = requests.request("GET", url, headers=headers, params=querystring)

print(response.text)

Answer 1

您可以使用w3lib.url.add_or_replace_parameter将query参数附加到网址。它将以与python-requests相同的方式进行URL编码：

$ scrapy shell
2017-07-18 11:03:28 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)
(...)
>>> url = "https://services.realestate.com.au/services/listings/search"
>>> querystring = {"query":"{\"channel\":\"buy\",\"page\":2,\"pageSize\":12,\"filters\":{\"agencyIds\":[\"CBPHMG\"]}}"}

这与您的python-requests示例输入数据相同。

将add_or_replace_parameter与参数名称及其值一起使用（注意：Scrapy已经依赖于w3lib。）：

>>> from w3lib.url import add_or_replace_parameter
>>> add_or_replace_parameter(url, 'query', querystring['query'])
'https://services.realestate.com.au/services/listings/search?query=%7B%22channel%22%3A%22buy%22%2C%22page%22%3A2%2C%22pageSize%22%3A12%2C%22filters%22%3A%7B%22agencyIds%22%3A%5B%22CBPHMG%22%5D%7D%7D'

在Scrapy shell中，获取新URL将获得JSON响应，如预期的那样：

>>> new_url = add_or_replace_parameter(url, 'query', querystring['query'])
>>> fetch(new_url)
2017-07-18 11:04:45 [scrapy.core.engine] INFO: Spider opened
2017-07-18 11:04:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://services.realestate.com.au/services/listings/search?query=%7B%22channel%22%3A%22buy%22%2C%22page%22%3A2%2C%22pageSize%22%3A12%2C%22filters%22%3A%7B%22agencyIds%22%3A%5B%22CBPHMG%22%5D%7D%7D> (referer: None)


>>> import json
>>> data = json.loads(response.text)
>>> data.keys()
dict_keys(['prettyUrl', 'totalResultsCount', 'resolvedQuery', '_links', 'tieredResults', 'channel'])

>>> from pprint import pprint
>>> pprint(data)
{'_links': {'adCall': {'href': 'https://sasinator.realestate.com.au/rea/hserver/site=rea/area=buy.resultslist/proptype=villa/constructionStatus=established/sub=marsden/state=qld/pcode=4132/region=logan/price=200k_300k/platform={platform}/version={version}/pos={position}/size={size}/viewid={viewId}/random={random}',
                       'templated': True},
            'canonical': {'href': 'http://www.realestate.com.au/buy/by-cbphmg/list-2'},
            'exclusiveShowcaseUrl': {'href': 'https://services.realestate.com.au/services/listings/exclusiveShowcase?query=%7B%22propertyTypes%22:[],%22atlasIds%22:[],%22channel%22:%22buy%22%7D'},
            'neighbourhoodsUrl': {'href': 'http://www.realestate.com.au/neighbourhoods?state=qld'},
            'next': {},
            'ofi': {'href': 'https://services.realestate.com.au/services/listings/ofi/{date}/daytotals?query=%7B%22channel%22:%22buy%22,%22pageSize%22:%2212%22,%22page%22:%222%22,%22filters%22:%7B%22agencyIds%22:%5B%22CBPHMG%22%5D%7D%7D',
                    'templated': True},
            'prettyUrl': {'href': '/buy/by-cbphmg/list-2'},
            'saveSearchUrl': {'href': 'https://www.realestate.com.au/saved-searches/#/save?search=%7B%22channel%22:%22buy%22,%22pageSize%22:%2212%22,%22page%22:%222%22,%22filters%22:%7B%22agencyIds%22:%5B%22CBPHMG%22%5D%7D%7D'},
            'self': {'href': 'https://services.realestate.com.au/services/listings/search?query=%7B%22channel%22:%22buy%22,%22pageSize%22:%2212%22,%22page%22:%222%22,%22filters%22:%7B%22agencyIds%22:%5B%22CBPHMG%22%5D%7D%7D'}},
 'channel': 'buy',
 'prettyUrl': '/buy/by-cbphmg/list-2',
 'resolvedQuery': {'channel': 'buy',
                   'filters': {'agencyIds': ['CBPHMG']},
                   'page': '2',
                   'pageSize': '12'},
 'tieredResults': [{'count': 11,
                    'results': [{...}],
                    'tier': 1}],
 'totalResultsCount': 23}

防止scrapy从url中删除方形和花括号

1 个答案: