为grequest创建URL列表

时间:2015-09-28 18:41:51

标签: python-2.7 asynchronous python-requests grequests

拥有多个有效负载dictionarys,如何才能更好地创建格式化网址列表,以便grequest可以迭代?

  

payload_single:{'search':'51 F ST SW,AUBURN KING 98001,WA,USA','app_code':'xyz','app_id':'xyz','lod':'9'} < / p>      

payload_single:{'search':'55 F ST SW,AUBURN KING 98001,WA,USA','app_code':'xyz','app_id':'xyz','lod':'9'} < / p>

能够通过使用以下代码迭代URLS的grequest发送请求:

unsentrequests=(grequests.get(u, hooks = {'response' : do_something}) for u in urls) 
responses=grequests.map(unsentrequests)

对于请求的单个调用我将使用此:

row = requests.get(url_single, params=payload_single)

编辑1 @rebeling

    for url in urls:
        unsent_request.append(grequests.get(url_single,
                                        hooks={'response': resphandler()},
                                        params=url))

def resphandler():
    rs = grequests.map(unsent_request)

    for r in rs:
        print r

1 个答案:

答案 0 :(得分:1)

import grequests  

def do_something(response, *args, **kwargs):  
    print response  


payloads = [
    {'search': '51 F ST SW,AUBURN KING 98001,WA,USA', 'app_code': 'xyz', 'app_id': 'xyz', 'lod': '9'},
    {'search': '55 F ST SW,AUBURN KING 98001,WA,USA', 'app_code': 'xyz', 'app_id': 'xyz', 'lod': '9'}]

unsent_request = []  

请参阅grequests代码line 35: 接受与Session.request相同的参数和一些额外的参数,因此我们像处理请求一样使用参数。

for payload_single in payloads:
    unsent_request.append(grequests.get('http://www.google.com', 
                                        hooks={'response': do_something}, 
                                        params=payload_single))

print grequests.map(unsent_request)