Question

大家好，我已经使用requests-futures库编写了一个多线程请求和响应处理程序。

然而，它似乎非常缓慢，并不像我想象的那样异步。输出速度很慢而且按顺序排列，而不是像我预期的那样正确穿线。

我的问题是为什么我的代码速度慢，我该怎么做才能加快速度？一个例子就是很棒。

这是代码：

#!/usr/bin/python
import requests
import time
from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession

session = FuturesSession(executor=ThreadPoolExecutor(max_workers=12))

def responseCallback( sess, resp ):
    response = resp.text
    if not "things are invalid" in response in response:
        resp.data = "SUCCESS %s" % resp.headers['content-length']
    else:
        resp.data = "FAIL %s" % resp.headers['content-length']

proxies = {
"http":"http://localhost:8080",
"https":"https://localhost:8080"
}

url = 'https://www.examplehere.com/blah/etc/'
headers= {
'Host':'www.examplehere.com',
'Connection':'close',
'Cache-Control':'max-age=0',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Origin':'https://www.examplehere.com',
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/533.32 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.123 Chrome/34.0.1847.123 Safari/337.12',
'Content-Type':'application/x-www-form-urlencoded',
'Referer':'https://www.exampleblah.etc/',
'Accept-Encoding':'gzip,deflate,sdch',
'Accept-Language':'en-US,en;q=0.8,de;q=0.6',
'Cookie':'blah=123; etc=456;',
}

for n in range( 0, 9999 ):
    #wibble = n.zfill( 4 )
    wibble = "%04d" % n
    payload = { 
    'name':'test',
    'genNum':wibble,
    'Button1':'Push+Now'
    }
    #print payload
    #r = requests.post( url, data=payload, headers=headers, proxies=proxies, verify=False )
    future = session.post( url, data=payload, headers=headers, verify=False, background_callback=responseCallback )
    response = future.result()
    print( "%s : %s" % ( wibble, response.data ) )

理想情况下，我想使用我已经使用的库来修复我的实际代码，但如果由于某种原因它很糟糕我会接受建议......

编辑：我目前正在使用带有concurrent.futures backport的python2。

编辑：慢 - 大约一个请求一秒钟，而不是并发，但一个接一个，所以request1，response1，request2，response2 - 我希望它们在请求发出并进入多个线程时交错？

Answer 1

以下代码是提交多个请求的另一种方法，一次处理多个请求，然后打印出结果。结果在准备好时打印，不一定与提交时的顺序相同。

它还使用广泛的日志记录来帮助调试问题。它捕获有效负载以进行日志记录。多线程代码很难，因此更多日志更好！

源

import logging, sys

import concurrent.futures as cf
from requests_futures.sessions import FuturesSession

URL = 'http://localhost'
NUM = 3

logging.basicConfig(
    stream=sys.stderr, level=logging.INFO,
    format='%(relativeCreated)s %(message)s',
    )

session = FuturesSession()
futures = {}

logging.info('start')
for n in range(NUM):
    wibble = "%04d" % n
    payload = { 
        'name':'test',
        'genNum':wibble,
        'Button1':'Push+Now'
    }
    future = session.get( URL, data=payload )
    futures[future] = payload

logging.info('requests done, waiting for responses')

for future in cf.as_completed(futures, timeout=5):
    res = future.result()
    logging.info(
        "wibble=%s, %s, %s bytes",
        futures[future]['genNum'],
        res,
        len(res.text),
    )

logging.info('done!')

输出

69.3101882935 start
77.9430866241 Starting new HTTP connection (1): localhost
78.3731937408 requests done, waiting for responses
79.4050693512 Starting new HTTP connection (2): localhost
84.498167038 wibble=0000, <Response [200]>, 612 bytes
85.0481987 wibble=0001, <Response [200]>, 612 bytes
85.1981639862 wibble=0002, <Response [200]>, 612 bytes
85.2642059326 done!

python请求 - 期货慢 - 没有正确的线程？

1 个答案:

源

输出