Question

我在制作多处理HTTP请求时遇到了麻烦。

我看到两个问题，其中一个是下面的问题，另一个是我每天增加请求数的试验（见完整代码）

Exception in thread Thread-75:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "requests.py", line 189, in doWork
    doSomethingWithResult(response)
  File "requests.py", line 206, in doSomethingWithResult
    msisdn, mno_code = response
ValueError: too many values to unpack

即使脚本继续工作，我也不能获得超过100 TPS，甚至增加队列数和并发进程。

使用变量发布帖子请求的最佳方法是什么？按照我到目前为止所做的尝试：

from logging.handlers import TimedRotatingFileHandler
import logging

from urlparse import urlparse
from threading import Thread
import httplib, sys
from Queue import Queue
import json

mno_list = {
    101: 'TELECALL - Fixo',
    102: 'Life Telecom - Fixo',
    104: 'BaydeNET - Fixo',
    105: 'T.P.A - Fixo',
    106: 'America NET - Fixo',
    108: 'Global OSI - Fixo',
    109: 'MAHA-TEL TELECOM - Fixo',

}

logHandler = TimedRotatingFileHandler('lista_msisdn.txt', when="midnight")
logFormatter = logging.Formatter('%(message)s', datefmt='%Y-%m-%d %H:%M:%S')
logHandler.setFormatter(logFormatter)

logger = logging.getLogger('time_rotate_logger')
logger.addHandler(logHandler)
logger.setLevel(logging.DEBUG)


concurrent = 1000

def doWork():
    while True:
        url = 'http://xxxxxxxxxxxxxxxxxxx'
        msisdn = q.get()

        response = getStatus(url, msisdn)

        doSomethingWithResult(response)
        q.task_done()

def getStatus(url, msisdn):
    try:
        url = urlparse(url)
        payload = "msisdns=%s" % msisdn
        conn = httplib.HTTPConnection(url.netloc)
        conn.request("POST", "/check/", payload)
        res = conn.getresponse()
        response = json.loads(res.read())
        return (msisdn, response['carrierId'])
    except:
        return "error"

def doSomethingWithResult(response):
    msisdn, mno_code = response
    mno_name = mno_list.get(mno_code, 'NOK:%s' % mno_code)

    logger.debug('%s,%s' % (msisdn, mno_name))

q = Queue(concurrent * 2)
for i in range(concurrent):
    t = Thread(target=doWork)
    t.daemon = True
    t.start()
try:
    for msisdn in open('msisdn.txt'):
        q.put(msisdn.strip())
    q.join()
except KeyboardInterrupt:
    sys.exit(1)

Python使用多线程发布请求

0 个答案: