Question

我现在有一个包含80个用户名的列表，我的脚本会检查每个用户名是否存在。然而，它需要比我喜欢的时间长一点，所以我想知道是否有任何我可以做的事情来加快检查每个用户名是否存在需要多长时间。

# ------------------------------
# Mass Kik Username Checker
# Script Made by: Ski
# ------------------------------

import requests, threading

def check(username):
    try:
        req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code

        if req == 302:
            return False
        if req == 200:
            return True
    except Exception as e:
        print e
        exit()


def _loadList(filename):
    item_list = []
    for item in str(open(filename, "r").read()).split("\n"):
        item_list.append(item)
    return item_list

def _thread(items):
    global _usernames
    for username in _usernames[items[0]:items[1]]:
        exists = check(username)

        if exists:
            print username+" exists\n"
        if not exists:
            print username+" doesn't exist\n"

if __name__ == '__main__':
    _usernames = _loadList("usernames.txt")

    thread1 = threading.Thread(target=_thread, args=([0, 20], )).start()
    thread2 = threading.Thread(target=_thread, args=([20, 40], )).start()
    thread3 = threading.Thread(target=_thread, args=([40, 60], )).start()
    thread4 = threading.Thread(target=_thread, args=([60, 80], )).start()

Answer 1

试用Python 3.x Pool of threads。您可以定义执行请求的工作人员数量。使用更多（例如32）而不是4，可以大大加快代码的速度。

import requests
from concurrent.futures import ThreadPoolExecutor


NUM_OF_WORKERS=32


def check(username):
    try:
        req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code

        if req == 302:
            print(username, " does not exist.")
        if req == 200:
            print(username, "exists.")
    except Exception as error:
        print(error)


usernames = _loadList(filename)

with ThreadPoolExecutor(max_workers=NUM_OF_WORKERS) as pool:
    pool.map(check, usernames)

这使您的代码方式更具可读性。

编辑：现在注意到Python 2.7标签。

Python 2有一个线程池，可在multiprocessing模块下使用。不幸的是，没有记录，因为没有提供任何测试。

import requests
from multiprocessing.pool import ThreadPool


NUM_OF_WORKERS=32


def check(username):
    try:
       req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code

       if req == 302:
           print(username, " does not exist.")
       if req == 200:
           print(username, "exists.")
    except Exception as error:
        print(error)


usernames = _loadList(filename)


pool = ThreadPool(processes=NUM_OF_WORKERS)
pool.map_async(check, usernames)
pool.close()
pool.join()

如果你想要一个更好的Python 2线程池，你可以尝试Pebble module

有什么方法可以加快处理速度吗？

1 个答案: