Python - 如何使下面的函数成为线程？

时间：2015-10-14 13:06:58

标签： python multithreading python-multiprocessing

我对python和编程一般都很陌生 - 寻找关于如何收紧以下功能并减少一些时间的建议。一些背景信息：

要求是我收集给定顶级文件夹中每个子文件/文件夹的名称和ID。问题是，我请求数据的服务器只返回单个文件夹的内容，响应将始终指定返回的对象是文件还是文件夹。

（伪代码示例，只是试图快速演示）：

Top_level_folderid = 1111
url = "fileserverapi.couldbebetter.com/thismighttakeawhile"
post(url, data=Top_level_folderid)
response({"jim's folder" : id=1234, filetype=folder}, {"weird_video.mp4" : id=4321, filetype=file})

然后我必须遍历每个响应并回发到服务器以获取下一个集合，在某些情况下，整个顶级文件夹可能包含多达15,000个文件夹和30,000多个随机分发的文件，其中一些文件夹包含1个文件和15个文件夹和其他包含7个文件和2个子文件夹等的文件夹。

API本身反应非常快，但是我不知道它可以处理多少并发连接所以我需要能够调整并找到函数内的最佳位置，猜测它可以处理10个以上的任何地方。 -50。我的功能现在是：

def drill_folder_loop(folder_list, project_id, directory, jsession_id):
    count = 0
    temp_folder_list = folder_list #\\ A list of dicts [{folder_name1 : folder_id1}, {folder_name2 : folder_id2}]
    while count == 0:
        temp_folder_list1 = []
        for folder in temp_folder_list:
            f_name = folder.keys()[0] #// folder_name (not actually used here)
            f_id = folder.values()[0] #// folder id
            folder_dict = list_folders_files(f_id, jsession_id) #// list_folders_files posts to the api and builds the response back into a list of dicts, same as the original passed to this function.
            folder_list = process_folder_files(folder_dict, directory, jsession_id) #// somewhat irrelevant to the question - I have to commit the file data to a DB, I could strip the folder list in this function but since i need it there I just return it instead.
            process_recipients = recipient_process(folder_dict, "no", f_id,
                                                   directory, project_id)#// more irrelevant but necessary processing.
            for i in range(0, len(folder_list)):
                append_f = folder_list[i]
                temp_folder_list1.append(append_f)#// append new folders to list outside loop
        temp_folder_list = [] #// empty temp_folder_list, loop may contain more than a single folder so I have to empty it once I've processed all the folders
        for i in range(0, len(temp_folder_list1)):#// Put the new folder list into temp_folder_list so the loop restarts
            append_f2 = temp_folder_list1[i]
            temp_folder_list.append(append_f2)
        if not temp_folder_list: #// end the loop if there are no more sub-folders
            count += 1
    return log_info("Folder loop complete")

重新阅读这是变量命名的一个很好的教训...并不是最简洁的..代码本身工作正常但是你可以想象到现在，它需要很长时间才能完成成千上万的文件夹......关于如何将其变成多线程/处理野兽的任何建议/方向？感谢您抽时间阅读！

编辑：

为了清楚起见，我不想在循环中处理文件夹，而是希望在线程中将它们关闭以具有多个文件夹，从而同时发布请求和响应，以便整个过程花费更少的时间。现在它一次只循环一个文件夹..希望这澄清..

修改 根据Noctis Skytower的建议，我做了一些小改动来支持python 2.7（Queue vs queue和.clock（）而不是perf_counter（））。太过分了！我遇到的问题是，当我将正在运行的线程更改为1时，它完美地完成 - 当我因某种原因（并随机）将其增加回25时，dfl_worker（）中的变量f_id为None。鉴于它适用于1个线程，我猜这不是建议的问题，而是我的代码中的其他内容，所以我将其标记为已接受。谢谢！

class ThreadPool:

    def __init__(self, count, timeout, daemonic):
        self.__busy = 0
        self.__idle = clock()
        self.__jobs = Queue()
        self.__lock = Lock()
        self.__pool = []
        self.__timeout = timeout
        for _ in range(count):
            thread = Thread(target=self.__worker)
            thread.daemon = daemonic
            thread.start()
            self.__pool.append(thread)

    def __worker(self):
        while True:
            try:
                function, args, kwargs = self.__jobs.get(True, 0.1)
            except Empty:
                with self.__lock:
                    if self.__busy:
                        continue
                    if clock() - self.__idle < self.__timeout:
                        continue
                    break
            else:
                with self.__lock:
                    self.__busy += 1
                try:
                    function(*args, **kwargs)
                except:
                    pass
                with self.__lock:
                    self.__busy -= 1
                    self.__idle = clock()

    def apply(self, function, *args, **kwargs):
        self.__pool = list(filter(Thread.is_alive, self.__pool))
        if not self.__pool:
            raise RuntimeError('ThreadPool has no running Threads')
        self.__jobs.put((function, args, kwargs))

    def join(self):
        for thread in self.__pool:
            thread.join()


def drill_folder_loop(folder_list, project_id, directory, jsession_id):
    tp = ThreadPool(25, 1, False)
    tp.apply(dfl_worker, tp, folder_list, project_id, directory, jsession_id)
    tp.join()


def dfl_worker(tp, folder_list, project_id, directory, jsession_id):
    for folder in folder_list:
        f_name = folder.keys()[0]
        f_id = folder.values()[0]
        f_dict = list_folders_files(f_id, jsession_id)
        f_list = process_folder_files(f_dict, directory, jsession_id)
        tp.apply(dfl_worker, tp, f_list, project_id, directory, jsession_id)
        recipient_process(f_dict, 'no', f_id, directory, project_id)
    log_info('One folder processed')

2 个答案:

答案 0 :(得分：1)

我可以推荐以下内容吗？

from queue import Empty, Queue
from threading import Lock, Thread
from time import perf_counter


def drill_folder_loop(folder_list, project_id, directory, jsession_id):
    while True:
        next_folder_list = []
        for folder in folder_list:
            f_name, f_id = folder.popitem()
            f_dict = list_folders_files(f_id, jsession_id)
            f_list = process_folder_files(f_dict, directory, jsession_id)
            recipient_process(f_dict, 'no', f_id, directory, project_id)
            next_folder_list.extend(f_list)
        if not next_folder_list:
            break
        folder_list = next_folder_list
    return log_info('Folder loop complete')

###############################################################################


class ThreadPool:

    def __init__(self, count, timeout, daemonic):
        self.__busy = 0
        self.__idle = perf_counter()
        self.__jobs = Queue()
        self.__lock = Lock()
        self.__pool = []
        self.__timeout = timeout
        for _ in range(count):
            thread = Thread(target=self.__worker)
            thread.daemon = daemonic
            thread.start()
            self.__pool.append(thread)

    def __worker(self):
        while True:
            try:
                function, args, kwargs = self.__jobs.get(True, 0.1)
            except Empty:
                with self.__lock:
                    if self.__busy:
                        continue
                    if perf_counter() - self.__idle < self.__timeout:
                        continue
                    break
            else:
                with self.__lock:
                    self.__busy += 1
                try:
                    function(*args, **kwargs)
                except:
                    pass
                with self.__lock:
                    self.__busy -= 1
                    self.__idle = perf_counter()

    def apply(self, function, *args, **kwargs):
        self.__pool = list(filter(Thread.is_alive, self.__pool))
        if not self.__pool:
            raise RuntimeError('ThreadPool has no running Threads')
        self.__jobs.put((function, args, kwargs))

    def join(self):
        for thread in self.__pool:
            thread.join()


def drill_folder_loop(folder_list, project_id, directory, jsession_id):
    tp = ThreadPool(25, 1, False)
    tp.apply(dfl_worker, tp, folder_list, project_id, directory, jsession_id)
    tp.join()


def dfl_worker(tp, folder_list, project_id, directory, jsession_id):
    for folder in folder_list:
        f_name, f_id = folder.popitem()
        f_dict = list_folders_files(f_id, jsession_id)
        f_list = process_folder_files(f_dict, directory, jsession_id)
        tp.apply(dfl_worker, tp, f_list, project_id, directory, jsession_id)
        recipient_process(f_dict, 'no', f_id, directory, project_id)
    log_info('One folder processed')

第一个drill_folder_loop是对您的函数的重写应该做同样的事情，但第二个版本应该使用ThreadPool类，以便您的文件夹列表可以同时处理多达25个线程。请注意，线程版本不会返回任何重要内容，并且在执行时几乎立即返回，如果最后删除了tp.join()。

答案 1 :(得分：-1)

理解你想用这段代码做什么有点棘手。

据我所知，你想让这个代码多线程化;要实现这一目标，你需要找到一个routine/taskset，它可以相互独立地执行，然后你可以把它从循环中取出并创建一个独立的函数。

def task(someparams):
    #mytasks using the required someparams

现在，您可以创建一组工作线程，并为其分配工作以运行task例程并完成工作。

这里是如何多线程task例程：

import thread

WORKER_THREAD = 10

c = 0
while c < WORKER_THREAD: 
    thread.start_new_thread( task, (someparams, ) )

while 1:
   pass

thread.start_new_thread（task，（someparams，））

相关问题
最新问题