如何控制在Python中打开的进程数

时间:2019-06-30 06:17:24

标签: python python-3.x multiprocessing python-multiprocessing multiprocessing-manager

我的主要目标是从父进程中打开30个子进程,然后从那30个子进程中的每个进程中打开未知数量的新进程。我要给那些新的子进程中的一些位置数据打电话给redis,但我不确定要打电话多少次,这应该是100次或超过1000次。当我打电话超过1000次时,我正在跨页限制,错误是:

OSError: [Errno 24] Too many open files

我不想在生产服务器上手动增加页面限制。我想限制流程的创建,因此它决不能打开超过1000个连接。

这是我的模板代码:

import multiprocessing
import time
from multiprocessing.dummy import Pool
from random import randint


class MultiProcessing():

    def second_calculation(self, index1, index2):
        random = randint(1, 10)
        time.sleep(random)
        print("Slept for: {} seconds".format(random))
        print("Call done: index: {} | index2: {}".format(index1, index2))

    def calculation(self, index):
        child_process = list()
        random = randint(1, 5)
        time.sleep(random)
        print("Slept for : {} seconds".format(random))

        counter = 0
        for i in range(0, 1500):
            counter += 1
            new_child_process = multiprocessing.Process(target=self.second_calculation, args=(index, counter))
            child_process.append(new_child_process)
            new_child_process.start()

        for process in child_process:
            process.join()

        print("Request done: {}".format(index))


if __name__ == '__main__':
    index = 0
    parent_process = list()
    m = MultiProcessing()
    for i in range(0, 30):
        index += 1
        print("Index: {}".format(index))
        new_process = multiprocessing.Process(target=m.calculation, args=(index,))
        parent_process.append(new_process)
        new_process.start()

    for process in parent_process:
        process.join()

谢谢。

0 个答案:

没有答案