在进程内部运行池

时间:2019-07-27 21:28:58

标签: python rabbitmq multiprocessing

我是Python多重处理的新手。我有一些班级,每个班级代表一些要完成的任务。下面是一小段类结构的片段

class TaskA():
    parallel_limit = 2       # Max number of process in Pool
    time_limit = 1           # Denotes time taken to perform some operation
    queue_name = 'read_here' # RabbitMQ queue to provide data for consumption

    # Starts a pool of processes listening on queue 
    def start_process_pool(self):
        pool = multiprocessing.Pool(processes=self.parallel_limit)
        for i in range(0, self.parallel_limit):
            pool.apply_async(self.consume)

        try:
            while True:
                continue
        except KeyboardInterrupt:
            print(' [*] Exiting...')
            pool.terminate()
            pool.join()

    # Reads from queue and performs some operation
    def consume(self):
        connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
        channel = connection.channel()

        channel.queue_declare(queue=self.queue_name, durable=True)
        channel.basic_consume(queue=self.queue_name, on_message_callback=self.perform_operation, auto_ack=True)

        print(' [*] Waiting for messages. To exit press CTRL+C')
        channel.start_consuming()

    # Performing some operation
    def perform_operation(self, ch, method, properties, body):
        print(" [x] %r received %r" % (multiprocessing.current_process(), body,))
        time.sleep(self.time_limit)
        ch.basic_ack(delivery_tag=method.delivery_tag)

我能够在Pool中运行此任务。当我有多个任务并且需要在池中运行每个任务时,就会出现问题。我正在考虑为每个任务创建一个进程,然后从中开始一个进程池,但是无法弄清楚该如何做。任何贡献都会有很大帮助。

0 个答案:

没有答案