在消耗队列和管道中的所有项目之后加入流程

时间:2019-06-11 22:33:26

标签: python multithreading multiprocessing python-multiprocessing python-multithreading

我有一个python代码,通过它我有两个由主程序创建的进程。其中一个(发送器)生成一堆数据并将其发送到另一个进程(接收器)以对其进行处理。在传输结束时,发送器通过在Pipe()上放置一个信号来通知传输结束,接收器等待直到接收到该信号,然后尝试消耗队列中的所有项目。但是,问题在于发送器进程最终不会加入主进程。我进行了很多搜索,但找不到解决方案。关闭发送器中的Pipe()或Queue()应该有问题。


from multiprocessing import Process, Pipe, Lock, Queue
import multiprocessing as mp
from time import sleep
from threading import Thread
import threading
import time
from numpy import random as rnd

def f1(tx, comm):

    for ij in range(100):
        tx.put(rnd.randn(1000,))

    comm.send(1)
    comm.close()
    print('\nThe transmitter is done\n')


def g(rx: Queue, comm, l):
    msg_lock = threading.Lock()

    msg = 0
    Rtimes = []
    def readingQ(msg_lock):

        while True:

            msg_lock.acquire()
            flg = msg
            msg_lock.release()
            if flg == 1:
                l.acquire()
                print('Waiting to consume all items')
                l.release()
                while rx.empty() is not True:
                    tim1 = time.time()
                    rx.get()
                    Rtimes.append(time.time()-tim1)

                print('The items are consumed ...')
                rx.close()
                rx.join_thread()
                l.acquire()
                print('The background thread in queue is joined! ')
                l.release()
                break

            tim1 = time.time()
            print('rx Q size:', rx.qsize()) #, 'retrieved value: ',rx.get())
            rx.get()
            Rtimes.append(time.time()-tim1)

        sleep(.001)
        l.acquire()
        print('Time taking to get the items: \n', Rtimes)
        l.release()

    def getmsg(msg_lock):
        nonlocal msg
        msg_lock.acquire()
        msg = comm.recv()
        msg_lock.release()
        print('message is received: ', msg)
        comm.close()

    t1 = Thread(target=getmsg, args=(msg_lock,))
    t2 = Thread(target=readingQ, args=(msg_lock,))

    t2.start()
    t1.start()

    t1.join()
    print('The message box thread is joined')
    t2.join()
    print('The consumer thread is joined')

if __name__ == '__main__':
    mp.set_start_method('spawn')
    printLock = Lock()

    p = []

    comm1, comm2 = Pipe(False)
    print('comm1 fileno {} and comm2 fileno {}'.format(comm1.fileno(),comm2.fileno()))
    qs = Queue()
    p.append(Process(target=g, args=(qs,comm1, printLock)))
    p.append(Process(target=f1, args=(qs,comm2)))

    for o in p:
        o.start()

    printLock.acquire()
    print('check if the second child process is alive or not: ', p[1].is_alive())
    print('active children: ', mp.active_children())
    print('Number of CPUs I have: ', mp.cpu_count())
    printLock.release()

    p[0].join()
    print('The receiver successfully joined')

    p[1].join()
    print('The transmitter successfully joined')



真正的代码应该加入发送器。

0 个答案:

没有答案