Question

这个脚本用于将Queue作为一个全局对象，可以在线程被实例化的地方和线程函数本身中访问，但是为了使事情变得更清晰，我在更多的“可接受的”中重构了一些东西。方式而是决定将Queue的get（）和task_done（）方法传递给其实例化的线程函数以摆脱全局;但是，我注意到join（）现在无限期地挂起，而在Queue全局之前，它总是运行完成并适当终止。

有关完整上下文，请参阅以下内容：http://dpaste.com/0PD6SFX

但是，我认为这是唯一相关的代码片段; init本质上是我的主要方法，它所属的类拥有Queue，而run是运行Transcode类的方法（其余的Transcode类详细信息并不相关，我觉得）：

def __init__(self, kargs):
        self.params = kargs

        # will store generated Transcode objects
        self.transcodes = []

        # Queue to thread ffmpeg processes with
        self.q = Queue.Queue()

        # generate transcodes with self.params for all rasters x formats
        self.generate_transcodes()

        # give the Queue the right number of task threads to make
        for i in range(self.get_num_transcodes()):
            self.q.put(i)

        for transcode in self.transcodes:
            transcode.print_commands()

        # testing code to be sure command strings are generating appropriately
        for transcode in self.transcodes:
            self.q.put(transcode)

        # kick off all transcodes by creating a new daemon (terminating on program close) thread;
        # the thread is given each Transcode's run() method, which is dynamically created within
        # the Transcode constructor given its command strings
        for transcode in self.transcodes:
            t = threading.Thread(target=transcode.run, args=(self.q.get, self.q.task_done))
            t.daemon = True
            t.start()

        print("Transcoding in progress...")

        # go through each transcode and print which process is currently underway, then sleep
        # 1 = first pass, 2 = second pass, 3 = complete
        while True:
            still_running = False
            for transcode in self.transcodes:
                if not transcode.complete:
                    still_running = True
                    print('Transcode %s still running!' % transcode.filename)
                if transcode.current_proc in range(3):
                    print(os.path.basename(transcode.filename) + ': pass %s' % transcode.current_proc)
                else:
                    print(os.path.basename(transcode.filename) + ': complete!')
                print(transcode.complete)
            if not still_running:
                break
            time.sleep(2)
            print('poll')

        # block until all items in queue are gotten and processed
        print('About to join...')
        self.q.join()

        print('End of transcode batch!')

'''
    Executes sequentially the command strings given to this Transcode via subprocess; it will
    first execute one, then the next, as the second command relies on the first being completed.
    It will take in a get() and done() method that are Queue methods and call them at the right
    moments to signify to an external Queue object that the worker thread needed for this instance
    is ready to start and finish.

    @param  get     A Queue get() function to be called at run()'s start.
    @param  done    A Queue done() function to be called at run()'s end.

    @return none
    '''
    def run(self, get, done):
        # get a thread from the queue
        get()

        # convert command lists to command strings
        for i in range(len(self.commands)):
            self.commands[i] = ' '.join(self.commands[i])

        # show that we're working with our first command string
        self.current_proc = 1

        # assign our current proc the first command string subprocess
        self.proc = Popen(self.commands[0], stdout=PIPE, stderr=PIPE, shell=True)
        # execute process until complete
        self.proc.communicate()

        print('Transcode %s first pass complete' % self.identifier)

        # run second command string if exists
        if len(self.commands) > 1:

            # show that we're working with second command string
            self.current_proc = 2

            # spawn second process and parse output line by line as before
            self.proc = Popen(self.commands[1], stdout=PIPE, stderr=PIPE, shell=True)
            # execute process until complete
            self.proc.communicate()

            print('Transcode %s second pass complete' % self.identifier)

        # delete log files when done
        if self.logfile and os.path.exists(self.logfile + '-0.log'):
            os.remove(self.logfile + '-0.log')
        if self.logfile and os.path.exists(self.logfile + '-0.log.mbtree'):
            os.remove(self.logfile + '-0.log.mbtree')
        if self.logfile and os.path.exists(self.logfile + '-0.log.temp'):
            os.remove(self.logfile + '-0.log.temp')
        if self.logfile and os.path.exists(self.logfile + '-0.log.mbtree.temp'):
            os.remove(self.logfile + '-0.log.mbtree.temp')

        self.complete = True
        print('Transcode %s complete' % self.identifier)

        # assign value of 3 to signify completed task
        self.current_proc = 3

        # finish up with Queue task
        done()

Answer 1

解决方案非常愚蠢：对于每个发生的转码，条目被添加到队列中两次。

    # give the Queue the right number of task threads to make
    for i in range(self.get_num_transcodes()):
        self.q.put(i)

    # code omitted...

    # testing code to be sure command strings are generating appropriately
    for transcode in self.transcodes:
        self.q.put(transcode)

Twas是重构的遗迹。因此，Queue认为它需要的工作线程数量是实际工作量的两倍，并且正在等待处理剩余的条目。

当get（）和task_done（）作为回调传入时，线程程序在Queue.join（）上挂起

1 个答案: