Question

我正在学习多进程，直到在使用队列时遇到此问题之前，我没有遇到任何问题。本质上，队列已满，但随后似乎出了点问题，它崩溃了。

我在Windows 10上运行python 3.6.8。当我不使用队列时，多处理似乎可以正常工作（我在下面构建了与下面类似的代码片段，而无需队列学习）。

import glob, multiprocessing, os

def appendFilesThreaded(inputDirectory, outputDirectory, inputFileType=".txt", outputFileName="appended_files.txt"):
    files = glob.glob(inputDirectory+'*'+inputFileType)
    fileQueue = multiprocessing.Queue()
    for file in files:
        fileQueue.put(file)
    threadsToUse = max(1, multiprocessing.cpu_count()-1)
    print("Using " + str(threadsToUse) + " worker threads.")
    processes = []
    for i in range(threadsToUse):
        p = multiprocessing.Process(target=appendFilesWorker, args=(fileQueue,outputDirectory+"temp-" + str(i) + outputFileName))
        processes.append(p)
        p.start()
    for process in processes:
        process.join()
    with open(outputDirectory + outputFileName, 'w') as outputFile:
        for i in range(threadsToUse):
            with open(outputDirectory+"temp-" + str(i) + outputFileName) as fileToAppend:
                outputFile.write(fileToAppend.read())
            os.remove(outputDirectory+"temp-" + str(i) + outputFileName)
    print('Done')
def appendFilesWorker(fileQueue, outputFileNamePath):
    with open(outputFileNamePath, 'w') as outputFile:
        while not fileQueue.empty:
            with open(fileQueue.get()) as fileToAppend:
                outputFile.write(fileToAppend.read())

if __name__ == '__main__':
    appendFilesThreaded(inputDir,outputDir)

我希望它可以成功附加文件，但是会崩溃。结果为BrokenPipeError: [WinError 232] The pipe is being closed

Answer 1

发现了问题：调用queue.empty是错误的。您需要括号（例如queue.empty()）

我会留下尴尬的错误，以防其他人：）

在Python中使用多重处理时如何解决BrokenPipeError

1 个答案: