在多处理中重写写入任务(pool.map)

时间:2016-10-20 14:26:20

标签: python python-2.7 multiprocessing file-writing

在运行以下代码时使用多处理时,我遇到了重叠的写入问题。

def spectrum(i):
    for j in range (num_x):
        coordinate = data[:,j,i]
        filtered = filter(lambda a: a != 0, coordinate)
        occupancy = float(len(filtered))/framespfile
        if filtered == [] or filtered[0] > 500:
            output = str([j, i]) + "\n" + str(filtered) + "\n"
            badpixelfile.write(output)
        else :
            output = str([j, i]) + "\n" + str(filtered) + "\n"
            coordinatefile.write(output)


pool2 = multiprocessing.Pool(multiprocessing.cpu_count())
pool2.map(spectrum, range(num_y))
pool2.close()
pool2.join()

它应该写出如下结果:

 [14,0]
 [50, 51, 84]
 [0, 314]
 [60, 74, 12, 202, 129]

但有时进程重叠并且文件看起来像(偶尔会发生这种情况,但会导致分析问题)

[149, 27]
[27, 34, 26, 25, 19, 45, 32, 36, 46, 29, 25, 25, 40, 62, 24, 31, 23, 46, 33, 35, 60, 33, 8, 24, 49, 29, 29, 42, 8, 22, 31, 28, 25, 25, 56, 32, 31, 27, 11, 20, 29, 23, 51, 28, 31, 29, 28, 30, 23, 16, 34, 36, 25, 17, 25, 19, 19, 51, 27, 37, 9, 32, 26, 28, 27, 3, 44, 4, 38, 20, 34, 28, 22, 26, 26, 19, 21, 25, 25, 48, 24, 29, 22, 20, 23, 29, 15, 32, 42, 3, 23, 26, 34, 28, 26, 39, 17, [0, 123]
[20, 43, 33, 34, 18, 44, 15, 22, 33, 20, 45, 30, 21, 33, 32, 43, 30, 8, 37, 54, 9, 46, 33, 16, 27, 29, 31, 47, 26, 38, 40, 29, 34, 38, 17, 33, 47, 28, 24, 33, 40, 47, 16, 32, 33, 21, 49, 34, 26, 21, 47, 46, 49, 13, 62, 62, 31, 41, 14, 65, 36, 49, 27, 38, 44, 54, 55, 64, 32, 50, 28, 34, 41, 49, 33, 40, 28, 32, 31, 56, 16, 35, 37, 50, 33, 41, 38, 26, 41, 26, 28, 25, 37, 27, 20, 47, 31, 35, 28, 43, 48, 37, 31, 24, 34, 36, 41, 19, 41, 41, 3, 36]
[1, 123]

因此它没有完成[149,27]的过程,并且在没有关闭[149,27]过程的情况下已经以[0,123]开始。

0 个答案:

没有答案