尝试使用mpi4py在许多进程中加载pickle。这是我在大多数时间里工作得很好的东西:
for ix in range(len(frames)):
if comm.rank == 0:
frame = frames[ix]
logging.debug("{0}: doing stuff, frame {1}".format(comm.rank,frame))
W, nodes = pickle.load(open(os.path.join(args.network_dir, '{0}.pkl'.format(frame))))
freqs = np.random.normal(0., 5.*np.pi/180., size = len(W))
else:
W = None
nodes = None
freqs = None
comm.Barrier()
W = comm.bcast(W, root = 0)
nodes = comm.bcast(nodes, root = 0)
freqs = comm.bcast(freqs, root = 0)
comm.Barrier()
但它可靠地停留在广播步骤上。 freqs对于所有进程都必须相同,因此它必须由一个进程生成。我可以让每个人自己加载泡菜,如:
for ix in range(len(frames)):
frame = frames[ix]
logging.debug("{0}: doing stuff, frame {1}".format(comm.rank,frame))
W, nodes = pickle.load(open(os.path.join(args.network_dir, '{0}.pkl'.format(frame))))
if comm.rank == 0:
freqs = np.random.normal(0., 5.*np.pi/180., size = len(W))
else:
freqs = None
# broadcast and stuff
然后每个人都加载一个不同的泡菜!这不应该发生,对吧?每个进程都应该在ix = 0处进入循环,正确吗?
感谢任何帮助。