Question

我正在尝试使用Multiprocessing and Pool在Python中进行并行处理。下面将给出并行处理的代码。

 all_rects = [[[] for x in range(net_config["grid_width"])] for y in range(net_config["grid_height"])]
 for n in range(len(bbox_list)):
     for k in range(net_config["grid_height"] * net_config["grid_width"]):
         conf = conf_list[n][k,1].flatten()[0]
         if conf > 0.75:
             y = int(k / net_config["grid_width"])
             x = int(k % net_config["grid_width"])
             bbox = bbox_list[n][k]

             abs_cx = pix_per_w/2 + pix_per_w*x + int(bbox[0,0,0])
             abs_cy = pix_per_h/2 + pix_per_h*y+int(bbox[1,0,0])
             w = bbox[2,0,0]
             h = bbox[3,0,0]
             all_rects[y][x].append(Rect(abs_cx,abs_cy,w,h,conf))

len(bbox_list)每次迭代都有所不同。我想根据processes的数量设置len(bbox_list)的数量。

所以我有一个功能

def doWork(box_list, conf_list, all_rects, w, h, n, pix_per_w):
    for k in range(h * w):
        conf = conf_list[n][k,1].flatten()[0]
        if conf > 0.75:
           y = int(k / w)
           x = int(k % w)
           bbox = bbox_list[n][k]
           abs_cx = pix_per_w/2 + pix_per_w*x + int(bbox[0,0,0])
           abs_cy = pix_per_h/2 + pix_per_h*y+int(bbox[1,0,0])
           w = bbox[2,0,0]
           h = bbox[3,0,0]
           all_rects[y][x].append(Rect(abs_cx,abs_cy,w,h,conf))

然后从main，我做并行处理

all_rects = [[[] for x in range(net_config["grid_width"])] for y in range(net_config["grid_height"])]
pool = multiprocessing.Pool(len(bbox_list))
pool.map(doWork, (bbox_list, conf_list, all_rects, net_config["grid_width"], net_config["grid_height"], [len(bbox_list)], pix_per_w))

我有一个错误（TypeError：doWork（）需要7个参数（给定1个）），另一个问题是all_rects。 all_rects将是三维数组，all_rects的数据同步如何，多处理是否安全？

编辑：

def doWork(worker_args, b_range):
    box_list = worker_args[0]
    conf_list = worker_args[1]
    all_rects = worker_args[2]
    w = worker_args[3]
    h = worker_args[4]    
    pix_per_w = worker_args[5]
    n = b_range
    for k in range(h * w):
        conf = conf_list[n][k,1].flatten()[0]
        if conf > 0.75:
           y = int(k / w)
           x = int(k % w)
           bbox = bbox_list[n][k]
           abs_cx = pix_per_w/2 + pix_per_w*x + int(bbox[0,0,0])
           abs_cy = pix_per_h/2 + pix_per_h*y+int(bbox[1,0,0])
           w = bbox[2,0,0]
           h = bbox[3,0,0]
           all_rects[y][x].append(Rect(abs_cx,abs_cy,w,h,conf))

然后从main，

         all_rects = [[[] for x in range(net_config["grid_width"])] for y in range(net_config["grid_height"])]
         pool = multiprocessing.Pool(len(bbox_list))
         box_ranges = range(len(bbox_list))
         worker_args = [bbox_list, conf_list, all_rects, net_config["grid_width"], net_config["grid_height"], pix_per_w]
         pool.map(doWork, worker_args, l) for l in box_ranges

我试图通过元组发送，并且我将错误视为无效语法

pool.map(doWork, worker_args, l) for l in box_ranges

由于

Answer 1

您可以编写worker函数来接受元组作为参数并展开函数中的元组，例如：

def doWork(worker_args):
  bbox_list, conf_list, all_rects, w, h, n, pix_per_w = worker_args

关于all_rects的关注，据我所知，追加是线程安全的。见Are lists thread-safe

编辑：

Pool.map与常规map功能非常相似。它接收两个参数，第一个是你的worker函数的名字，第二个应该是传递给你的worker的参数的迭代器。所以你的第二个参数应该是一个元组列表。每个元组将作为doWork函数的参数传递给一个进程：

pool.map(doWork, [(bbox_list, conf_list, all_rects, w, h, n, pix_per_w) for _ in xrange(len(bbox_list))])

多处理，在Python中的进程之间传递多个参数和数据同步

1 个答案: