Question

如何读取和更新Python中多个工作人员之间共享的变量？

例如，我在Python中使用多个进程扫描文件列表，并希望检查父目录是否已被扫描。

def readFile(filename):
  """ Add the parent folder to the database and process the file
  """

  path_parts = os.path.split(filename)
  dirname = os.path.basename(path_parts[0])
  if dirname not in shared_variable:
    # Insert into the database


   #Other file functions


def main():
  """ Walk through files and pass each file to readFile()
  """
  queue = multiprocessing.Queue()
  pool = multiprocessing.Pool(None, init, [queue])

  for dirpath, dirnames, filenames in os.walk(PATH):

    full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
                           filenames)
    pool.map(readFile, full_path_fnames)

Answer 1

您可以使用multiprocessing.Manager来帮助解决此问题。它允许您创建可在进程之间共享的列表：

from functools import partial
import multiprocessing

def readFile(shared_variable, filename):
  """ Add the parent folder to the database and process the file
  """

  path_parts = os.path.split(filename)
  dirname = os.path.basename(path_parts[0])
  if dirname not in shared_variable:
    # Insert into the database


   #Other file functions


def main():
  """ Walk through files and pass each file to readFile()
  """
  manager = multiprocessing.Manager()
  shared_variable = manager.list()
  queue = multiprocessing.Queue()
  pool = multiprocessing.Pool(None, init, [queue])

  func = partial(readFile, shared_variable)
  for dirpath, dirnames, filenames in os.walk(PATH):

    full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
                           filenames)
    pool.map(func, full_path_fnames)

partial仅用于通过shared_variable更轻松地将readFile传递给full_path_fnames的每次调用以及map的每个成员。

Answer 2

看看https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes。您可以使用Value或Array使用共享内存在两个或多个线程之间共享数据。

使用Python多处理在工作者之间共享变量

2 个答案: