Python中的并行/多处理(嵌套)循环并存储结果?

时间:2020-07-09 21:09:00

标签: python loops indexing multiprocessing

我有两个问题,我认为这两个问题的相关性足以使其成为单个问题的一部分。但是,如果它们不是,我可以将它们作为单独的问题提出。请告诉我。我也提前道歉,因为我有做错很错的印象,但我不知道这是什么。

到目前为止,我在Python中运行以下代码(如果有区别,请使用Jupyter笔记本):

首先,我初始化一个很长的(多级?)列表:

object = [[[[[[[[None for i in range(2)]  
               for j in range(2)] 
              for k in range(2)] 
             for l in range (2)] 
            for m in range (2)] 
           for n in range (2)]
          for o in range (2)]
         for p in range (2)]

接下来,我运行了一堆循环,一个循环在另一个循环中,并运行一个函数(取决于我在循环中使用的所有索引),将结果分配给我在上面创建的一个位置:

for i in range(2):
    for j in range(2):
        for k in range(2):
            for l in range(2):
                for m in range(2):
                    for n in range(2):
                        for o in range(2):
                            for p in range(2):
                                object[i][j][k][l][m][n][o][p] = function(i,j,k,l,m,n,o,p) 

以下是两个相关的问题:

  1. 函数在每次迭代中返回的对象是彼此完全独立的(例如,我可以在一台计算机上运行循环的每次迭代,例如稍后再收集它们)。因此,我认为此循环将是并行/多处理中要解决的理想候选对象。如果是这样,我该怎么做?我发现了一些并行运行嵌套循环的提法,但是我不明白它如何适用于我的情况。全面披露:我从未在Python中并行运行任何内容。

  2. 此列表(是否引用此令人不快的object[i][j][k][l][m][n][o][p])是否正确地保留了结果(以以后可以找到的方式)?还是可以建议一个更好的方法?如果相关,则函数返回的对象具有诸如pandas数据帧,数字和字符串之类的属性。

1 个答案:

答案 0 :(得分:0)

对于第一个问题,建议您在此处查看顶部答案,以了解如何并行化下面概述的for循环(回答问题2):

How do I parallelize a simple Python loop?

第二个问题:

#dummy function for illustrative purposes
def function(a,b,c,d,e,f,g,h):
  return a+b+c+d+e+f+g+h

如果函数的输出是可哈希的,我将创建一个字典:

#This is your 'objects'
O={}

for y in range(2**8):
    #this generates all the permutations you were after I believe
    s=format(y, '#010b')[2:]
    #print(s) #uncomment to see what it does
    #This is slightly messy, in that you have to split up your integer into its components, but I've seen worse.
    O[y]=function(int(s[0]),int(s[1]),int(s[2]),int(s[3]),int(s[4]),int(s[5]),int(s[6]),int(s[7]))

#Now, if you wanted to print the output of f(1,1,1,1,1,1,1,1):
g='11111111'
print(O[int(g,2)]) #uncomment to see what it does 

#print(O) #uncomment to see what it does 

如果输出不可散列,则保留为列表:

O=[] 

for y in range(2**8):
    #this generates all the permutations you were after I believe
    s=format(y, '#010b')[2:]
    #print(s) #uncomment to see what it does
    #This is slightly messy, in that you have to split up your integer into its components, but I've seen worse.
    O.append(function(int(s[0]),int(s[1]),int(s[2]),int(s[3]),int(s[4]),int(s[5]),int(s[6]),int(s[7])))

#Now, if you wanted to print the output of f(1,1,1,1,1,1,1,1):
g='11111111'
#print(O[int(g,2)]) #uncomment to see what it does 

#print(O) #uncomment to see what it does