我想使用带有迭代器的多处理池，以便在一个线程中执行一个函数，将迭代器分解为N个元素，直到迭代器完成。
import arcpy
from multiprocessing import Pool
def insert(rows):
with arcpy.da.InsertCursor("c:\temp2.gdb\test" fields=["*"]) as i_cursor:
#i_cursor is an iterator
for row in rows:
i_cursor.insertRow(row)
input_rows = []
count = 0
pool = Pool(4)
with arcpy.da.SearchCursor("c:\temp.gdb\test", fields=["*"]) as s_cursor:
#s_cursor is an iterator
for row in s_cursor:
if (count < 100):
input_rows.append(row)
count += 1
else:
#send 100 rows to the insert function in a new thread
pool.apply_async(insert, input_rows)
#reset count and input_rows
count = 1
input_rows = [row]
pool.join()
pool.close()
我的问题是，这个脚本是正确的方法吗？还有更好的方法吗？
该脚本可能有问题，因为我在pool.join()
Traceback (most recent call last):
File "G:\Maxime\truncate_append_pool.py", line 50, in <module>
pool.join()
File "C:\App\Python27\ArcGIS10.3\lib\multiprocessing\pool.py", line 460, in join
assert self._state in (CLOSE, TERMINATE)
AssertionError
答案 0 :(得分：6)
如果我必须猜测代码的主要错误，我会说你将input_rows传递给你的流程函数insert() - multiprocessing.Pool.apply_async()的工作方式是解压缩传递给它的参数，因此你的insert()函数实际上转发100个参数而不是一个带有100元素列表的参数。在您的过程功能甚至有机会启动之前，这会立即导致错误。如果你改变对pool.apply_async(insert, [input_rows])的调用，它可能会开始工作......你也会破坏迭代器的目的，你可能只是将整个输入迭代器转换成一个列表并将100的切片提供给{ {3}}并完成它。
但你问是否有“更好”的方法。虽然“更好”是一个相对类别，但在理想的世界中，multiprocessing.Pool.map()附带了一个方便的multiprocessing.Pool（和imap()）方法，旨在使用迭代并将它们分散到选定的池中懒惰的方式（因此在处理之前没有遍历整个迭代器），所以你需要构建的是你的迭代器切片并将其传递给它进行处理，即：
import arcpy
import itertools
import multiprocessing
# a utility function to get us a slice of an iterator, as an iterator
# when working with iterators maximum lazyness is preferred
def iterator_slice(iterator, length):
iterator = iter(iterator)
while True:
res = tuple(itertools.islice(iterator, length))
if not res:
break
yield res
def insert(rows):
with arcpy.da.InsertCursor("c:\temp2.gdb\test" fields=["*"]) as i_cursor:
for row in rows:
i_cursor.insertRow(row)
if __name__ == "__main__": # guard for multi-platform use
with arcpy.da.SearchCursor("c:\temp.gdb\test", fields=["*"]) as s_cursor:
pool = multiprocessing.Pool(processes=4) # lets use 4 workers
for result in pool.imap_unordered(insert, iterator_slice(s_cursor, 100)):
pass # do whatever you want with your result (return from your process function)
pool.close() # all done, close cleanly
（顺便说一句。您的代码不会为您提供不是100的倍数的所有s_cursor尺寸的最后一个切片。
但是......如果它真的像宣传的那样工作会很棒。虽然这些年来已经修复了很多，但是imap_unordered()在生成自己的迭代器时仍会使用迭代器的大样本（远远大于实际池进程的数量），因此，如果这是一个问题，你将不得不自己沮丧，并且你走在正确的轨道上 - apply_async()是你要控制如何喂养你的游泳池的方式，你只需要确保您正确地提供您的游泳池：
if __name__ == "__main__":
with arcpy.da.SearchCursor("c:\temp.gdb\test", fields=["*"]) as s_cursor:
pool = multiprocessing.Pool(processes=4) # lets use 4 workers
cursor_iterator = iterator_slice(s_cursor, 100) # slicer from above, for convinience
queue = [] # a queue for our current worker async results, a deque would be faster
while cursor_iterator or queue: # while we have anything to do...
try:
# add our next slice to the pool:
queue.append(pool.apply_async(insert, [next(cursor_iterator)]))
except (StopIteration, TypeError): # no more data, clear out the slice iterator
cursor_iterator = None
# wait for a free worker or until all remaining finish
while queue and (len(queue) >= pool._processes or not cursor_iterator):
process = queue.pop(0) # grab a process response from the top
process.wait(0.1) # let it breathe a little, 100ms should be enough
if not process.ready(): # a sub-process has not finished execution
queue.append(process) # add it back to the queue
else:
# you can use process.get() to get the result if needed
pass
pool.close()
现在只有在需要接下来的100个结果时才会调用s_cursor迭代器（当你的insert()进程函数完全退出时）。
更新 - 如果需要捕获的结果，以前发布的代码在关闭队列时有一个错误，这个应该很好地完成工作。我们可以使用一些模拟函数轻松测试它：
import random
import time
# just an example generator to prove lazy access by printing when it generates
def get_counter(limit=100):
for i in range(limit):
if not i % 3: # print every third generation to reduce verbosity
print("Generated: {}".format(i))
yield i
# our process function, just prints what's passed to it and waits for 1-6 seconds
def test_process(values):
time_to_wait = 1 + random.random() * 5
print("Processing: {}, waiting: {:0.2f} seconds".format(values, time_to_wait))
time.sleep(time_to_wait)
print("Processed: {}".format(values))
现在我们可以将它们交织在一起：
if __name__ == "__main__":
pool = multiprocessing.Pool(processes=2) # lets use just 2 workers
count = get_counter(30) # get our counter iterator set to iterate from 0-29
count_iterator = iterator_slice(count, 7) # we'll process them in chunks of 7
queue = [] # a queue for our current worker async results, a deque would be faster
while count_iterator or queue:
try:
# add our next slice to the pool:
queue.append(pool.apply_async(test_process, [next(count_iterator)]))
except (StopIteration, TypeError): # no more data, clear out the slice iterator
count_iterator = None
# wait for a free worker or until all remaining workers finish
while queue and (len(queue) >= pool._processes or not count_iterator):
process = queue.pop(0) # grab a process response from the top
process.wait(0.1) # let it breathe a little, 100ms should be enough
if not process.ready(): # a sub-process has not finished execution
queue.append(process) # add it back to the queue
else:
# you can use process.get() to get the result if needed
pass
pool.close()
结果是（当然，它会因系统而异）：
Generated: 0
Generated: 3
Generated: 6
Generated: 9
Generated: 12
Processing: (0, 1, 2, 3, 4, 5, 6), waiting: 3.32 seconds
Processing: (7, 8, 9, 10, 11, 12, 13), waiting: 2.37 seconds
Processed: (7, 8, 9, 10, 11, 12, 13)
Generated: 15
Generated: 18
Processing: (14, 15, 16, 17, 18, 19, 20), waiting: 1.85 seconds
Processed: (0, 1, 2, 3, 4, 5, 6)
Generated: 21
Generated: 24
Generated: 27
Processing: (21, 22, 23, 24, 25, 26, 27), waiting: 2.55 seconds
Processed: (14, 15, 16, 17, 18, 19, 20)
Processing: (28, 29), waiting: 3.14 seconds
Processed: (21, 22, 23, 24, 25, 26, 27)
Processed: (28, 29)
证明我们的生成器/迭代器仅在池中有空闲插槽时才用于收集数据，以确保最小的内存使用量（和/或如果迭代器最终执行此操作而导致I / O冲击）。你不会比这更精简。你可以获得的唯一额外的，虽然是微不足道的加速是减少等待时间（但是你的主要进程将占用更多的资源）并增加被锁定的允许queue大小（以内存为代价）上面代码中的进程数 - 如果使用while queue and (len(queue) >= pool._processes + 3 or not count_iterator):，它将加载3个迭代器切片，确保在进程结束并且池中的插槽释放时的情况下延迟较短。