我有一个程序,可以将大型数据集的处理分为台球并行处理,如下所示:
import billiard
from billiard import Queue
def process_cursor(skip_n, limit_n, out_q):
# ... lines of code ...
# potentially long method call that should return after max. 5 seconds
result = possible_long_running_function(arguments)
# ... more lines of code ...
out_q.put(function_result)
batch_size = round(collection_size / n_cores + 0.5)
skips = range(0, n_cores * batch_size, batch_size)
out_q = Queue()
processes = [billiard.Process(
target=process_cursor, args=(skip_n, batch_size, out_q)) for skip_n in skips]
for process in processes:
process.start()
results = []
for i in range(n_cores):
results.append(out_q.get())
for process in processes:
process.join()
作为每个线程处理的一部分,一行代码执行的功能可能要花费很长时间才能执行。
result = possible_long_running_function(input)
如果函数花费的时间超过给定的时间,我怎么抛出Exception
。该进程可以创建一个子进程并在花费太长时间的情况下将其杀死吗?