跟踪joblib.Parallel执行的进度

Question

是否有一种简单的方法可以跟踪joblib.Parallel执行的整体进度？

我有一个由数千个作业组成的长时间执行，我想跟踪并记录在数据库中。但是，要做到这一点，每当Parallel完成一项任务时，我都需要它来执行回调，报告剩余的剩余作业数。

我之前使用Python的stdlib multiprocessing.Pool完成了类似的任务，启动了一个记录Pool的工作列表中待处理作业数量的线程。

查看代码，Parallel继承了Pool，所以我认为我可以使用相同的技巧，但它似乎没有使用这些列表，我还没有能够弄清楚如何否则＆＃34;阅读＆＃34;它的内部状态是其他任何方式。

Answer 1

您链接到Parallel具有可选进度表的状态的文档。它是使用callback提供的multiprocessing.Pool.apply_async关键字参数实现的：

# This is inside a dispatch function
self._lock.acquire()
job = self._pool.apply_async(SafeFunction(func), args,
            kwargs, callback=CallBack(self.n_dispatched, self))
self._jobs.append(job)
self.n_dispatched += 1

...

class CallBack(object):
    """ Callback used by parallel: it is used for progress reporting, and
        to add data to be processed
    """
    def __init__(self, index, parallel):
        self.parallel = parallel
        self.index = index

    def __call__(self, out):
        self.parallel.print_progress(self.index)
        if self.parallel._original_iterable:
            self.parallel.dispatch_next()

这里是print_progress：

def print_progress(self, index):
    elapsed_time = time.time() - self._start_time

    # This is heuristic code to print only 'verbose' times a messages
    # The challenge is that we may not know the queue length
    if self._original_iterable:
        if _verbosity_filter(index, self.verbose):
            return
        self._print('Done %3i jobs       | elapsed: %s',
                    (index + 1,
                     short_format_time(elapsed_time),
                    ))
    else:
        # We are finished dispatching
        queue_length = self.n_dispatched
        # We always display the first loop
        if not index == 0:
            # Display depending on the number of remaining items
            # A message as soon as we finish dispatching, cursor is 0
            cursor = (queue_length - index + 1
                      - self._pre_dispatch_amount)
            frequency = (queue_length // self.verbose) + 1
            is_last_item = (index + 1 == queue_length)
            if (is_last_item or cursor % frequency):
                return
        remaining_time = (elapsed_time / (index + 1) *
                    (self.n_dispatched - index - 1.))
        self._print('Done %3i out of %3i | elapsed: %s remaining: %s',
                    (index + 1,
                     queue_length,
                     short_format_time(elapsed_time),
                     short_format_time(remaining_time),
                    ))

他们实现这一点的方式有点奇怪，说实话 - 它似乎假设任务总是按照它们启动的顺序完成。转到index的{​​{1}}变量只是作业实际启动时的print_progress变量。所以推出的第一份工作总是以self.n_dispatched为0完成，即使说第三份工作先完成。这也意味着他们实际上并没有跟踪已完成的作业的数量。因此，您无需监控实例变量。

我认为你最好的办法就是制作自己的CallBack课程，以及猴子补丁并行：

index

输出：

from math import sqrt
from collections import defaultdict
from joblib import Parallel, delayed

class CallBack(object):
    completed = defaultdict(int)

    def __init__(self, index, parallel):
        self.index = index
        self.parallel = parallel

    def __call__(self, index):
        CallBack.completed[self.parallel] += 1
        print("done with {}".format(CallBack.completed[self.parallel]))
        if self.parallel._original_iterable:
            self.parallel.dispatch_next()

import joblib.parallel
joblib.parallel.CallBack = CallBack

if __name__ == "__main__":
    print(Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10)))

这样，只要作业完成，就会调用您的回调，而不是默认的回调。

Answer 2

为什么不能简单地使用tqdm？以下为我工作

from joblib import Parallel, delayed
from datetime import datetime
from tqdm import tqdm

def myfun(x):
    return x**2

results = Parallel(n_jobs=8)(delayed(myfun)(i) for i in tqdm(range(1000))
100%|██████████| 1000/1000 [00:00<00:00, 10563.37it/s]

Answer 3

扩展dano对最新版本的joblib库的回答。内部实施有一些变化。

from joblib import Parallel, delayed
from collections import defaultdict

# patch joblib progress callback
class BatchCompletionCallBack(object):
  completed = defaultdict(int)

  def __init__(self, time, index, parallel):
    self.index = index
    self.parallel = parallel

  def __call__(self, index):
    BatchCompletionCallBack.completed[self.parallel] += 1
    print("done with {}".format(BatchCompletionCallBack.completed[self.parallel]))
    if self.parallel._original_iterator is not None:
      self.parallel.dispatch_next()

import joblib.parallel
joblib.parallel.BatchCompletionCallBack = BatchCompletionCallBack

Answer 4

以下是您的问题的另一个答案，语法如下：

aprun = ParallelExecutor(n_jobs=5)

a1 = aprun(total=25)(delayed(func)(i ** 2 + j) for i in range(5) for j in range(5))
a2 = aprun(total=16)(delayed(func)(i ** 2 + j) for i in range(4) for j in range(4))
a2 = aprun(bar='txt')(delayed(func)(i ** 2 + j) for i in range(4) for j in range(4))
a2 = aprun(bar=None)(delayed(func)(i ** 2 + j) for i in range(4) for j in range(4))

https://stackoverflow.com/a/40415477/232371

Answer 5

文字进度条

对于那些想要文本进度条而没有像tqdm这样的附加模块的人来说，还有一个变种。在16.04.2018的linux上，joblib = 0.11，python 3.5.2的实际值，显示了子任务完成时的进度。

重新定义本机类：

class BatchCompletionCallBack(object):
    # Added code - start
    global total_n_jobs
    # Added code - end
    def __init__(self, dispatch_timestamp, batch_size, parallel):
        self.dispatch_timestamp = dispatch_timestamp
        self.batch_size = batch_size
        self.parallel = parallel

    def __call__(self, out):
        self.parallel.n_completed_tasks += self.batch_size
        this_batch_duration = time.time() - self.dispatch_timestamp

        self.parallel._backend.batch_completed(self.batch_size,
                                           this_batch_duration)
        self.parallel.print_progress()
        # Added code - start
        progress = self.parallel.n_completed_tasks / total_n_jobs
        print(
            "\rProgress: [{0:50s}] {1:.1f}%".format('#' * int(progress * 50), progress*100)
            , end="", flush=True)
        if self.parallel.n_completed_tasks == total_n_jobs:
            print('\n')
        # Added code - end
        if self.parallel._original_iterator is not None:
            self.parallel.dispatch_next()

import joblib.parallel
joblib.parallel.BatchCompletionCallBack = BatchCompletionCallBack

使用前定义全局常量与作业总数：

total_n_jobs = 10

这将导致类似这样的事情：

Progress: [########################################          ] 80.0%

Answer 6

在Jupyter中，每次输出时，tqdm都会在输出中开始新行。 因此对于Jupyter Notebook，它将是：

from joblib import Parallel, delayed
from datetime import datetime
from tqdm import tqdm_notebook

def myfun(x):
    return x**2

results = Parallel(n_jobs=8)(delayed(myfun)(i) for i in tqdm_notebook(range(1000)))  
100% 1000/1000 [00:06<00:00, 143.70it/s]

Answer 7

从dano和Connor的答案又向前走的一步是将整个内容包装为上下文管理器：

import contextlib
import joblib
from tqdm import tqdm
from joblib import Parallel, delayed

@contextlib.contextmanager
def tqdm_joblib(tqdm_object):
    """Context manager to patch joblib to report into tqdm progress bar given as argument"""
    class TqdmBatchCompletionCallback:
        def __init__(self, time, index, parallel):
            self.index = index
            self.parallel = parallel

        def __call__(self, index):
            tqdm_object.update()
            if self.parallel._original_iterator is not None:
                self.parallel.dispatch_next()

    old_batch_callback = joblib.parallel.BatchCompletionCallBack
    joblib.parallel.BatchCompletionCallBack = TqdmBatchCompletionCallback
    try:
        yield tqdm_object
    finally:
        joblib.parallel.BatchCompletionCallBack = old_batch_callback
        tqdm_object.close()    

然后，您可以像这样使用它，一旦完成，就不要留下猴子修补的代码：

with tqdm_joblib(tqdm(desc="My calculation", total=10)) as progress_bar:
    Parallel(n_jobs=16)(delayed(sqrt)(i**2) for i in range(10))

我认为这很棒，它看起来与tqdm pandas集成类似。

Answer 8

TLDR解决方案：

使用python 3.5与joblib 0.14.0和tqdm 4.46.0一起使用。感谢frenzykryger提供contextlib建议，感谢dano和Connor提供猴子修补想法。

import contextlib
import joblib
from tqdm import tqdm
from joblib import Parallel, delayed

@contextlib.contextmanager
def tqdm_joblib(tqdm_object):
    """Context manager to patch joblib to report into tqdm progress bar given as argument"""

    def tqdm_print_progress(self):
        if self.n_completed_tasks > tqdm_object.n:
            n_completed = self.n_completed_tasks - tqdm_object.n
            tqdm_object.update(n=n_completed)

    original_print_progress = joblib.parallel.Parallel.print_progress
    joblib.parallel.Parallel.print_progress = tqdm_print_progress

    try:
        yield tqdm_object
    finally:
        joblib.parallel.Parallel.print_progress = original_print_progress
        tqdm_object.close()

您可以按照frenzykryger所述的相同方式使用

import time
def some_method(wait_time):
    time.sleep(wait_time)

with tqdm_joblib(tqdm(desc="My method", total=10)) as progress_bar:
    Parallel(n_jobs=2)(delayed(some_method)(0.2) for i in range(10))

详细说明：

Jon的解决方案易于实现，但仅测量已分派的任务。如果任务花费很长时间，则在等待上一个分派的任务完成执行时，进度条将停留在100％。

frenzykryger的上下文管理器方法是从dano和Connor改进而来的，虽然更好，但是也可以在任务完成之前用BatchCompletionCallBack调用ImmediateResult（请参见Intermediate results from joblib）。这将使我们的计数超过100％。

我们可以只对BatchCompletionCallBack中的print_progress函数进行修补，而不必像猴子那样修补Parallel。 BatchCompletionCallBack已经调用了print_progress。如果设置了详细级别（即Parallel(n_jobs=2, verbose=100)），则print_progress将打印出已完成的任务，尽管不如tqdm好。看代码，print_progress是一个类方法，因此它已经有self.n_completed_tasks记录了我们想要的数字。我们要做的只是将其与joblib的当前状态进行比较，并仅在存在差异时进行更新。

这已使用python 3.5在joblib 0.14.0和tqdm 4.46.0中进行了测试。