Question

我最近一直在努力想出一种好的方法来等待任务生产者完成芹菜工作者的任务。我想出了一种方法，但这种方法似乎运行得不够快，这里是：

芹菜生产者：

leafs = []

def chain_tasks():
    for i in range(1, 10):
        p1 = ping1.si(i)
        p2 = ping2.si(i)
        p3 = ping3.si(i)
        p4 = ping4.si(i)
        mychain = chain(p1, p2, p3, p4)
        leaf_id = mychain.apply_async()
        leafs.append(leaf_id)
    print('[INFO] Total leafs ->', leafs)


def _cancel_tasks(msg):
    print("[ERROR] Dummy Task canceller->", msg)

def parent_succeeds(t):
    if t.parent == None:
        return True
    else:
        parent_succeeded = True
        parent = t.parent
        if parent.state == 'PENDING':
            parent_succeeded = parent_succeeds(parent)
        if not parent_succeeded:
            return False
        print('[INFO] Waiting on parentTask({0})...at {1} - {2}'.format(parent, datetime.now().strftime("%H:%M:%S"), parent.state), end='')
        parent.wait(propagate=True)
        print('Done.')
        return parent.state != 'FAILURE'

def wait_for_comp():
    print("[INFO] Waiting for celery to finish...")
    max_fail = round(len(leafs) / 2)
    fail_count = 0
    for t in leafs:
        if fail_count <= max_fail:
            print('[INFO] Waiting on Task({0})...at {1}'.format(t, datetime.now().strftime("%H:%M:%S")))
            try:
                if parent_succeeds(t):
                    t.wait(propagate=True)
                else:
                    print('[ERROR] One of the parent failed -> ', t.parent)
            except Exception as e:
                fail_count += 1
                print('[ERROR] Exception Occurred [' + datetime.now().strftime("%H:%M:%S") + '] ->', str(e), fail_count)
                print('[ERROR] Traceback [' + datetime.now().strftime("%H:%M:%S") + '] ->', traceback.format_exc())
        else:
            print("[ERROR] Failed!")
            _cancel_tasks('failure of more than half tasks({0}/{1})'.format(fail_count, max_fail))
            break
    print("[INFO] Done.")

if __name__ == '__main__':
    time_start = time.time()
    chain_tasks()
    wait_for_comp()
    print('Finish time %s', time.time() - time_start)

这种方法的错误之处在于，它会在任务序列（for-loop）上等待，而不必在工作人员端进行维护，因为工作人员的执行基于Rabbit-mq条目。因此，这涉及很多等待。

有没有其他方法可以使等待更有效率？

Answer 1

如果我没有错过重要的事情，那么简单的解决方案是将'\\<'更改为类似的内容：

leaf_id = mychain.apply_async()

注意：do not call get() in your tasks。

如何等待芹菜生产商完成芹菜工作者的任务

1 个答案: