Question

假设我有许多可通过SSH访问的主机，以及（可以多得多）可以提交给任何这些主机的命令。以下假设是正确的：

命令应按FIFO安排。
每个命令的执行时间为未知和与主机无关。
一个主机一次只能执行一个命令。
一旦任何命令在任何主机上失败，我希望中断所有执行并引发异常。

考虑到这一点，我实现了一些基于concurrent.futures的代码来完成此调度。下面提供了一个独立的版本，仅假定您可以SSH到localhost。

import concurrent.futures
import shlex
import subprocess
import random

import logging

from collections import deque


logging.basicConfig(format=logging.BASIC_FORMAT, level=logging.DEBUG)

random.seed(13)

MAX_DURATION = 10
NUM_COMMANDS = 100
NUM_HOSTS = 5
HOSTS = ['localhost'] * NUM_HOSTS


def simulate_remote_call(host: str) -> None:
    duration = random.randrange(1, MAX_DURATION + 1)
    cmd = ['sleep', str(duration)]
    quoted_cmd = [shlex.quote(token) for token in cmd]
    logging.debug("Sleeping for %d seconds on %s", duration, host)
    subprocess.check_call(['ssh', host, *quoted_cmd])
    logging.debug("Done sleeping on %s", host)


with concurrent.futures.ThreadPoolExecutor(max_workers=len(HOSTS)) as executor:
    it = iter(range(1, NUM_COMMANDS + 1))
    queue = deque(HOSTS, len(HOSTS))
    future_to_host = {}

    while True:
        return_when = concurrent.futures.FIRST_EXCEPTION

        while queue:
            try:
                command = next(it)
            except StopIteration:
                break
            else:
                host = queue.popleft()
                logging.debug("Submitting command %d", command)
                future = executor.submit(simulate_remote_call, host)
                future_to_host[future] = host
        else:
            return_when = concurrent.futures.FIRST_COMPLETED

        if not future_to_host:
            break

        done, not_done = concurrent.futures.wait(future_to_host,
                                                 return_when=return_when)

        for future in done:
            host = future_to_host[future]
            exception = future.exception()

            if exception is not None:
                for other in not_done:
                    other.cancel()

                raise exception

            del future_to_host[future]
            queue.append(host)

但是，这段代码看起来非常不自然，我正在寻找一种更简单的方法来完成此操作。有什么想法吗？

具有并发功能的FIFO调度

0 个答案: