Question

我通过concurrent.futures的ThreadPoolExecutor下载了一个网址列表，但是在所有第一次尝试结束后，我可能会有一些超时网址要重新下载。我不知道该怎么做，这是我的尝试，但是无休止的打印失败＆＃39; time_out_again＆＃39;：

import concurrent.futures

def player_url(url):
    # here. if timeout, return 1. otherwise do I/O and return 0.
    ...

urls = [...]
time_out_futures = [] #list to accumulate timeout urls
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    future_to_url = (executor.submit(player_url, url) for url in urls)
    for future in concurrent.futures.as_completed(future_to_url):
        if future.result() == 1:
            time_out_futures.append(future)

# here is what I try to deal with all the timeout urls       
while time_out_futures:
    future = time_out_futures.pop()
    if future.result() == 1:
        print('time_out_again')
        time_out_futures.insert(0,future)   # add back to the list

那么，有什么方法可以解决这个问题吗？

Answer 1

Future个对象只能使用一次。 Future本身对返回结果的函数一无所知 - ThreadPoolExecutor对象负责创建Future，返回它，并在后台运行函数：

def submit(self, fn, *args, **kwargs):
    with self._shutdown_lock:
        if self._shutdown:
            raise RuntimeError('cannot schedule new futures after shutdown')

        f = _base.Future()
        w = _WorkItem(f, fn, args, kwargs)

        self._work_queue.put(w)
        self._adjust_thread_count()
        return f

class _WorkItem(object):
    def __init__(self, future, fn, args, kwargs):
        self.future = future
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    def run(self):
        if not self.future.set_running_or_notify_cancel():
            return

        try:
            result = self.fn(*self.args, **self.kwargs)  # sefl.fn is play_url in your case
        except BaseException as e:
            self.future.set_exception(e)
        else:
            self.future.set_result(result)  # The result is set on the Future

如您所见，当函数完成时，结果将在Future对象上设置。因为Future对象实际上并不知道提供结果的函数，所以无法尝试使用Future对象重新运行该函数。您所能做的就是在超时发生时返回url和1，然后重新submit到ThreadPoolExecutor的网址：

def player_url(url):
    # here. if timeout, return 1. otherwise do I/O and return 0.
    ...
    if timeout:
        return (1, url)
    else:
        return (0, url)

urls = [...]
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    while urls:
        future_to_url = executor.map(player_url, urls)
        urls = []  # Clear urls list, we'll re-add any timed out operations.
        for future in future_to_url:
            if future.result()[0] == 1:
                urls.append(future.result()[1]) # stick url into list

Python 3 concurrent.futures：如何将失败的期货添加回ThreadPoolExecutor？

1 个答案: