Question

我想优化脚本以尽可能多地发出网络请求。我看到max_workers可能仅限于机器上的核心数量。这是否意味着如果此脚本在EC2计算机上运行，例如t2.2xlarge有8个vCPU，则脚本实际上将限制为8，例如WORKERS = 8？

如果是这样，是否有更好的方法一次发出8个以上的请求？

示例：

WORKERS = 16 # should this be limited to 8?

def make_req_futures(url_list):
    # We can use a with statement to ensure threads are cleaned up promptly
    with concurrent.futures.ThreadPoolExecutor(max_workers=WORKERS) as executor:
        # Start the load operations and mark each future with its URL
        future_to_url = {executor.submit(load_url, url, 60): url for url in url_list}
        for future in concurrent.futures.as_completed(future_to_url):
            url = future_to_url[future]

            try:
                print("getting: ", url)
                data = future.result()
            except Exception as exc:
                failed_urls.append([url, exc])
                print('%r generated an exception: %s' % (url, exc))
            else:
                success_urls.append(url)
                print('"%s" fetched in %ss' % (url,(time.time() - start)))
    print("Elapsed Time: %ss" % (time.time() - start))

并发期货：多个异步请求的最佳设置？

0 个答案: