Question

我如何在这个python循环中清除内存？

import concurrent.futures as futures
with futures.ThreadPoolExecutor(max_workers=100) as executor:
    fs = [executor.submit(get_data, url) for url in link]
    for i, f in enumerate(futures.as_completed(fs)):
        x= (f.result())
        results.append(x)
        del x 
        del f

get_data - 使用请求的简单函数

Answer 1

我的解决方案是这样的：
将concurrent.futures导入为期货

#split the original grand list into smaller batches  

batchurlList = [grandUrlList[x:x+batchSize] for x in range(0, len(grandUrlList), batchSize)]
for tmpurlList in batchurlList:
    with futures.ThreadPoolExecutor(max_workers=100) as executor:
        myfuture = {executor.submit(myFunction, url): url for url in tmpurlList}
        for future in futures.as_completed(myfuture, timeout=60):
            originalUrl = myfuture[future]
            results.append(future.result())

Answer 2

我认为我最近遇到了同样的问题，答案不是del而是引入睡眠功能...尝试;

import time
import concurrent.futures as futures
with futures.ThreadPoolExecutor(max_workers=100) as executor:
    fs = [executor.submit(get_data, url) for url in link]
    for i, f in enumerate(futures.as_completed(fs)):
           x= (f.result())
           results.append(x)
           time.sleep(n_seconds)

或类似的东西（我在网址列表上使用了一个while循环）

在python循环中清除内存

2 个答案: