以下代码是非异步代码的示例,有没有办法异步获取图像?
import urllib
for x in range(0,10):
urllib.urlretrieve("http://test.com/file %s.png" % (x), "temp/file %s.png" % (x))
我也看过Grequests库,但如果可行,或者如何从文档中做到这一点我无法理解。
答案 0 :(得分:9)
您不需要任何第三方库。只需为每个请求创建一个线程,启动线程,然后等待所有这些线程在后台完成,或者在下载图像时继续运行应用程序。
import threading
results = []
def getter(url, dest):
results.append(urllib.urlretreave(url, dest))
threads = []
for x in range(0,10):
t = threading.Thread(target=getter, args=('http://test.com/file %s.png' % x,
'temp/file %s.png' % x))
t.start()
threads.append(t)
# wait for all threads to finish
# You can continue doing whatever you want and
# join the threads when you finally need the results.
# They will fatch your urls in the background without
# blocking your main application.
map(lambda t: t.join(), threads)
您可以选择创建一个从队列中获取urls
和dests
的线程池。
如果您使用的是Python 3,则已在futures
模块中为您实现。
答案 1 :(得分:3)
这样的事可以帮到你
import grequests
urls = ['url1', 'url2', ....] # this should be the list of urls
requests = (grequests.get(u) for u in urls)
responses = grequests.map(requests)
for response in responses:
if 199 < response.status_code < 400:
name = generate_file_name() # generate some name for your image file with extension like example.jpg
with open(name, 'wb') as f: # or save to S3 or something like that
f.write(response.content)
这里只有图像的下载是并行的,但是将每个图像内容写入文件是顺序的,这样你就可以创建一个线程或做其他事情来使它成为并行或异步