Python Requests.get()无法与多重处理pool.map一起使用

时间:2019-03-01 10:28:53

标签: python python-requests multiprocessing screen-scraping

from multiprocessing import Pool
import requests
import time as time
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
def foo(url):
    print('requesting')
    x = requests.get('http://www.facebook.com', timeout = 10, headers = headers)
    time.sleep(1)
    print('requested')
pool = Pool(6)
try:
    dump = pool.map(foo, range(20))
finally:
    pool.terminate()
    pool.join()

我正在尝试使用pool.map并行抓取网站。上面的代码是一个最小的示例。我在jupyter笔记本中运行了以上代码,并将Requests.get()挂在那里。

我在做什么错了?

0 个答案:

没有答案