有没有什么方法可以将简单请求与cloudflare-scrape结合起来?

时间:2017-03-22 02:42:37

标签: python web-scraping python-requests

我一直试图使用simple-requests BeautifulSoup 异步地从多个页面中删除一些数据,但我一直在努力绕过Cloudfare使用cloudfare-scrape过滤异步

from bs4 import BeautifulSoup
from simple_requests import Requests
import cfscrape
import requests

session = requests.session()
scraper = cfscrape.create_scraper(sess=session)

urls = ['http://cfsite.com/1','http://cfsite.com/2','http://cfsite.com/3']

for response in Requests(concurrent=8, minSecondsBetweenRequests=0.01).swarm(urls):
    soup = BeautifulSoup(response.content, "html5lib")
    ''' BeautifulSoup magic happens here '''

现在,如何将请求会话替换为使用cfscrape创建的会话?

0 个答案:

没有答案