我一直试图使用simple-requests和 BeautifulSoup 异步地从多个页面中删除一些数据,但我一直在努力绕过Cloudfare使用cloudfare-scrape过滤异步 。
from bs4 import BeautifulSoup
from simple_requests import Requests
import cfscrape
import requests
session = requests.session()
scraper = cfscrape.create_scraper(sess=session)
urls = ['http://cfsite.com/1','http://cfsite.com/2','http://cfsite.com/3']
for response in Requests(concurrent=8, minSecondsBetweenRequests=0.01).swarm(urls):
soup = BeautifulSoup(response.content, "html5lib")
''' BeautifulSoup magic happens here '''
现在,如何将请求会话替换为使用cfscrape创建的会话?