我正在尝试从网站的目标网页上抓取图像。所有图像都在search_results
类名称之内。当我运行下面的脚本时,没有任何结果。我检查了status_code
,发现脚本正在获取403
。
由于图像是静态的并且在页面源中可用,我如何使用请求来刮擦图像链接?
import requests
from bs4 import BeautifulSoup
url = 'https://pixabay.com/images/search/office/'
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
}
r = requests.get(url,headers=headers)
print(r.status_code)
soup = BeautifulSoup(r.text,"lxml")
for item in soup.select(".search_results a > img[src]"):
print(item.get("src"))
任何与任何浏览器模拟器相关的解决方案,例如硒,都不是我想要的。
答案 0 :(得分:3)
这使用Selenium
。但是由于某种原因,似乎无法在无头模式下找到图像:
from selenium import webdriver
from bs4 import BeautifulSoup
options = webdriver.ChromeOptions()
#options.add_argument("headless")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
try:
driver.implicitly_wait(3)
driver.get('https://pixabay.com/images/search/office')
images = driver.find_elements_by_css_selector('.search_results a > img[src]') # wait for images to show up
soup = BeautifulSoup(driver.page_source, 'lxml')
for item in soup.select(".search_results a > img[src]"):
print(item.get("src"))
finally:
driver.quit()
打印:
https://cdn.pixabay.com/photo/2016/03/09/09/22/workplace-1245776__340.jpg
https://cdn.pixabay.com/photo/2015/01/08/18/26/write-593333__340.jpg
https://cdn.pixabay.com/photo/2015/02/02/11/09/office-620822__340.jpg
https://cdn.pixabay.com/photo/2014/05/02/21/50/home-office-336378__340.jpg
https://cdn.pixabay.com/photo/2016/02/19/11/19/office-1209640__340.jpg
https://cdn.pixabay.com/photo/2015/02/02/11/08/office-620817__340.jpg
https://cdn.pixabay.com/photo/2016/03/26/13/09/cup-of-coffee-1280537__340.jpg
https://cdn.pixabay.com/photo/2017/05/11/11/15/workplace-2303851__340.jpg
https://cdn.pixabay.com/photo/2015/01/09/11/08/startup-594090__340.jpg
https://cdn.pixabay.com/photo/2015/01/08/18/25/startup-593327__340.jpg
https://cdn.pixabay.com/photo/2015/01/08/18/27/startup-593341__340.jpg
https://cdn.pixabay.com/photo/2014/05/02/21/49/home-office-336373__340.jpg
https://cdn.pixabay.com/photo/2015/01/09/11/11/office-594132__340.jpg
https://cdn.pixabay.com/photo/2017/05/04/16/37/meeting-2284501__340.jpg
https://cdn.pixabay.com/photo/2014/05/03/01/03/macbook-336704__340.jpg
https://cdn.pixabay.com/photo/2018/01/11/21/27/desk-3076954__340.jpg
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
答案 1 :(得分:2)
此页面使用JavaScript
和Cookies
,这会引起问题。它还会检查其他标头,而不仅仅是User-Agent
。
首先:您必须使用requests.Session()
来保存cookie。第二:您必须加载一些页面(即主页)以获取这些cookie。当您拥有Cookie时,它将接受其他URL。第三:它还会检查其他标头以发送cookie。
我在浏览器中运行页面,并在Chrome / Firefox中使用DevTools
复制真实浏览器使用的所有标头,然后使用不同的标头开始测试请求。终于我发现了它的需要
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
'Accept-Language': 'en-US;q=0.7,en;q=0.3',
'Cache-Control': 'no-cache',
}
另一个问题是,滚动页面时页面使用JavaScript
加载图像(“延迟加载”),并且某些URL不在scr
中,而是在data-lazy
中,然后在{{ 1}}具有src
'blank.gif'
答案 2 :(得分:2)
Pixabay正在使用Cloudflare的Web应用程序防火墙(WAF)或类似的产品。手动解决这个问题很繁琐。
cloudflare-scrape
是一个可能有帮助的库:https://github.com/Anorov/cloudflare-scrape