我正在尝试将图片从网页上刮掉。它使用JS渲染,并且源代码中的图片链接不完整。这是图片的来源:
<script language="javascript" type="text/javascript">
</script>
<div id="ImagesSection" class="ImagesSection">
<div id='HybridImageViewPrimaryImageDiv'>
<a href='/ItemImages/000450/18190933_1_lg.jpeg' class="MagicZoom" data-options=" zoomMode:off; cssClass: dark-bg; zoomOn: click" title='Multi-Faced Doll By Cark Bergner.' id="xxxyyyzzz" ><img id='fullimage' src='/ItemImages/000450/18190933_1_med.jpeg' alt='Multi-Faced Doll By Cark Bergner.' /></a>
</div>
<div style="margin-top:15px;width:300px;"> <button class="cfg-btn" onclick="MagicZoom.prev('xxxyyyzzz');return false;">Prev</button> <button class="cfg-btn" onclick="MagicZoom.next('xxxyyyzzz') ;return false;">Next</button>
</div><div style="margin-top:15px;" width="350px" >
<a data-zoom-id="xxxyyyzzz" href="/ItemImages/000450/18190933_1_lg.jpeg" data-image="/ItemImages/000450/18190933_1_med.jpeg" > <img src="/ItemImages/000450/18190933_1_sm.jpeg" height="60px" /> </a>
<a data-zoom-id="xxxyyyzzz" href="/ItemImages/000450/18190933_2_lg.jpeg" data-image="/ItemImages/000450/18190933_2_med.jpeg" > <img src="/ItemImages/000450/18190933_2_sm.jpeg" height="60px" /> </a>
<a data-zoom-id="xxxyyyzzz" href="/ItemImages/000450/18190933_3_lg.jpeg" data-image="/ItemImages/000450/18190933_3_med.jpeg" > <img src="/ItemImages/000450/18190933_3_sm.jpeg" height="60px" /> </a>
<a data-zoom-id="xxxyyyzzz" href="/ItemImages/000450/18190933_4_lg.jpeg" data-image="/ItemImages/000450/18190933_4_med.jpeg" > <img src="/ItemImages/000450/18190933_4_sm.jpeg" height="60px" /> </a>
<a data-zoom-id="xxxyyyzzz" href="/ItemImages/000450/18190933_5_lg.jpeg" data-image="/ItemImages/000450/18190933_5_med.jpeg" > <img src="/ItemImages/000450/18190933_5_sm.jpeg" height="60px" /> </a>
</div>
</div>
我要提取的是以下图像:
/ItemImages/000450/18190933_1_sm.jpeg
/ItemImages/000450/18190933_2_sm.jpeg
/ItemImages/000450/18190933_3_sm.jpeg
/ItemImages/000450/18190933_4_sm.jpeg
/ItemImages/000450/18190933_5_sm.jpeg
这是我的代码:
import os
import shutil
import time
import requests
from bs4 import BeautifulSoup as bSoup
from selenium import webdriver
url = "https://auctions.morphyauctions.com/French_Fashion_Doll_with_Unusual_Body_-LOT450029.aspx"
driver = webdriver.Chrome(executable_path="/mypath/")
driver.get(url)
iterations = 0
while iterations <10:
html = driver.execute_script("return document.documentElement.outerHTML")
sel_soup = bSoup(html, 'html.parser')
print (sel_soup.findAll('img'))
images = []
for i in sel_soup.findAll('img'):
src = i['src']
images.append(src)
print(images)
current_path = os.getcwd()
for img in images:
try:
file_name = os.path.basename(img)
img_r = requests.get(img, stream=True)
new_path = os.path.join(current_path, 'images', file_name)
with open(new_path, 'wb') as output_file:
shutil.copyfilobj(img_r.raw, output_file)
del img_r
except:
pass
iterations +=1
time.sleep(5)
运行该代码的结果是未保存任何图像。任何帮助将不胜感激。
答案 0 :(得分:0)
图像的html不是由Javascript渲染的,因此您不需要硒。与re.compile
一起使用beautifulsoup,以匹配href
开头的a
元素中的/ItemImages/
。
请注意,它使用相对URL,您需要将域名附加到图片URL。
base_url = 'https://auctions.morphyauctions.com'
url = base_url + "/French_Fashion_Doll_with_Unusual_Body_-LOT450029.aspx"
html = requests.get(url).text
sel_soup = BeautifulSoup(html, 'html.parser')
images = []
for a in sel_soup.findAll('a', href=re.compile(r'^/ItemImages/')):
ahref = base_url + a['href'] # wee need to append the domain here
images.append(ahref)
print(images)
current_path = os.getcwd()
for img in images:
try:
file_name = os.path.basename(img)
img_r = requests.get(img)
new_path = os.path.join(current_path, 'images', file_name)
with open(new_path, 'wb') as output_file:
output_file.write(img_r.content)
except:
print(ex)