Python搜寻器仅在前几个div中找到所需的标签

Question

我正在尝试从购物网站（https://www.grailed.com/shop/EkpEBRw4rw）上抓取一些图像，但是由于列表在滚动时会更新，因此我遇到了一些麻烦。我正在尝试在下面的HTML标签中获取图像源：

现在我正在使用的代码如下所示：

from bs4 import BeautifulSoup
from selenium import webdriver

url = 'https://www.grailed.com/shop/EkpEBRw4rw'
driver = webdriver.Chrome(executable_path='chromedriver.exe')
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'html.parser')

listing = soup.select('.listing-cover-photo ')
for item in listing:
    print(item.select('img'))

问题在于，尽管它确实找到了每个列表的标签，但只能找到前6个列表的标签。我的代码的输出如下所示：

输出：

[<img alt="Off-White Off White Caravaggio Hoodie" src="https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:480,height:640,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/yX8vvvBsTaugadX0jssT"/>]
(...a few more of these...)
[<img alt="Off-White Off-White Arrows Hoodie Black" src="https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:480,height:640,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/9CMvJoQIRaqgtK0u9ov0"/>]
[]
[]
[]
[]
(...many more empty lists...)

即使循环遍历侧面的所有页面（向URL添加'？page = n'），这种情况仍然存在，并且仅显示每页的前6个条目。

Answer 1

要使用shopping site在Selenium中刮擦中<img>标记的 src 属性，您需要为 WebDriverWait visibility_of_all_elements_located()，您可以使用以下解决方案：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions()
  options.add_argument('start-maximized')
  options.add_argument('disable-infobars')
  options.add_argument('--disable-extensions')
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get('https://www.grailed.com/shop/EkpEBRw4rw')
  print([my_image.get_attribute("src") for my_image in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.listing-cover-photo>img")))])
  

• 控制台输出：

  ['https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/yX8vvvBsTaugadX0jssT', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/YjiErjJNQrarKGDuGr3S', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/G9CwIli8QUW3uGgZeirk', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/Ta9DAxg4SeKAT6kBLyJo', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/QglmTKyTxu31PeDFWFnw', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/9CMvJoQIRaqgtK0u9ov0', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/MCJY9cSQsiSU4TlSTcD7', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/L4NHu1ByT3Kwn8dRsdBX']