Question

我正在尝试在此网站的列表中提取每个所谓的抽奖活动的标题和链接。但是，当我尝试抓取这些数据时，似乎找不到。

我尝试抓取页面上的所有链接，但我认为这些“框”可能是通过javascript加载的。

我收到的结果是一些与我想要获得的内容无关的链接。此列表中应该显示40多个链接，但大多数链接未显示。任何帮助都会很棒，请坚持一段时间

For some reason, this link and many others aren't showing up when I am scraping

def raffle_page_collection():
    chrome_driver()
    page = requests.get('https://www.soleretriever.com/yeezy-boost-350-v2-black/')
    soup = BeautifulSoup(page.text,'html.parser')
    product_header = soup.find('h1').text
    product_colorway = soup.find('h2').text
    product_sku_and_release_date_and_price = soup.find('h3').text

    container = soup.find(class_='main-container')
    raffles = container.find_all('a')

    raffle_list = []

    for items in raffles:
        raffle_list.append(items.get('href'))

    print(raffle_list)

Answer 1

您应该尝试自动化硒库。它允许您剪贴动态呈现请求（js或ajax）页面数据。

尝试一下：

from selenium import webdriver
from bs4 import BeautifulSoup
import time

browser = webdriver.Chrome()
browser.get('https://www.soleretriever.com/yeezy-boost-350-v2-black/')
time.sleep(3)

soup = BeautifulSoup(browser.page_source,'html.parser')

product_header = soup.find('h1').text
product_colorway = soup.find('h2').text
product_sku_and_release_date_and_price = soup.find('h3').text

container = soup.find(class_='main-container')
raffles = container.find("div",{"class":"vc_pageable-slide-wrapper vc_clearfix"})

raffle_list = []

for items in raffles.find_all("a",href=True):
    raffle_list.append(items.get('href'))

print(product_header)
print(product_colorway)
print(product_sku_and_release_date_and_price)
print(raffle_list)

O / P：

Yeezy Boost 350 v2 Black
Black/ Black/ Black
FU9006 | 07/06/19 | $220
['https://www.43einhalb.com/en/adidas-yeezy-boost-350-v2-black-328672#SR', 'https://www.adidas.co.uk/yeezy#SR', 'https://www.allikestore.com/default/adidas-yeezy-boost-350-v2-static-black-black-fu9006-105308.html#SR', 'https://archive1820.com/en/forms/6/yeezy-raffle#SR', 'https://drops.blackboxstore.com/blackbox_launches_view/catalog/product/view/id/22296/s/yeezy-boost-350-v2#SR', 'https://woobox.com/4szm9v#SR', 'https://launches.endclothing.com/product/yeezy-boost-350-v2-fu9006#SR', 'https://www.instagram.com/p/ByEIHSHDSY6/', 'https://www.instagram.com/p/ByFG1G0lWf7/', 'https://releases.footshop.com/adidas-yeezy-boost-350-v2-agqn6WoBJZ9y4RSnzw9G#SR', 'https://launches.goodhoodstore.com/launches/yeezy-boost-350-v2-black-33#SR', 'https://www.hervia.com/launches/yeezy-350#SR', 'https://www.hibbett.com/adidas-yeezy-350-v2-black-mens-shoe/M0991.html#SR', 'https://reporting.jdsports.co.uk/cgi-bin/msite?yeezy_comp+a+0+0+0+0+0&utm_source=RedEye&utm_medium=Email&utm_campaign=Yeezy%20Boost%20351%20Clay&utm_content=0905%20Yeezy%20Clay#SR', 'https://www.instagram.com/p/ByDnK6uH6kE/', 'https://www.nakedcph.com/yeezy-boost-v2-350-static-black-raffle/s/635#SR', 'https://www.instagram.com/p/ByIXT8zHvYz/', 'https://launches.sevenstore.com/launch/yeezy-boost-350-v2-black-4033024#SR', 'https://shelta.eu/news/adidas-yeezy-boost-350-v2-black-fu9006x#SR', 'https://www.instagram.com/p/ByDI_6JAfty/', 'https://www.sneakersnstuff.com/en/product/38889/adidas-yeezy-350-v2#SR', 'https://www.instagram.com/p/ByHtt3HFkE0/', 'https://www.instagram.com/p/ByCaKR7Cde1/', 'https://tres-bien.com/adidas-yeezy-boost-350-v2-black-fu9006-fw19#SR', 'https://yeezysupply.com/products/yeezy-boost-350-v2-black-june-7-2019#SR']

对于Chrome浏览器：

http://chromedriver.chromium.org/downloads

为Chrome浏览器安装Web驱动程序：

https://christopher.su/2015/selenium-chromedriver-ubuntu/

硒教程

https://selenium-python.readthedocs.io/

在页面上找不到数据

1 个答案: