理想情况下，我有兴趣从本网站的“男装”部分抓取所有可用产品页面的所有产品信息：
https://www.adidas.de/manner-schuhe-sneakers
当我尝试使用此代码来汇总产品名称和网址时，它停止在页面上显示的前5个项目上，我不知道为什么：
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
import pandas as pd
url = 'https://www.adidas.de/manner-schuhe-sneakers'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
driver = webdriver.Chrome()
driver.get(url)
soup = BeautifulSoup(requests.get(url, headers = headers).content, 'lxml')
driver.quit()
for a in soup.select('div[class^="product-container"] a.gl-product-card__assets-link'):
label = a.find_next(class_='gl-label')
print('{:<50} {}'.format(label.text, 'https://www.adidas.com' + a['href']))
结果如下。有人可以帮我确保页面上的所有48个产品都被捕获吗？或者更好的是，有人可以帮我弄清楚如何遍历每个产品，获取所有可用样式的样式名称，颜色，价格，然后单击“下一步”，直到所有的产品信息都被获取为止：
Superstar Schuh https://www.adidas.com/superstar-schuh/FW2293.html
ZX 2K Boost Schuh https://www.adidas.com/zx-2k-boost-schuh/FV9996.html
NMD_R1 V2 Schuh https://www.adidas.com/nmd_r1-v2-schuh/FY6862.html
ZX 2K Boost Schuh https://www.adidas.com/zx-2k-boost-schuh/FV9993.html
NMD_R1 V2 Schuh https://www.adidas.com/nmd_r1-v2-schuh/FV9022.html
答案 0 :(得分：0)
我建议您使用 const winURL = process.env.NODE_ENV === 'development' ? `http://localhost:9080`:`file://${__dirname}/index.html`
mainWindow.loadURL(winURL)
，然后保存网址。
Xpaths输出：
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)
page = 0
url = 'https://www.adidas.de/manner-schuhe-sneakers?start={}'.format(page)
driver.get(url)
time.sleep(4)
element = driver.find_elements_by_xpath('//*[@data-auto-id="glass-hockeycard-link"]')
all_shoes = [e.get_attribute("href") for e in element]
print("Found {} shoes.".format(len(all_shoes)))
for shoe in all_shoes:
print('{:<50} {}'.format(shoe.split("/")[-2], shoe))
with open("shoes_page_{}.txt".format(page), "w") as f:
f.writelines('\n'.join(all_shoes))
为什么将URL保存到文件中？好吧，所以您不必继续抓取页面，并且可以使用URL来查询API。
Found 48 shoes.
superstar-schuh https://www.adidas.de/superstar-schuh/FW2293.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FV9996.html
nmd_r1-v2-schuh https://www.adidas.de/nmd_r1-v2-schuh/FY6862.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FV9993.html
nmd_r1-v2-schuh https://www.adidas.de/nmd_r1-v2-schuh/FV9022.html
ultraboost-schuh https://www.adidas.de/ultraboost-schuh/BB6168.html
superstar-schuh https://www.adidas.de/superstar-schuh/EG4958.html
ozweego-schuh https://www.adidas.de/ozweego-schuh/FV9667.html
nite-jogger-schuh https://www.adidas.de/nite-jogger-schuh/FV1267.html
continental-80-schuh https://www.adidas.de/continental-80-schuh/G27706.html
stan-smith-schuh https://www.adidas.de/stan-smith-schuh/M20325.html
gazelle-schuh https://www.adidas.de/gazelle-schuh/BB5478.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FY2001.html
zx-2k-flux-schuh https://www.adidas.de/zx-2k-flux-schuh/FX2044.html
ultraboost-schuh https://www.adidas.de/ultraboost-schuh/F36641.html
nmd_r1-schuh https://www.adidas.de/nmd_r1-schuh/FV8727.html
zx-500-schuh https://www.adidas.de/zx-500-schuh/FW2811.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FX8835.html
zx-700-hd-schuh https://www.adidas.de/zx-700-hd-schuh/FY0995.html
nmd_r1-v2-schuh https://www.adidas.de/nmd_r1-v2-schuh/FY5913.html
3mc-vulc-schuh https://www.adidas.de/3mc-vulc-schuh/B22705.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FV9997.html
ultraboost-20-laufschuh https://www.adidas.de/ultraboost-20-laufschuh/FV8329.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FV8453.html
ozweego-schuh https://www.adidas.de/ozweego-schuh/FV9655.html
continental-80-vegan-schuh https://www.adidas.de/continental-80-vegan-schuh/FW2336.html
nmd_r1-schuh https://www.adidas.de/nmd_r1-schuh/D96635.html
ultraboost-winter.rdy-laufschuh https://www.adidas.de/ultraboost-winter.rdy-laufschuh/EG9801.html
stan-smith-schuh https://www.adidas.de/stan-smith-schuh/FU9609.html
superstar-schuh https://www.adidas.de/superstar-schuh/EG4959.html
carrera-low-pride-schuh https://www.adidas.de/carrera-low-pride-schuh/FY9019.html
zx-flux-schuh https://www.adidas.de/zx-flux-schuh/S32279.html
supercourt-schuh https://www.adidas.de/supercourt-schuh/EE6037.html
zx-700-hd-schuh https://www.adidas.de/zx-700-hd-schuh/FY1102.html
terrex-ax3-beta-schuh https://www.adidas.de/terrex-ax3-beta-schuh/G26523.html
terrex-swift-r2-mid-gore-tex-wanderschuh https://www.adidas.de/terrex-swift-r2-mid-gore-tex-wanderschuh/CM7500.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FX8836.html
zx-500-schuh https://www.adidas.de/zx-500-schuh/FW2812.html
zx-2k-flux-schuh https://www.adidas.de/zx-2k-flux-schuh/FV9977.html
swift-run-rf-schuh https://www.adidas.de/swift-run-rf-schuh/FV5358.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FX8834.html
zx-500-schuh https://www.adidas.de/zx-500-schuh/FW4410.html
zx-2k-boost-schuh https://www.adidas.de/zx-2k-boost-schuh/FV9999.html
ultraboost-20-laufschuh https://www.adidas.de/ultraboost-20-laufschuh/FV8359.html
sabalo-schuh https://www.adidas.de/sabalo-schuh/FV0689.html
zx-700-hd-schuh https://www.adidas.de/zx-700-hd-schuh/FY0996.html
terrex-skychaser-lt-gtx-schuh https://www.adidas.de/terrex-skychaser-lt-gtx-schuh/F36099.html
terrex-free-hiker-cold.rdy-wanderschuh https://www.adidas.de/terrex-free-hiker-cold.rdy-wanderschuh/FU7217.html
输出：
import requests
headers = {
"accept": "*/*",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
"content-type": "application/json",
"referer": "https://www.adidas.de/en/men-trainers-shoes",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0",
}
with open("shoes_page_0.txt") as f:
shoes = f.readlines()
print('{:<20} {:<50} {:<10}'.format('Shoe Model', 'Color', 'Price'))
for shoe in shoes:
id_ = shoe.split("/")[-1].replace(".html", "")
shoe_data = requests.get(f"https://www.adidas.de/api/search/product/{id_}?sitePath=en", headers=headers).json()
print('{:<20} {:<50} {:<10}'.format(shoe_data['modelId'], shoe_data['color'], shoe_data['price']))
编辑：
要获得 all 鞋（到目前为止），您可以尝试一下，然后运行通过API获取信息的代码：
Shoe Model Color Price
DVF77 Cloud White / Cloud White / Core Black 97.43
KYJ02 Cloud White / Solar Red / Blue 136.42
KYK47 Core Black / Core Black / Cardboard 136.42
KYJ02 Core Black / Core Black / Shock Pink 136.42
KYK47 Cloud White / Core Black / Cloud White 136.42
DWG43 Cloud White / Cloud White / Cloud White 155.92
DVF77 Cloud White / Core Black / Cloud White 97.43
EFK26 Off White / Off White / Signal Pink 116.93
BTO93 Cloud White / Cloud White / Cloud White 126.68
DRA67 Cloud White / Scarlet / Collegiate Navy 97.43
ION05 Core White / Dark Blue / Dark Blue 92.56
IAZ12 Collegiate Navy / White / Gold Metallic 92.56
KYJ02 Linen / Core Black / Orange 136.42
KYJ11 Cloud White / Core Black / Blue 97.43
DWG43 Core Black / Core Black / Active Red 155.92
BSV73 Cloud White / Core Black / Cloud White 136.42
KYX38 Grey Four / Grey Six / Grey Three 97.43
and so on...
这会向所有鞋子输出一个带有url的文件。