Question

我对编码，Python和Web抓取非常陌生。我正在尝试通过Webscrape获取商品价格的Amazon搜索结果页面。每当我将此代码作为一个块运行时，price都会返回0个元素。但是，如果我一次手动运行每一行，则price将作为页面上价格的正确列表返回。

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Safari()
url = 'https://www.amazon.com/s?k=crystal+chandelier&qid=1596736254&ref=sr_pg_1'
get = driver.get(url)
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
price = soup.find_all('span', class_ = 'a-offscreen')

for data in price:
    print(data.text)

运行整个程序没有任何输出。当我逐行运行时，这是我的输出：

$169.99
$99.99
$31.99
$169.97
$31.99
$39.99
$129.00
$72.35
$99.99
$49.90
$79.99
$214.98
$39.99
$89.99
$169.99
$56.99
$59.99
$149.99
$148.00
$124.28
$150.00
$124.50
$148.00
$438.00
$526.00
$119.99
$187.77
$139.99
$99.99
$369.00
$159.99
$169.99
$55.69
$29.98
$229.99
$169.99
$119.99
$119.99
$69.99
$369.00
$75.00
$63.99
$35.90
$75.99
$33.99
$88.00
$30.99
$298.00
$82.00
$36.99
$29.60
$119.99
$116.99
$124.80
$25.99
$32.99
$161.77
$114.99
$139.99
$256.16
$129.00
$229.99
$63.99
$199.00
$129.99
$129.99
$109.99

Answer 1

您可以使用带有正确HTTP标头的requests库来获取页面的正确版本：

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0',
    'Accept-Language': 'en-US,en;q=0.5'
}

url = 'https://www.amazon.com/s?k=crystal+chandelier&qid=1596736254&ref=sr_pg_1'
soup = BeautifulSoup(requests.get(url, headers=headers).content, 'lxml')
price = soup.find_all('span', class_ = 'a-offscreen')

for data in price:
    print(data.text)

打印：

$169.99
$99.99
$31.99
$89.99
$169.99
$119.99
$82.00
$148.00
$214.98

...and so on.

Beautiful Soup仅在逐行手动执行代码时起作用

1 个答案: