Question

我对这个概念很陌生，但我正在尝试学习如何使用python来操作HTML数据。

我写了一个python（版本3.4.1）脚本，它获取URL并返回一些信息，我使用BeautifulSoup解析了一些信息（版本4）。

import urllib.request
from bs4 import BeautifulSoup

response = urllib.request.urlopen('http://www.walmart.ca/en/ip/xbox-one/6000187109065')
html = response.read()

soup = BeautifulSoup(html)

print(soup.find_all('div', {"class" : "price-current"}))

在这个例子中，我试图获得Xbox One的价格。我选择了这个div，因为它是在网页上向用户显示价格的div。我知道有一个<span itemprop="price">$399.99</span>可用于刮擦。从中进行刮痧是相当直截了当的，我更为好奇的是为什么价格不会出现在它应该的div中。

我假设它与HTML标题有关，或者某种POST / GET数据与使用标准Web浏览器浏览时自动发送。任何人都可以解释为什么价格没有出现以及为了让它们按预期显示我必须做些什么？

Answer 1

使用javascript加载价格。这是请求：

Request URL:http://www.walmart.ca/ws/online/products
Request Method:POST

Form Data:
products:[{"productid":"6000187109066","skus":[[{"skuid":"6000187109066","status":"10"}]]}]
csrfToken:b08bfe580f3d9a0d893435fb

由于它包含csrfToken，因此您需要弄清楚在发出请求之前如何生成或提供它。他们可能也依赖会话cookie。

看来walmart有一个API：https://developer.walmartlabs.com/

Answer 2

免责声明：我在 SerpApi 工作。

您可以通过 SerpApi 的 google-search-results 包抓取沃尔玛产品的价格。这是一个免费试用的付费 API。

Full example

import os
from serpapi import GoogleSearch

params = dict(
    engine="walmart",
    query="xbox one",
    api_key=os.getenv("API_KEY"),
)

search = GoogleSearch(params)
results = search.get_dict()

for organic_result in results["organic_results"]:
    product_id = organic_result["us_item_id"] or organic_result["product_id"]

    search.params_dict = dict(
        engine="walmart_product",
        product_id=product_id,
        api_key=os.getenv("API_KEY"),
    )

    product_result = search.get_dict()

    print(
        f"Price of '{product_result['product_result']['title']}': '{product_result['product_result']['price_map']['price']} {product_result['product_result']['price_map']['currency']}'"
    )

输出

Price of 'Microsoft Xbox One S 1TB Console (Certified Refurbished)': '429 $'
Price of 'Star Wars Jedi: Fallen Order, Electronic Arts, Xbox One': '39.82 $'
Price of 'Microsoft - Xbox One S 1TB All-Digital Edition Console with Xbox One Wireless Controller (Renewed)': '499.99 $'
Price of 'Need for Speed: Heat, Electronic Arts, Xbox One, 014633373233': '25 $'
Price of 'Call of Duty: Black Ops Cold War, Activision, Xbox One': '49.94 $'
Price of 'NBA 2K21, 2K, Xbox One, 710425596858': '29.96 $'
Price of 'Refurbished Microsoft Factory Xbox One X 1TB, 4K Ultra HD Gaming Console in Black, FMQ-00042, 889842246971': '499.99 $'
Price of 'Madden NFL 21, Electronic Arts, Xbox One': '29.66 $'
Price of 'Need for Speed Payback, Electronic Arts, Xbox One, 014633370058': '14.96 $'
Price of 'Nickelodeon Kart Racers, Gamemill, Xbox One, 856131008060': '14.96 $'
Price of 'Sonic Mania (Sega), Xbox One, 010086640939': '14.96 $'
Price of 'Minecraft Master Collection, Microsoft, Xbox One, 889842394979': '42.88 $'
Price of 'Mortal Kombat 11, Warner Bros., Xbox One, 883929668977': '15 $'
Price of 'UFC 4, Electronic Arts, Xbox One': '29.96 $'
Price of 'Grand Theft Auto V, Rockstar Games, Xbox One': '22.8 $'
Price of 'Man Eater, Deep Silver, Xbox One, 816819017517': '24.96 $'
Price of 'Crash N. Sane Trilogy, Activision, Xbox One, 047875881969': '39.99 $'
Price of 'Spyro Reignited Trilogy, Activision, Xbox One, 047875882423': '29.83 $'
Price of 'Call of Duty: Modern Warfare, Activision, Xbox One, 0047875884366': '47.9 $'
Price of 'Cyberpunk 2077, Warner Bros, Xbox One': '39.74 $'
Price of 'Mass Effect Andromeda, Electronic Arts, Xbox One, 014633734096': '6.91 $'
Price of 'THQ theHunter: Call Of the Wild (Xbox One)': '19.93 $'
Price of 'Wreckfest, THQ-Nordic, Xbox One, 0811994021649': '29.83 $'
Price of 'Assassin’s Creed Valhalla Xbox Series X|S, Xbox One Standard Edition': '36.51 $'
Price of 'NBA 2K20, 2K, Xbox One, 710425595264': '21.99 $'
Price of 'LEGO Worlds, Warner Bros, Xbox One': '15 $'
Price of 'Call of Duty: Black Ops 3 Zombie Chronicles Edition, Activision, Xbox One, 047875881228': '37.88 $'
Price of 'Jumanji The Video Game, Bandai Namco, Xbox One, 819338020792': '19.93 $'
Price of 'Red Dead Redemption 2, Rockstar Games, Xbox One': '34.99 $'
Price of 'HALO 5, Microsoft, Xbox One, 885370928518': '19 $'
Price of 'LEGO Jurassic World, Warner, Xbox One, 883929472727': '19.8 $'
Price of 'Plants vs. Zombies: Battle for Neighborville, Electronic Arts, Xbox One, 014633736007': '14.96 $'
Price of 'The LEGO Marvel Collection, Warner Bros., Xbox One, 00883929670499': '24.98 $'
Price of 'LEGO The Incredibles, Warner Bros, Xbox One, 883929633005': '18 $'
Price of 'Assassin's Creed: Origins, Ubisoft, Xbox One, 887256028459': '19.93 $'
Price of 'Spongebob Squarepants: Battle for Bikini Bottom Rehydrated, THQ-Nordic, Xbox One, 811994022165': '29.77 $'
Price of 'The SIMS 4, Electronic Arts, Xbox One': '33.84 $'
Price of 'Star Wars Battlefront 2, Electronic Arts, Xbox One, 014633735321': '14.96 $'
Price of 'Pac-Man Championship Edition 2 + Arcade Game Series, Bandai/Namco, Xbox One, 722674220705': '14.49 $'
Price of 'Microsoft Xbox One S 1TB All Digital Edition 3 Game Bundle (Disc-free Gaming), White, NJP-00050': '198.95 $'

沃尔玛价格刮擦Python 3

2 个答案: