Question

我需要从该网站https://shop.freedompop.com/products?page=1抓取数据

我使用BeautifulSoup解析html，发现我需要查找所有class_="product-results-item-link layout-row flex-gt-sm-33 flex-50"

我尝试使用 containers = html_soup.find_all('a', class_="product-results-item-link layout-row flex-gt-sm-33 flex-50") 但找不到

    from requests import get
    from bs4 import BeautifulSoup
    from time import sleep
    from random import randint
    import pandas as pd

    product_names = []
    status = []
    ori_prices = []
    sale_prices = []

    headers = {"Accept-Language": "en-US, en;q=0.5"}

    pages = [str(i) for i in range(1,2)]
    #pages = [str(i) for i in range(1,24)]

    for page in pages:

        response = get('https://shop.freedompop.com/products' + page, headers = headers)
        sleep(2)

        html_soup = BeautifulSoup(response.text, 'html.parser')

        containers = html_soup.find_all('a', class_="product-results-item-link layout-row flex-gt-sm-33 flex-50")

        print(containers)

我希望输出为18，但实际输出为[]

Answer 1

网站通过api动态访问所有产品条目。因此，您可以直接使用其API并获取数据：

https://shop.freedompop.com/api/shop/store/555/item?page=1&pageSize=500&sort=RELEVANCE

import json
from requests import get
from bs4 import BeautifulSoup


response = get('https://shop.freedompop.com/api/shop/store/555/item?pageSize=410&sort=RELEVANCE')
html_soup = BeautifulSoup(response.text, 'html.parser')
parsed_response = json.loads(html_soup.text)


for index,value in enumerate(a.get('results')):
    print(index, value)

Answer 2

正如Pankaj所说的（所以接受他的回答，因为我只是在扩展他的最初回答），请使用请求url以不错的json格式获取数据。您还可以更改参数（即更改'pageSize': '500'以获得比首页上的18种产品更多的产品：

import requests


url = 'https://shop.freedompop.com/api/shop/store/555/item'

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
params = {
'page': '1',
'pageSize': '18',
'sort': 'RELEVANCE'}

jsonData = requests.get(url, headers=headers, params=params).json()

for product in jsonData['results']:
    print (product['title'])

输出：

Netgear Unite Mobile Hotspot (GSM)
LG Tribute 2, 8GB Blue (CDMA)
Samsung Galaxy S5, 16GB Charcoal Black (CDMA)
Samsung Galaxy S5, 16GB Shimmery White (CDMA)
Samsung Galaxy S5, 16GB Shimmery White (CDMA)
Samsung Galaxy S4 Enhanced, 16GB Black Mist (CDMA)
Kyocera Hydro Vibe, 8GB Black (CDMA)
Samsung Galaxy S4, 16GB White Frost (CDMA)
Samsung Galaxy S4, 16GB White Frost (CDMA)
Motorola Moto E (2nd Generation), 8GB Black (CDMA)
Apple iPhone 5s, 16GB Gold (CDMA)
Samsung Galaxy S4, 16GB Black Mist (CDMA)
Franklin Wireless R850 4G LTE Mobile Hotspot (CDMA)
Apple iPhone 6, 16GB Space Gray (CDMA)
Samsung Galaxy S4 Enhanced, 16GB White Frost (CDMA)
Huawei Union, 8GB Black (CDMA)
Samsung Galaxy S5, 16GB Copper Gold (CDMA)
Samsung Galaxy S4 Enhanced, 16GB Black Mist (CDMA)

不断变化的参数

import requests


url = 'https://shop.freedompop.com/api/shop/store/555/item'

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
params1 = {
'page': '1',
'pageSize': '18',
'sort': 'RELEVANCE'}

params2 = {
'page': '1',
'pageSize': '500',
'sort': 'RELEVANCE'}

jsonData = requests.get(url, headers=headers, params=params1).json()
print (len(jsonData['results']))

jsonData = requests.get(url, headers=headers, params=params2).json()
print (len(jsonData['results']))

输出：

18
405

如何找到标记“ a”以抓取数据？

2 个答案: