抓取网页：总是返回空列表

Question

我在浏览此网页https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas时遇到了一些麻烦，我尝试了许多选择器，h1，标题，价格，并且总是返回相同的值：一个空列表，有人可以帮我提取产品名称吗？这是一个测试，非常感谢

scrapy shell "https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas"

response.css(".tileList-title").extract() #for the title
response.css(".price-dollars::text").extract() # for the prices 
response.css(".shelfProductTile-descriptionLink") #for the products na:e

Answer 1

网站上有一个私人api，可用于获取产品数据。我将json响应的一部分放入了pandas数据框。该api提供了有关您可以提取的产品的更多信息。

代码

import json, requests, pandas as pd

data = json.loads(
    r'{"categoryId":"1_9573995","pageNumber":1,"pageSize":24,"sortType":"TraderRelevance","url":"/shop/browse/drinks/cordials-juices-iced-teas/iced-teas","location":"/shop/browse/drinks/cordials-juices-iced-teas/iced-teas","formatObject":"{\"name\":\"Iced Teas\"}","isSpecial":false,"isBundle":false,"isMobile":false,"filters":null}')
url = 'https://www.woolworths.com.au/apis/ui/browse/category'
r = requests.post(url, data=data)
api = json.loads(r.text)

to_df = {'Names': [], 'Prices': [], 'Cup Strings': []}
for item in api['Bundles']:
    for product_stuff in item['Products']:
        to_df['Names'].append(product_stuff['Name'])
        to_df['Prices'].append(product_stuff['Price'])
        to_df['Cup Strings'].append(product_stuff['CupString'])

df = pd.DataFrame(data=to_df)
print(df)

输出

                                        Names  Prices Cup Strings
0                 Lipton Lipton Ice Tea Peach    10.0  $3.57 / 1L
1                  Arizona Ice Tea With Peach     2.0  $4.00 / 1L
2                 Arizona Ice Tea Pomegranate     2.0  $4.00 / 1L
3        Arizona Green Tea With Honey Ice Tea     2.0  $4.00 / 1L
4            Lipton Ice Green Tea Light Lemon     3.9  $2.60 / 1L
5                        Lipton Ice Tea Lemon     3.9  $2.60 / 1L
6                        Lipton Ice Tea Lemon     1.9  $3.80 / 1L
7                  Lipton Ice Tea Light Peach     3.9  $2.60 / 1L
8                  Lipton Light Peach Ice Tea     1.9  $3.80 / 1L
9                    Lipton Ice Tea Raspberry     3.9  $2.60 / 1L
10                   Lipton Ice Tea Raspberry     1.9  $3.80 / 1L
11                       Lipton Ice Tea Peach     3.9  $2.60 / 1L
12                       Lipton Peach Ice Tea     1.9  $3.80 / 1L
13                       Lipton Green Ice Tea     1.9  $3.80 / 1L
14                    Arizona Iced Lemon Tea      2.0  $4.00 / 1L
15                       Lipton Ice Tea Peach     2.7  $7.71 / 1L
16              Lipton Ice Green Tea Original     3.9  $2.60 / 1L
17                       Lipton Ice Tea Mango     3.9  $2.60 / 1L
18  Lipton Ice Tea Matcha Ginger & Lemongrass     3.9  $2.60 / 1L
19                         Fuze Ice Tea Mango     2.1  $1.68 / 1L
20                         Fuze Ice Tea Peach     2.1  $1.68 / 1L