刮亚马逊产品名称

时间:2019-03-20 19:26:15

标签: python web-scraping beautifulsoup

我正在尝试根据卖家名称在亚马逊上收集前两页产品名称。当我请求页面时,它具有我需要的所有元素,但是,当我使用BeautifulSoup时-它们并未列出。这是我的代码:

import requests
from bs4 import BeautifulSoup
headers = {'User-Agent':'Mozilla/5.0'}
res = requests.get("https://www.amazon.com/s?me=A3WE363L17WQR&marketplaceID=ATVPDKIKX0DER", headers=headers)
#print(res.text)
soup = BeautifulSoup(res.text, "html.parser")
soup.find_all("a",href=True)

未列出产品的链接。如果Amazon API提供了此信息,我可以使用它(请提供一些用法示例)。提前谢谢。

1 个答案:

答案 0 :(得分:0)

我从alt属性中提取了产品名称。这是预期的吗?

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://www.amazon.com/s?me=A3WE363L17WQR&marketplaceID=ATVPDKIKX0DER')
soup = bs(r.content, 'lxml')
items = [item['alt'] for item in soup.select('.a-link-normal [alt]')]
print(items)

超过两页:

import requests
from bs4 import BeautifulSoup as bs
url = 'https://www.amazon.com/s?i=merchant-items&me=A3WE363L17WQR&page={}&marketplaceID=ATVPDKIKX0DER&qid=1553116056&ref=sr_pg_{}'
for page in range(1,3):
    r = requests.get(url.format(page,page))
    soup = bs(r.content, 'lxml')
    items = [item['alt'] for item in soup.select('.a-link-normal [alt]')]
    print(items)