Question

我正在尝试从亚马逊列表中删除标题。但是似乎我的代理中存在一些错误。这是我尝试过的代码：

import requests
from bs4 import BeautifulSoup
url="https://www.amazon.com/Kindle-Paperwhite-Essentials-Bundle-including/dp/B0898ZK226/ref=sr_1_3?dchild=1&keywords=kindle&qid=1597051671&sr=8-3"

headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}
proxyDict = { 
             "http": "http://10.10.1.10:3128",
             "https": "http://10.10.1.10:1080",
             "ftp"   : "ftp://10.10.1.10:3128"
           }
page = requests.get(url, headers=headers, proxies=proxyDict) #Error here.
print(page.status_code)
soup = BeautifulSoup(page.content, "html.parser")
title = soup.find(id = "productTitle")
if title:
   title = title.get_text().strip()
else:
   title = "Title: Error 404"
print(title)

没有代理，代码在输出中显示无。

Answer 1

先前的代码无法删除JavaScript。这是正确的代码：

from bs4 import BeautifulSoup
import requests
from requests_html import HTMLSession
# create an HTML Session object
session = HTMLSession()
# Use the object above to connect to needed webpage
resp = session.get("https://www.amazon.com/Sceptre-E248W-19203R-Monitor-Speakers-Metallic/dp/B0773ZY26F/ref=sr_1_2?crid=1861TM8A5NDPX&dchild=1&keywords=monitors&qid=1597071906&sprefix=monitors%2Caps%2C364&sr=8-2")
# Run JavaScript code on webpage
resp.html.render()
soup = BeautifulSoup(resp.html.html, "lxml")
title = soup.find(id = "productTitle").get_text().strip()
print (title)

使用Beautiful Soup和Proxy从Amazon抓取数据

1 个答案: