BeautifulSoup：获取特定类别的所有产品链接

Question

我希望通过在Python中使用BeautifulSoup来获取特定类别的所有产品链接。

我尝试过以下但未获得结果：

import lxml
import urllib2
from bs4 import BeautifulSoup
html=urllib2.urlopen("http://www.bedbathandbeyond.com/store/category/bedding/bedding/quilts-coverlets/12018/1-96?pagSortOpt=DEFAULT-0&view=grid")

br= BeautifulSoup(html.read(),'lxml')

for links in br.findAll('a', class_='prodImg'):
    print links['href']

Answer 1

你使用urllib2错误。

import lxml
import urllib2
from bs4 import BeautifulSoup

#create a http request
req=urllib2.Request("http://www.bedbathandbeyond.com/store/category/bedding/bedding/quilts-coverlets/12018/1-96?pagSortOpt=DEFAULT-0&view=grid")
# send the request
response = urllib2.urlopen(req)
# read the content of the response
html = response.read()
br= BeautifulSoup(html,'lxml')

for links in br.findAll('a', class_='prodImg'):
    print links['href']

Answer 2

from bs4 import BeautifulSoup
import requests

html=requests.get("http://www.bedbathandbeyond.com/store/category/bedding/bedding/quilts-coverlets/12018/1-96?pagSortOpt=DEFAULT-0&view=grid")

br= BeautifulSoup(html.content,"lxml")

data=br.findAll('div',attrs={'class':'productShadow'})
for div in br.find_all('a'):
        print div.get('href')

试试这段代码