我正在使用 html 请求和漂亮的汤来开发网络爬虫（我是新手）。对于 1 个网页 (https://www.selfridges.com/GB/en/cat/beauty/make-up/?pn=1)，我试图抓取产品网格中每个产品的链接。我曾尝试使用 absolute_links 和 xpath：
session = HTMLSession()
for x in range(1, 30):
url = f'https://www.selfridges.com/GB/en/cat/beauty/make-up/?pn={x}'
r = session.get(url)
r.html.render(sleep=2)
products = r.html.xpath('//*[@id="content"]/div[3]/div/div/div/div[2]/div[1]/div[2]/div[6]/div/div/div[1]/div/div/div/div[2]', first=True)
productlist = products.absolute_links
productlinks.extend(productlist)
print(productlinks)
和 BeautifulSoup：
session = HTMLSession()
for x in range(1, 30):
url = f'https://www.selfridges.com/GB/en/cat/beauty/make-up/?pn={x}'
r = session.get(url)
r.html.render(sleep=2)
soup = BeautifulSoup(r.content, 'lxml')
productlist = r.html.find('div', class_="listing-items c-listing-items initialized")
print(productlist)
for item in productlist:
for link in item.find_all('a', href=True):
productlinks.append(baseurl + link['href'])
print(productlinks)
两者都返回空列表或 AttributeError: 'NoneType' object has no attribute 'absolute_links'。我不确定为什么会发生这种情况。任何帮助将不胜感激。
答案 0 :(得分：1)
要获取所有链接，请使用 CSS 类“c-prod-card__cta-box-link-mask”。另外，请确保您没有获得 Cloudflare 验证码页面（使用 User-Agent HTTP 标头）：
import requests
from bs4 import BeautifulSoup
url = "https://www.selfridges.com/GB/en/cat/beauty/make-up/?pn=1"
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
for a in soup.select("a.c-prod-card__cta-box-link-mask"):
print("https://www.selfridges.com" + a["href"])
打印：
https://www.selfridges.com/GB/en/cat/guerlain-kisskiss-tender-matte-lipstick-28g_R03746795/?previewAttribute=530+Dreamy+Rose
https://www.selfridges.com/GB/en/cat/hermes-rose-herms-silky-blush-6g_R03752945/?previewAttribute=32+Rose+Pommette
https://www.selfridges.com/GB/en/cat/hermes-rosy-lip-enhancer-refill-4g_R03752948/?previewAttribute=14+Rose+Abricote
https://www.selfridges.com/GB/en/cat/pat-mcgrath-labs-liquilust-legendary-wear-matte-lipstick-5ml_R03761316/?previewAttribute=Elson+4
https://www.selfridges.com/GB/en/cat/charlotte-tilbury-glowgasm-beauty-light-wand-12ml_455-3003231-FGLWX12X1R/?previewAttribute=Goldgasm
https://www.selfridges.com/GB/en/cat/giorgio-armani-luminous-silk-foundation-30ml_317-77011643-LUMSILKFDT/?previewAttribute=3.5
https://www.selfridges.com/GB/en/cat/nars-radiant-creamy-concealer-6ml_318-2000192-CREAMYCON/?previewAttribute=CHANTILLY
https://www.selfridges.com/GB/en/cat/charlotte-tilbury-hollywood-contour-wand-12g_455-3003231-FHCWX12X1R22/?previewAttribute=FAIR+MEDIUM
https://www.selfridges.com/GB/en/cat/huda-beauty-baby-bake-loose-powder-6g_R03760466/?previewAttribute=Pound+Cake
https://www.selfridges.com/GB/en/cat/charlotte-tilbury-brow-fix-sculpting-gel-6g_R03757642/?previewAttribute=CLEAR
https://www.selfridges.com/GB/en/cat/dior-dior-forever-natural-bronze-powder-9g_R03748648/?previewAttribute=005
https://www.selfridges.com/GB/en/cat/byredo-dysco-eyeshadow-5-colours-palette-6g_R03753732/?previewAttribute=Dysco
https://www.selfridges.com/GB/en/cat/anastasia-beverly-hills-no-fade-brow-kit_R03765470/?previewAttribute=TAUPE
https://www.selfridges.com/GB/en/cat/hermes-rouge-hermes-lip-brush_R03677495/
https://www.selfridges.com/GB/en/cat/huda-beauty-brown-obsessions-eyeshadow-palette-75g_R03760181/?previewAttribute=TOFFEE
https://www.selfridges.com/GB/en/cat/gucci-gucci-rouge-de-beaut-brillant-lipstick-18g_R03750148/?previewAttribute=714
https://www.selfridges.com/GB/en/cat/westman-atelier-lit-up-highlight-stick-5g_R03728001/?previewAttribute=Lit
https://www.selfridges.com/GB/en/cat/dior-rouge-dior-vernis-nail-polish-10ml_R03748652/?previewAttribute=323+Dune
https://www.selfridges.com/GB/en/cat/bobbi-brown-luxe-defining-lipstick-6ml_R03751332/?previewAttribute=TERRACOTTA
https://www.selfridges.com/GB/en/cat/bobbi-brown-long-wear-line-and-define-eye-kit-worth-6375_R03754304/
https://www.selfridges.com/GB/en/cat/huda-beauty-brown-obsessions-liquid-matte-mini-gift-set_R03760182/?previewAttribute=BROWNS
https://www.selfridges.com/GB/en/cat/huda-beauty-power-bullet-cream-glow-sweet-nude-lipstick-3g_R03760463/?previewAttribute=Hahibi
https://www.selfridges.com/GB/en/cat/pat-mcgrath-labs-permagel-ultra-glide-eye-pencil-12g_R03739396/?previewAttribute=Blk+Coffee
https://www.selfridges.com/GB/en/cat/nars-powermatte-lip-pigment_318-2000192-2760/?previewAttribute=SLOW+RIDE
https://www.selfridges.com/GB/en/cat/charlotte-tilbury-hyaluronic-happikiss-lipstick-gloss-balm-25g_R03734145/?previewAttribute=PILLOW+TALK
https://www.selfridges.com/GB/en/cat/mac-studio-radiance-face-and-body-radiant-sheer-foundation-50ml_R03751324/?previewAttribute=N1
https://www.selfridges.com/GB/en/cat/benefit-gimme-brow-jumbo-volumising-eyebrow-gel-6g_R03755671/?previewAttribute=05
https://www.selfridges.com/GB/en/cat/anastasia-beverly-hills-eye-brag-eyeliner-and-mascara-kit_R03746282/
https://www.selfridges.com/GB/en/cat/tom-ford-eye-colour-quad-6g_R03731171/?previewAttribute=Desert+Fox
https://www.selfridges.com/GB/en/cat/laura-mercier-secret-camouflage-concealer-duo-1g_R03745454/?previewAttribute=1N
https://www.selfridges.com/GB/en/cat/westman-atelier-lip-suede-lip-colour-48g_R03728007/?previewAttribute=Les+Rouges
https://www.selfridges.com/GB/en/cat/pat-mcgrath-labs-skin-fetish-ultra-glow-highlighter-10g_R03654289/?previewAttribute=Divine+Rose
https://www.selfridges.com/GB/en/cat/nars-natural-radiant-longwear-foundation_318-2000192-6601/?previewAttribute=OSLO
https://www.selfridges.com/GB/en/cat/hermes-rosy-lip-enhancer-6g_R03752946/?previewAttribute=14+Rose+Abricote
https://www.selfridges.com/GB/en/cat/byredo-practical-brown-eyeliner-17ml_R03753733/?previewAttribute=Practical+Brown
https://www.selfridges.com/GB/en/cat/bobbi-brown-pretty-powerful-limited-edition-gift-set-worth-69_R03740437/
https://www.selfridges.com/GB/en/cat/urban-decay-naked-wild-west-eyeshadow-palette-114g_R03736731/
https://www.selfridges.com/GB/en/cat/mac-mac-x-harris-reed-fighting-for-the-beauty-of-fluidity-eyeshadow-palette-585g_R03742192/
https://www.selfridges.com/GB/en/cat/mac-mac-x-harris-reed-embrace-your-duality-palette-44g_R03742191/
https://www.selfridges.com/GB/en/cat/gucci-crayon-contour-des-lvres-lip-liner-105g_R03743420/?previewAttribute=001+Nude
https://www.selfridges.com/GB/en/cat/milk-makeup-hydro-grip-set-refresh-setting-spray-100ml_R03652449/
https://www.selfridges.com/GB/en/cat/the-ordinary-high-coverage-concealer-8ml_R03737083/?previewAttribute=1.0+N
https://www.selfridges.com/GB/en/cat/mac-strip-down-lipglass-3g_329-81004873-S3HT5G0000/?previewAttribute=LOVE+NECTAR
https://www.selfridges.com/GB/en/cat/tom-ford-translucent-finishing-powder-9g_450-3001058-TRANSLUCENTFINISHINGPOWDER/?previewAttribute=ALABASTER+NUDE
https://www.selfridges.com/GB/en/cat/dior-rouge-dior-couture-colour-refillable-lipstick-collection-gift-set_R03722732/
https://www.selfridges.com/GB/en/cat/lancome-lash-idle-mascara-9ml_R03731596/?previewAttribute=01
https://www.selfridges.com/GB/en/cat/charlotte-tilbury-charlottes-magic-icons-gift-set-worth-142_R03719702/
https://www.selfridges.com/GB/en/cat/tom-ford-traceless-foundation-stick_450-3001058-TRACELESSFNDTNSTICK/?previewAttribute=FAWN
https://www.selfridges.com/GB/en/cat/suqqu-eyelash-waterproof-mascara_396-3000994-63566/?previewAttribute=01+BLACK
https://www.selfridges.com/GB/en/cat/charlotte-tilbury-the-pillow-talk-look-set-worth-175_R03639616/?previewAttribute=V2
https://www.selfridges.com/GB/en/cat/suqqu-glow-powder-foundation-compact_R03741601/
https://www.selfridges.com/GB/en/cat/suqqu-nuance-liquid-eyeliner-035ml_R03732473/?previewAttribute=01+BLACK
https://www.selfridges.com/GB/en/cat/dior-addict-lip-maximizer-6ml_359-84011246-C006500001/?previewAttribute=010
https://www.selfridges.com/GB/en/cat/hourglass-confessions-ultra-slim-high-intensity-refillable-lipstick-09g_R03745528/?previewAttribute=Red0
https://www.selfridges.com/GB/en/cat/nars-euphoria-afterglow-lip-shine-lip-gloss-6ml_R03733341/?previewAttribute=Deep+Realm
https://www.selfridges.com/GB/en/cat/lancome-hypnocircse-customndashwear-volume-mascara_340-84010339-HYPNOSE/?previewAttribute=NOIR
https://www.selfridges.com/GB/en/cat/dior-5-couleurs-couture-summer-dune-collection-limited-edition-eyeshadow-palette-4g_R03748650/?previewAttribute=699+Mirage
https://www.selfridges.com/GB/en/cat/dior-rouge-dior-forever-liquid-lipstick-6ml_R03748653/?previewAttribute=741+Forever+Star
https://www.selfridges.com/GB/en/cat/urban-decay-all-nighter-setting-spray-118ml_367-3003701-S2385800/
https://www.selfridges.com/GB/en/cat/hermes-rosy-lip-enhancer-6g_R03752946/?previewAttribute=49+Rose+Tan