Question

我使用BeautifulSoup抓取了一个网页，分配给了汤＃39;我可以得到文字＆＃39; Aberdeen＆＃39;只需将.text添加到＆＃39; site_url＆＃39;的末尾即可。

我真正想要的是字符串中的完整网址，例如＆＃34; http://www.somewebsite.com/networks/site-info?site_id=ABD＆＃34;

>>>site_link = soup.find_all('a', string='Aberdeen')[0]
>>>site_row = site_link.findParent('td').findParent('tr')
>>>site_column = site_row.findAll('td')
>>>site_url = site_column[0].contents[0]
>>>print(site_url)

<a href="../networks/site-info?site_id=ABD">Aberdeen</a>

到目前为止我没有运气，也不知道还有什么可以尝试的。我怎样才能获得网址？

Answer 1

您可以使用正则表达式来获取使用urljoin获取正确URL的链接。

import requests
import re

try:
    from urlparse import urljoin  # Python2
except ImportError:
    from urllib.parse import urljoin  # Python3

from bs4 import BeautifulSoup
url= 'https://uk-air.defra.gov.uk/latest/currentlevels'
r = requests.get(url, headers={'User-Agent': 'Not blank'})
data = r.text
soup = BeautifulSoup(data, 'html.parser')
for elem in soup('a', href=re.compile(r'site_id')):
    print (elem.text)
    print (urljoin(url,elem['href']))

输出：

Auchencorth Moss
https://uk-air.defra.gov.uk/networks/site-info?site_id=ACTH
Bush Estate
https://uk-air.defra.gov.uk/networks/site-info?site_id=BUSH
Dumbarton Roadside
https://uk-air.defra.gov.uk/networks/site-info?site_id=DUMB
Edinburgh St Leonards
https://uk-air.defra.gov.uk/networks/site-info?site_id=ED3
Glasgow Great Western Road
https://uk-air.defra.gov.uk/networks/site-info?site_id=GGWR
Glasgow High Street
https://uk-air.defra.gov.uk/networks/site-info?site_id=GHSR
...

如果您只想使用Aberdeen：

for elem in soup('a',href=re.compile(r'site_id'), string='Aberdeen'):

而不是：

for elem in soup('a', href=re.compile(r'site_id')):

输出：

Aberdeen
https://uk-air.defra.gov.uk/networks/site-info?site_id=ABD

Answer 2

试试这个。我希望它能满足你的所有要求：

import requests ; from lxml import html

base_link = "https://uk-air.defra.gov.uk"
response = requests.get("https://uk-air.defra.gov.uk/latest/currentlevels", headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36'}).text
tree = html.fromstring(response)
for title in tree.cssselect("table.current_levels_table td a:not(.smalltext)"):
    print(base_link + title.attrib['href'][2:])

部分结果：

https://uk-air.defra.gov.uk/networks/site-info?site_id=ACTH
https://uk-air.defra.gov.uk/networks/site-info?site_id=BUSH
https://uk-air.defra.gov.uk/networks/site-info?site_id=DUMB
https://uk-air.defra.gov.uk/networks/site-info?site_id=ED3
https://uk-air.defra.gov.uk/networks/site-info?site_id=GGWR
https://uk-air.defra.gov.uk/networks/site-info?site_id=GHSR
https://uk-air.defra.gov.uk/networks/site-info?site_id=GLA4

使用Python中的BeautifulSoup从超链接获取URL

2 个答案: