Question

我试图在以下链接中打印地名（邪教，银行头）

https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN#results

我已经编写了以下代码，但是当我运行它时，它会认为有一段时间会生成一个空白输出，cmd会提示我再次输入。我最后尝试了两种不同的打印状态，我在第一次打印前放了一个＃任何想法，谢谢和此致

import requests
from bs4 import BeautifulSoup
import csv

# connection header
header={'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, sdch, br',
'Accept-Language':'en-US,en;q=0.8,fr;q=0.6',
'Connection':'keep-alive',
'Cookie':'mdtp=y4Ts2Vvql5V9MMZNjqB9T+7S/vkQKPqjHHMIq5jk0J1l5l131dU0YXsq7Rr15GDyghKHrS/qcD2vdsMCVtzKByJEDZFI+roS6tN9FN5IS70q8PkCCBjgFPDZjlR1A3H9FJ/zCWXMNJbaXqF8MgqE+nhR3/lji+eK4mm/GP9b8oxlVdupo9KN9SKanxu/JFEyNXutjyN+BsxRztNem1Z+ExSQCojyxflI/tc70+bXAu3/ppdP7fIXixfEOAWezmOh3ywchn9DV7Af8wH45t8u4+Y=; mdtpdi=mdtpdi#f523cd04-e09e-48bc-9977-73f974d50cea#1484041095424_zXDAuNhEkKdpRUsfXt+/1g==; seen_cookie_message=yes; _ga=GA1.4.666959744.1484041122; _gat=1',
'Host':'https://www.saa.gov.uk/',
'Referer':'https://www.saa.gov.uk/',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.51 Safari/537.36'
}

session = requests.session()

user_agent = {'User-Agent': 'Mozilla/5.0'}
url = 'https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN#results'



response = session.get(url, headers=header)


soup = BeautifulSoup(response.content,"lxml")

for link in soup.findAll('a'):
    #print link.get('href')
    print link.find('a')['href']

Answer 1

此网页没有硬性限制，因此简单的requests.get会执行抓取工作，对于BeautifulSoup部分，您要查找的地名和链接位于{{1}没有额外属性的标签，因此CSS selector更适合解析。

所以在这种情况下：

td

结果：

import requests
from bs4 import BeautifulSoup as soup

url = 'https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN#results'
baseurl = 'https://www.saa.gov.uk'

response = requests.get(url)
html = soup(response.text, 'lxml')

for link in html.select('td a'):
    print link.text, baseurl + link['href']

修改

如果您只需要查找第一个表中的所有链接，您只需要先找到第一个表... ... BOYD ORR CLOSE https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+ABERDEEN%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=BOYD+ORR+CLOSE%2C+ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN&DD_STREET=BOYD+ORR+CLOSE#results DON TERRACE https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+ABERDEEN%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=DON+TERRACE%2C+ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN&DD_STREET=DON+TERRACE#results BYRON CRESCENT https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+ABERDEEN%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=BYRON+CRESCENT%2C+ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN&DD_STREET=BYRON+CRESCENT#results MARINE TERRACE https://www.saa.gov.uk/search.php?SEARCHED=1&ST=&SEARCH_TERM=aberdeen%2C+ABERDEEN%2C+Aberdeen+City&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&x=0&y=0&PAGE=0&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&ORIGINAL_SEARCH_TERM=aberdeen&DRILL_SEARCH_TERM=MARINE+TERRACE%2C+ABERDEEN%2C+Aberdeen+City&DD_UNITARY_AUTHORITY=Aberdeen+City&DD_TOWN=ABERDEEN&DD_STREET=MARINE+TERRACE#results ... ...，然后在此表中搜索链接。

first_table = html.find('table')

无法在python中提取url链接

1 个答案: