Question

<div class="columns small-5 medium-4 cell header">Ref No.</div>
<div class="columns small-7 medium-8 cell">110B60329</div>

网站为https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results

我想运行循环并返回'110B60329'。我已经运行了漂亮的汤并完成了find_all（div），然后我根据他们的类定义了2个不同的标签作为头部和数据。然后我通过'head'标签进行迭代，希望它能在我定义为数据的div标签中返回信息。

Python返回一个空白（cmd提示重新打印了filepth）。

有人会知道我怎么解决这个问题。我的完整代码是.....谢谢

import requests
from bs4 import BeautifulSoup as soup
import csv


url = 'https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results'

baseurl = 'https://www.saa.gov.uk'

session = requests.session()

response = session.get(url)

# content of search page in soup 
html= soup(response.content,"lxml")   
properties_col = html.find_all('div')



for col in properties_col:
    ref = 'n/a'
    des = 'n/a'

    head = col.find_all("div",{"class": "columns small-5 medium-4 cell header"}) 

    data = col.find_all("div",{"class":"columns small-7 medium-8 cell"})

    for i,elem in enumerate(head):
    #for i in range(elems):
        if head [i].text == "Ref No.":
            ref = data[i].text
            print ref

Answer 1

你可以通过两种方式做到这一点。

1）如果您确定您正在抓取的网站不会更改其内容，您可以找到该类的所有div并通过提供索引来获取内容。

2）查找所有左侧div（标题），如果其中一个匹配您想要的下一个兄弟来获取文本。

示例：

import requests
from bs4 import BeautifulSoup as soup

url = 'https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results'

baseurl = 'https://www.saa.gov.uk'

session = requests.session()

response = session.get(url)

# content of search page in soup 
html = soup(response.content,"lxml")

#Method 1
LeftBlockData = html.find_all("div", class_="columns small-7 medium-8 cell")
Reference = LeftBlockData[0].get_text().strip()
Description = LeftBlockData[2].get_text().strip()
print(Reference)
print(Description)

#Method 2
for column in html.find_all("div", class_="columns small-5 medium-4 cell header"):
    RightColumn = column.next_sibling.next_sibling.get_text().strip()
    if "Ref No." in column.get_text().strip():
        print (RightColumn)
    if "Description" in column.get_text().strip():
        print (RightColumn)

将打印出打印件（按顺序）：

110B60329

STORE

110B60329

STORE

您的问题是您正在尝试将具有大量标签的节点文本与非间隔字符串进行匹配。

例如，您的head [i].text变量包含Ref No.，因此如果您将其与Ref No.进行比较，则会给出错误的结果。划线它将解决。

Answer 2

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results")
soup = BeautifulSoup(r.text, 'lxml')
for row in soup.find_all(class_='table-row'):

    print(row.get_text(strip=True, separator='|').split('|'))

出：

['Ref No.', '110B60329']
['Office', 'LOTHIAN VJB']
['Description', 'STORE']
['Property Address', '29 BOSWALL PARKWAY', 'EDINBURGH', 'EH5 2BR']
['Proprietor', 'SCOTTISH MIDLAND CO-OP SOCIETY LTD.']
['Tenant', 'PROPRIETOR']
['Occupier']
['Net Annual Value', '£1,750']
['Marker']
['Rateable Value', '£1,750']
['Effective Date', '01-APR-10']
['Other Appeal', 'NO']
['Reval Appeal', 'NO']

get_text()是非常强大的工具，你可以剥离空格并在文本中放置分隔符。

您可以使用此方法获取干净的数据并对其进行过滤。

Findall to div tag使用漂亮的汤产生空白回报

2 个答案: