我正在使用beautifulSoup进行网页抓取。我设法抓取了名字,但是问题是,我真的不确定如果数据在元素中(例如,电话号码和电子邮件位于以下图片中),该如何抓取:
我的代码:
import requests
from bs4 import BeautifulSoup
raw = requests.get('https://www.iproperty.com.my/property/findanagent.aspx?ty=as&ak=&rk=&pg=1&rmp=10&st=KL&ct=&st1=&ct1=#40091').text
raw = raw.replace("</br>", "")
soup = BeautifulSoup(raw, 'html.parser')
import re
phone = ['data-content'])[0][1:][:-1] for d in soup.find_all('a',{'class':'csagentphonelead'})]
name = [x.text.strip().split("\r\n")[-1].strip() for x in soup.find_all("p", class_='box-listing_agentCS')]
website = [x.text.strip().split("\r\n")[-1].strip() for x in soup.find_all("a", class_='csagentemaillead')]
num_page_items = len(name)
with open('results180.csv', 'a') as f:
for i in range(num_page_items):
f.write(name[i] + "," + phone[i] + "," + website[i] + "," + "\n")
我的抓取结果是“点击电子邮件”和“点击电话”。 我应该如何解决才能使结果正确的电子邮件和电话号码?
答案 0 :(得分:1)
您必须从链接获取data属性的值。您可以尝试使用此代码-
import requests
from bs4 import BeautifulSoup
raw = requests.get('https://www.iproperty.com.my/property/findanagent.aspx?ty=as&ak=&rk=&pg=1&rmp=10&st=KL&ct=&st1=&ct1=#40091').text
raw = raw.replace("</br>", "")
soup = BeautifulSoup(raw, 'html.parser')
import re
#['data-content'])[0][1:][:-1] ## note sure what is this
# for d in soup.find_all('a',{'class':'csagentphonelead'}):
name = [x.text.strip().split("\r\n")[-1].strip() for x in soup.find_all("p", class_='box-listing_agentCS')]
phone = [x['data'].strip().split("\r\n")[-1].strip() for x in soup.find_all("a", class_='csagentphonelead')]
website = [x['data'].strip().split("\r\n")[-1].strip() for x in soup.find_all("a", class_='csagentemaillead')]
num_page_items = len(name)
with open('results180.csv', 'a') as f:
for i in range(num_page_items):
f.write(name[i] + "," + phone[i] + "," + website[i] + "," + "\n")