我试图从此网站获取手机/办公室电话号码信息:https://www.zillow.com/lender-profile/DougShoemaker/
ive尝试使用bs4,但我只能获得第一个电话号码。我正在尝试获取办公室和手机号码。
from selenium import webdriver
from bs4 import BeautifulSoup
import time
#Chrome webdriver filepath...Chromedriver version 74
driver = webdriver.Chrome(r'C:\Users\mfoytlin\Desktop\chromedriver.exe')
driver.get('https://www.zillow.com/lender-profile/DougShoemaker/')
soup = BeautifulSoup(driver.page_source, 'html.parser')
time.sleep(2)
phoneNum = driver.find_element_by_class_name('zsg-list_definition')
trial = phoneNum.find_element_by_class_name('zsg-sm-hide')
print(trial.text)
答案 0 :(得分:2)
您不必使用Selenium甚至BeautifulSoup。如果您检查来自Developer Tools (F12) > Network
的网络请求,您会看到使用XHR请求获取了数据
您可以自己发出此请求,并随时使用JSON响应。
POST https://mortgageapi.zillow.com/getRegisteredLender?partnerId=RD-CZMBMCZ
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
Referer: https://www.zillow.com/lender-profile/DougShoemaker/
Content-Type: application/json
{
"fields": [
"aboutMe",
"address",
"cellPhone",
# ... other fields
"website"
],
"lenderRef": {
"screenName": "DougShoemaker"
}
}
现在,使用requests
库,您可以尝试:
import requests
if __name__ == '__main__':
payload = {
"fields": [
"screenName",
"cellPhone",
"officePhone",
"title",
],
"lenderRef": {
"screenName": "DougShoemaker"
}
}
res = requests.post('https://mortgageapi.zillow.com/getRegisteredLender?partnerId=RD-CZMBMCZ',
json=payload)
res.raise_for_status()
data = res.json()
cellphone, office_phone = data['lender']['cellPhone'], data['lender']['officePhone']
cellphone_num = '({areaCode}) {prefix}-{number}'.format(**cellphone)
office_phone_num = '({areaCode}) {prefix}-{number}'.format(**office_phone)
print(office_phone_num, cellphone_num)
打印:
(618) 619-4120 (618) 795-0790
答案 1 :(得分:0)
尝试遵循每个电话号码的xpath
Office Phone:
//dt[contains(text(),'Office')]/following-sibling::dd/div/span
Cell Phone:
//dt[contains(text(),'Cell')]/following-sibling::dd/div/span
Fax Number:
//dt[contains(text(),'Fax')]/following-sibling::dd/div/span
答案 2 :(得分:0)
要提取 Office ,手机和传真编号,您必须为{{生成 WebDriverWait 1}},则可以使用以下任何Locator Strategies:
代码块:
visibility_of_element_located()
控制台输出:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
# options.add_argument('disable-infobars')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.zillow.com/lender-profile/DougShoemaker/')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Office']//following::dd[1]//span"))).get_attribute("innerHTML"))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Cell']//following::dd[1]//span"))).get_attribute("innerHTML"))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Fax']//following::dd[1]//span"))).get_attribute("innerHTML"))