Question

我无法从单个标签获取数据。单个标签具有许多数据属性，例如名称，电话，公司和url。我需要从许多标签以及与此标签相似的所有标签中获取数据。

HTML代码：

datasource

Python代码：

 <div class="ListingDetails">
        <div class="ListingDisplayName">
            <a href="/members/jeremy.counter1/default.aspx">
                Jeremy Counter
            </a> 
        </div>
                                        Mortgage Officer&nbsp; - 
       &nbsp;American Pacific Mortgage<br>                                            

       Anchorage,&nbsp;Alaska&nbsp;&nbsp;99503<br>
                                        phone:&nbsp;(907) 519- 
       6656&nbsp;|&nbsp;(907) 250-0766

      <div class="listingurl">
          <a rel="nofollow" href="http://www.jeremycounter.com" target="_blank">
            jeremycounter.com
          </a>
      </div>
 </div>

输出：

url = "http://www.mortgagenewsdaily.com/directory/mortgage/alabama"
data=requests.get(url)
soup=bs4.BeautifulSoup(data.text,'html.parser')

page = soup.find('div', class_="CommonContentBox DirectoryListings")

listing_box = page.find('div', class_="BusinessListingUser") 
name  = listing_box.find('div', class_="ListingDisplayName").text
#print(name)

details  = listing_box.find('div', 
class_="ListingDetails").text.strip('\n\t\r')
print(details)

任何人都告诉我获取数据的最佳解决方案。

预期结果：

Tyler Tullis
                                          -                                              
                                        Montgomery, Alabama  36117
                                         | (334) 322-3707

Answer 1

您可以使用硒来完成此任务：

.delete

输出：

from selenium import webdriver
import os

# setup path to chrome driver
chrome_driver = os.getcwd() + '/chromedriver'
# initialise chrome driver
browser = webdriver.Chrome(chrome_driver)

# load url
url = 'http://www.mortgagenewsdaily.com/directory/mortgage/alabama'
browser.get(url)
# find all elements
content = browser.find_elements_by_xpath('//*[@id="CommonContentInner"]/div/div/div/div/div')

# get text from each element
collected_data = []
for item in content:
    personal_data = item.get_attribute("innerText")
    collected_data.append(personal_data)

# clean list
collected_data = filter(None, collected_data)

Answer 2

基于您的html，您可以尝试以下代码。

from bs4 import BeautifulSoup
data='''<div class="ListingDetails">
        <div class="ListingDisplayName">
            <a href="/members/jeremy.counter1/default.aspx">
                Jeremy Counter
            </a> 
        </div>
                                        Mortgage Officer&nbsp; - 
       &nbsp;American Pacific Mortgage<br>                                            

       Anchorage,&nbsp;Alaska&nbsp;&nbsp;99503<br>
                                        phone:&nbsp;(907) 519- 
       6656&nbsp;|&nbsp;(907) 250-0766

      <div class="listingurl">
          <a rel="nofollow" href="http://www.jeremycounter.com" target="_blank">
            jeremycounter.com
          </a>
      </div>
 </div>'''


soup=BeautifulSoup(data,'html.parser')

items= soup.find_all('div', class_="ListingDetails")

for item in items:
    print("name: " + item.find('a').text.strip())
    print('company: ' + item.find_all('br')[0].previous_element.strip().split('-')[1].strip())
    print('Phone: ' + item.find_all('br')[1].next_element.strip().split('|')[0].strip())
    print('url: ' + item.find('div',class_='listingurl').find('a').text.strip())

输出：

name: Jeremy Counter
company: American Pacific Mortgage
Phone: phone: (907) 519- 
       6656
url: jeremycounter.com

Answer 3

这里不需要硒。只需提取数据并对其进行遍历以清理并打印即可：

import requests
import bs4

url = "http://www.mortgagenewsdaily.com/directory/mortgage/alabama"
data=requests.get(url)
soup=bs4.BeautifulSoup(data.text,'html.parser')

page = soup.find_all('div', class_="BusinessListingUser")

for each in page:
    content = each.find('div', class_='ListingDetails').text.split('\n')
    content = [ text.strip() for text in content if text.strip() != '' ]
    for strings in content:
        print (strings)
    print ('\n')

输出：

Tyler Tullis
-
Montgomery, Alabama  36117
| (334) 322-3707


Nathan Stotlar
Mortgage Production Manager  -  PrimeLending, a PlainsCapital Company
Fitchburg, Wisconsin  53717
phone: (608) 467-4249
nathanstotlar.com


Anna  Mendonca
Mortgage Loan Originator   -  CrossCountry Mortgage, Inc
Wakefield , Massachusetts  01880
phone: (781) 618-3154 | (781) 290-6383
myccmhomeloan.com/Default.aspx


Pouyan Broukhim
Owner  -  Probate Funding, Inc.
Los Angeles, California  90048
phone: (323) 935-5577
probatefunding.com

...

附加：

import requests
import bs4
import pandas as pd

url = "http://www.mortgagenewsdaily.com/directory/mortgage/alabama"
data=requests.get(url)
soup=bs4.BeautifulSoup(data.text,'html.parser')

page = soup.find_all('div', class_="BusinessListingUser")

results = pd.DataFrame()
for each in page:
    content = each.find('div', class_='ListingDetails').text.split('\n')
    content = [ text.strip() for text in content if text.strip() != '' ]

    try:
        name = content[0]
    except:
        name = 'N/A'

    try:
        company = content[1]
    except:
        company = 'N/A'

    try:
        location = content[2]
    except:
        location = 'N/A'

    try:
        phone = content[3]
    except:
        phone = 'N/A'

    try:
        website = content[4]
    except:
        website = 'N/A'

    temp_df = pd.DataFrame([[name,company,location,phone,website]], columns = ['name','company','location','phone','website'])
    results = results.append(temp_df).reset_index(drop=True)

results.to_excel('C:/file.xlsx', index=False)

如何从单个标签获取许多属性数据？

3 个答案: