Question

在工作中，我被要求创建一个电子表格，其中包含美国所有同种疗法医学院的名称和地址。我是python的新手，我认为这是尝试Web抓取的理想情况。当我最终编写了一个返回所需数据的程序时，我知道有一种更好的方法，因为我必须进入excel并手动删除一些多余的字符（例如：“，”，[]）。我只想知道是否有更好的方法可以编写这段代码，以便我得到所需的内容，减去多余的字符。

编辑：我还附加了一个csv文件的图像，该图像是为了显示我正在谈论的无关字符而创建的。

from bs4 import BeautifulSoup
import requests
import csv  

link = "https://members.aamc.org/eweb/DynamicPage.aspx?site=AAMC&webcode=AAMCOrgSearchResult&orgtype=Medical%20School" # noqa
# link to the site we want to scrape from

page_response = requests.get(link)
# fetching the content using the requests library

soup = BeautifulSoup(page_response.text, "html.parser")
# Calling BeautifulSoup in order to parse our document

data = []
# Empty list for the first scrape. We only get one column with many rows. 
# We still have the line break tags here </br>
for tr in soup.find_all('tr', {'valign': 'top'}):
    values = [td.get_text('</b>', strip=True) for td in tr.find_all('td')]
    data.append(values)

data2 = []
# New list that we'll use to have name on index i, address on index i+1
for i in data:
    test = list(str(i).split('</b>'))
    # Using the line breaks to our advantage. 
    name = test[0].strip("['")
    '''Here we are saying that the name of the school is the first element
       before the first line break'''

    addy = test[1:]
    # The address is what comes after this first line break
    data2.append(name)
    data2.append(addy)
    # Append the name of the school and address to our new list.

school_name = data2[::2]
# Making a new list that consists of the school name
school_address = data2[1::2]
# Another list that consists of the school's address.

with open("Medschooltest.csv", 'w', encoding='utf-8') as toWrite:
    writer = csv.writer(toWrite)
    writer.writerows(zip(school_name, school_address))
    '''Zip the two together making a 2 column table with the schools name and
       it's address'''

print("CSV Completed!")

Created CSV file

Answer 1

似乎将条件语句与字符串操作一起使用可以解决问题。我认为以下脚本将使您真正接近所需的内容。

from bs4 import BeautifulSoup
import requests
import csv

link = "https://members.aamc.org/eweb/DynamicPage.aspx?site=AAMC&webcode=AAMCOrgSearchResult&orgtype=Medical%20School" # noqa

res = requests.get(link)
soup = BeautifulSoup(res.text, "html.parser")

with open("membersInfo.csv","w",newline="") as infile:
    writer = csv.writer(infile)
    writer.writerow(["Name","Address"])

    for tr in soup.find_all('table', class_='bodyTXT'):
        items = ', '.join([item.string for item in tr.select_one('td') if item.string!="\n" and item.string!=None])
        name = items.split(",")[0].strip()
        address = items.split(name)[1].strip(",")
        writer.writerow([name,address])

Answer 2

如果您具有SQL知识并且数据是以这种结构化的方式进行的，那么将数据提取到数据库中将是最佳的解决方案。

有没有更好的方式来抓取这些数据？

2 个答案: