Question

如何将名称列表放入for循环中并搜索其许可证ID和有效期？

我要通过网络剪贴该网站“ http://mbsweblist.fsco.gov.on.ca/agents.aspx”。我有代理商的姓氏。使用姓氏搜索后，它会返回许可证ID，这是一个超链接单击，可将您带到其许可证信息（例如到期日期）的另一页。

我的输入将是.csv文件中的代理名称的列表，输出应是具有代理名称，许可证编号和到期日期的.csv或excel文件。我宁愿不使用硒。请让我知道是否有办法？以下代码一次搜索并返回一个代理的输出。谢谢

列表图片：enter image description here

import requests
from bs4 import BeautifulSoup

    def get_result_page_ontario(name):
    r = requests.post("http://mbsweblist.fsco.gov.on.ca/agents.aspx",
    data={
    'ctl00$ctl00$MainPlaceHolder$Content4$bkmbname:': 'crossley',
    '_EVENTTARGET': '',
    '__EVENTARGUMENT': '',
    '__LASTFOCUS': '',
    '__VIEWSTATE': '/wEPDwULLTEwMzk1Nzk2NDAPZBYCZg9kFgJmD2QWBAIBD2QWAgIBDxYCHgRUZXh0BcUHPGxpbmsgcmVsPSdzdHlsZXNoZWV0JyBocmVmPSdodHRwOi8vd3d3LmZzY28uZ292Lm9uLmNhLy9TdHlsZSBMaWJyYXJ5L0ZTQ08vSW50ZXJuZXQvQ1NTL21hc3Rlci5jc3MnIHR5cGU9J3RleHQvY3NzJyBtZWRpYT0nc2NyZWVuJyAvPjxsaW5rIHJlbD0nc3R5bGVzaGVldCcgaHJlZj0naHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vU3R5bGUgTGlicmFyeS9GU0NPL0ludGVybmV0L0NTUy9jaGFuZ2VtZS5jc3MnIHR5cGU9J3RleHQvY3NzJyBtZWRpYT0nc2NyZWVuJy8+PCEtLVtpZiBsdGUgSUUgNl0+PGxpbmsgcmVsPSdzdHlsZXNoZWV0JyBocmVmPSdodHRwOi8vd3d3LmZzY28uZ292Lm9uLmNhLy9TdHlsZSBMaWJyYXJ5L0ZTQ08vSW50ZXJuZXQvQ1NTL2llNi5jc3MnIHR5cGU9J3RleHQvY3NzJyBtZWRpYT0nc2NyZWVuJyAvPiA8IVtlbmRpZl0tLT48bGluayByZWw9J3N0eWxlc2hlZXQnIGhyZWY9J2h0dHA6Ly93d3cuZnNjby5nb3Yub24uY2EvL1N0eWxlIExpYnJhcnkvRlNDTy9JbnRlcm5ldC9DU1MvcHJpbnQuY3NzJyB0eXBlPSd0ZXh0L2NzcycgbWVkaWE9J3ByaW50JyAvPjxsaW5rIHJlbD0nc3R5bGVzaGVldCcgaHJlZj0naHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vU3R5bGUgTGlicmFyeS9GU0NPL0ludGVybmV0L0NTUy9tb2JpbGUuY3NzJyB0eXBlPSd0ZXh0L2NzcycgbWVkaWE9J2hhbmRoZWxkJyAvPjxsaW5rIHJlbD0nc3R5bGVzaGVldCcgaHJlZj0naHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vU3R5bGUgTGlicmFyeS9GU0NPL0ludGVybmV0L0NTUy9GU0NPQ3VzdG9tLmNzcycgdHlwZT0ndGV4dC9jc3MnIG1lZGlhPSdzY3JlZW4nIC8+PGxpbmsgcmVsPSdzdHlsZXNoZWV0JyBocmVmPSdodHRwOi8vd3d3LmZzY28uZ292Lm9uLmNhLy9TdHlsZSBMaWJyYXJ5L0ZTQ08vSW50ZXJuZXQvQ1NTL2dlbmVyYWwuY3NzJyB0eXBlPSd0ZXh0L2NzcycgbWVkaWE9J3NjcmVlbicgLz5kAgMPZBYKAgkPFgIeBGhyZWYFHmh0dHA6Ly93d3cuZnNjby5nb3Yub24uY2EvL2VuLxYCAgEPDxYCHghJbWFnZVVybAVKaHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vU3R5bGUgTGlicmFyeS9GU0NPL0ludGVybmV0L0ltYWdlcy9GU0NPbG9nby5naWZkZAILD2QWCAIBDxYCHwEFHmh0dHA6Ly93d3cuZnNjby5nb3Yub24uY2EvL2VuL2QCAw8WAh8BBTZodHRwOi8vd3d3LmZzY28uZ292Lm9uLmNhLy9lbi9BYm91dC9QYWdlcy9kZWZhdWx0LmFzcHhkAgUPFgIfAQU8aHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vZW4vQWJvdXQvUGFnZXMvZGVmYXVsdC5hc3B4I3N1cGVyZAIHDxYCHwEFPmh0dHA6Ly93d3cuZnNjby5nb3Yub24uY2EvL2VuL0Fib3V0L2NvbnRhY3QvUGFnZXMvZGVmYXVsdC5hc3B4ZAIPD2QWAgIBD2QWAgIBD2QWCgICD2QWAgIBD2QWAmYPEGRkFgFmZAIDD2QWAgIBD2QWAmYPDxYEHglCYWNrQ29sb3IKpAEeBF8hU0ICCGRkAgUPZBYEZg8PFgIfAAUXQWdlbnQvQnJva2VyIExhc3QgTmFtZTpkZAIBD2QWAmYPDxYEHwMKpAEfBAIIZGQCBg9kFgICAQ9kFgJmDw8WBB8DCqQBHwQCCGRkAgcPZBYCAgEPZBYCZg8QDxYEHgdDaGVja2VkaB4HRW5hYmxlZGhkZGRkAhMPFgIfAQU+aHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vZW4vQWJvdXQvY29udGFjdC9QYWdlcy9kZWZhdWx0LmFzcHhkAhUPFgIfAQUjaHR0cDovL3d3dy5mc2NvLmdvdi5vbi5jYS8vZW4vSGVscC9kZHbfkxqmdOpuM/SlhkyMUMPoqe7xpqvhLZfOLed46aNe',
    '__VIEWSTATEGENERATOR': '160FAD78',
    '__EVENTVALIDATION': '/wEdAA7Q4cvANrpN5o7qvL/AjrRj3ieM1gYOLXKttt+dfEvWRlC30MDmLdG0SqBLRp4Edr0smFtAlmZS0w2+VR/uBTUgaQpcj9uHtwyf+rB2XgM9KzV/VOD8+NmupvzRtXx7cILCclsqUAusKL6yu6LPJYCYN93eHCeJb+Wv6Dc0KUw/tN8+BEUTySkHJ91vQ/nzu4CsVp8wE0Bpab2MDGOxbDBR3HNdVeUhWlxmX6SwVRp9GtD5VgtZgtwF9KTW5gMitXmBcXMJkDk9iOnoeSz/z5VWv/AwskRm5Qo6YdBnxt7SdQEcL98iN0RCUjhr/FmBpke28iIjJEQtlWEoAG7jfIg+',
    'ctl00$ctl00$MainPlaceHolder$Content4$searchoption': 'Agents or Broker',
    'ctl00$ctl00$MainPlaceHolder$Content4$bkmbno': '',
    'ctl00$ctl00$MainPlaceHolder$Content4$bkmbname': name,
    'ctl00$ctl00$MainPlaceHolder$Content4$agbkcity': '',
    'ctl00$ctl00$MainPlaceHolder$Content4$srButton': 'Search',
    'ctl00$ctl00$hLocal': 'en',
    'ctl00$ctl00$hIsWide': '0'
    })
    return r.text


def parse_result_page_ontario(page):
    soup = BeautifulSoup(page, 'html.parser')
    allA = soup.find_all('a', href=True)
    licenses = []
    for a in allA:
        if('ShowLicence.aspx' in a['href']):
            licenses.append(a.text)
    return licenses


def parse_license_page_ontario(license):
    r = requests.get("http://mbsweblist.fsco.gov.on.ca/ShowLicence.aspx?" + 
    license + "~")
    soup = BeautifulSoup(r.text, 'html.parser')
    return soup.find("span", {"id": 
    "MainPlaceHolder_Content4_cragexpiry"}).text

name = 'crossley'

page = get_result_page_ontario(name)
licenses = parse_result_page_ontario(page)


for l in licenses:
    print(l)
    print(parse_license_page_ontario(l))

Answer 1

在这种情况下，您只需要遍历input.csv就可以将其定义为names

import pandas as pd
names = pd.DataFrame({"First Name":["Grant", "Orlando",
                                    "Adam", "Levan"],
                      "Last Name": ["Morris", "Fantini",
                                    "Crossley", "Sichinava"]})

out = []
for _, name in names.iterrows():
    page = get_result_page_ontario(name["Last Name"])
    licenses = parse_result_page_ontario(page)
    for l in licenses:
        out.append([name["First Name"],
                    name["Last Name"],
                    l, parse_license_page_ontario(l)])

out = pd.DataFrame(out, columns=["First Name",
                                 "Last Name",
                                 "License",
                                 "Expires"])

out.to_csv("output.csv", index=False)

网页剪贴簿，从csv列表传递搜索值并返回csv

1 个答案: