Question

我正在使用Selenium来为同一张表抓取多个URL，但是这些表的xpath略有不同。

下面是我的编码：

my_urls = ["https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001548760",
"https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001366010",
"https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001164390"]

driver = webdriver.Chrome()
for url in my_urls:
    driver.get(url)    
    export_table=driver.find_elements_by_xpath('')[0]
    export_table.text

xpath1：/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[3]/td/table/tbody

xpath2：/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[2]/td/table/tbody

如何使用一个xpath从这些URL中提取内容？并将所有结果导出到字典中？

谢谢您的帮助！

Answer 1

如果要从每个xpath获取文本，请尝试此操作。如果您希望每个网址都有一个路径，则应使用字典在网址和xpath之间建立映射。您可以遍历该词典以执行您想做的事情。

import json
from selenium import webdriver
my_urls = ["https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001548760",
"https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001366010",
"https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001164390"]

xpath1 = """/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[3]/td/table/tbody"""
xpath2 = """/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[2]/td/table/tbody"""

def getpath(element):
    try:
        return element[0].text
    except IndexError as _:
        return None

export_table = {}

driver = webdriver.Chrome("chromedriver.exe")
for url in my_urls:
    driver.get(url)
    export_table[url] = {path: getpath(driver.find_elements_by_xpath(path)) for path in [xpath1, xpath2]}

driver.close()

json.dumps(export_table)

输出

{
  "https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001548760": {
    "/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[3]/td/table/tbody": "Issuer Filings Transaction Date Type of Owner\\nFacebook Inc 0001326801 2019-04-26 director, 10 percent owner, officer: COB and CEO",
    "/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[2]/td/table/tbody": "Mailing Address\\nC/O FACEBOOK, INC.\\n1601 WILLOW ROAD\\nMENLO PARK CA 94025"
  },
  "https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001366010": {
    "/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[3]/td/table/tbody": "Issuer Filings Transaction Date Type of Owner\\nFacebook Inc 0001326801 2019-07-08 director, officer: Chief Operating Officer\\nSVMK Inc. 0001739936 2019-02-21 director\\nWALT DISNEY CO/\\nCurrent Name:TWDC Enterprises 18 Corp. 0001001039 2017-11-22 director\\nSTARBUCKS CORP 0000829224 2011-11-14 director\\neHealth, Inc. 0001333493 2008-06-10 director",
    "/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[2]/td/table/tbody": "Mailing Address\\n1 FACEBOOK WAY\\nMENLO PARK CA 94025"
  },
  "https://www.sec.gov/cgi-bin/own-disp?action=getowner&CIK=0001164390": {
    "/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[3]/td/table/tbody": null,
    "/html/body/div/table[1]/tbody/tr[2]/td/table/tbody/tr[2]/td/table/tbody": "Issuer Filings Transaction Date Type of Owner\\nACE LTD\\nCurrent Name:Chubb Ltd 0000896159 2019-06-06 officer: Executive Vice President*"
  }
}

Selenium-Web对相同内容但xpath稍有不同的网站抓取多个URL

1 个答案: