Question

我正在尝试从网页上的表中提取数据到pandas数据框中。该网站是 https://nextgenstats.nfl.com/stats/passing/2019/1 。我正在使用硒和镀铬的webdriver。我相信我的问题是我无法识别表的元素ID。我没有使用html的经验，因此很难进行故障排除。

我尝试使用熊猫内置的read_html（）函数，但遇到“未找到表”错误。我切换到Selenium，并且正在使用Chrome浏览器，但仍然存在相同的错误。我还尝试添加延迟以让页面加载，但这似乎无济于事。

import pandas as pd
from selenium import webdriver

driver= webdriver.Chrome()
# scrap webpage
driver.implicitly_wait(10)
driver.get('https://nextgenstats.nfl.com/stats/passing/2019/1')

html = driver.page_source
tables = pd.read_html(html)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-29-e2e5cae6f0b5> in <module>
      1 url = f'https://nextgenstats.nfl.com/stats/passing/2019/1'
----> 2 tats_list = pd.read_html(url)

C:\Python37-32\lib\site-packages\pandas\io\html.py in read_html(io, match, flavor, header, index_col, skiprows, attrs, parse_dates, thousands, encoding, decimal, converters, na_values, keep_default_na, displayed_only)
   1103         na_values=na_values,
   1104         keep_default_na=keep_default_na,
-> 1105         displayed_only=displayed_only,
   1106     )

C:\Python37-32\lib\site-packages\pandas\io\html.py in _parse(flavor, io, match, attrs, encoding, displayed_only, **kwargs)
    910             break
    911     else:
--> 912         raise_with_traceback(retained)
    913 
    914     ret = []

C:\Python37-32\lib\site-packages\pandas\compat\__init__.py in raise_with_traceback(exc, traceback)
     45     if traceback == Ellipsis:
     46         _, _, traceback = sys.exc_info()
---> 47     raise exc.with_traceback(traceback)
     48 
     49 

ValueError: No tables found

import pandas as pd
from selenium import webdriver

driver= webdriver.Chrome()
# scrap webpage
driver.implicitly_wait(10)

html = driver.page_source
# find table by using suspected table id
tables = driver.find_element_by_id("gs-data-table")

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
<ipython-input-31-5596e919e5ff> in <module>
      5 html = driver.page_source
      6 
----> 7 tables = driver.find_element_by_id("gs-data-table")

C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element_by_id(self, id_)
    358             element = driver.find_element_by_id('foo')
    359         """
--> 360         return self.find_element(by=By.ID, value=id_)
    361 
    362     def find_elements_by_id(self, id_):

C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element(self, by, value)
    976         return self.execute(Command.FIND_ELEMENT, {
    977             'using': by,
--> 978             'value': value})['value']
    979 
    980     def find_elements(self, by=By.ID, value=None):

C:\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
    319         response = self.command_executor.execute(driver_command, params)
    320         if response:
--> 321             self.error_handler.check_response(response)
    322             response['value'] = self._unwrap_value(
    323                 response.get('value', None))

C:\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
    240                 alert_text = value['alert'].get('text')
    241             raise exception_class(message, screen, stacktrace, alert_text)
--> 242         raise exception_class(message, screen, stacktrace)
    243 
    244     def _value_or_default(self, obj, key, default):

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="gs-data-table"]"}
  (Session info: chrome=77.0.3865.120)

我希望输出是位于网页上的表格的数据框，但无法解决此错误。对替代方法的任何帮助，或对标识表元素ID的帮助，都将非常有用。谢谢。

Answer 1

请尝试以下解决方案：

from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome('C:\chromedriver.exe')
driver.maximize_window()
driver.get("https://nextgenstats.nfl.com/stats/passing/2019/1")
driver.implicitly_wait(10)
table_id=driver.find_element_by_xpath("//div[@class='el-table__body-wrapper']//tbody");

rows = table_id.find_elements(By.TAG_NAME, "tr") # get all of the rows in the table
for row in rows:
    # Get the columns (all the column 2)
    col = row.find_elements(By.TAG_NAME, "td")[1] #note: index start from 0, 1 is col 2
    print col.text #prints text from the element

如何使用硒修复“未找到表”错误

1 个答案: