Question

我正在运行chromedriver尝试从网站上抓取一些数据。没有无头选项，一切工作正常。但是，当我添加该选项时，网络驱动程序将花费很长的时间来加载URL，并且当我尝试查找某个元素（在不使用--headless的情况下运行时会发现该元素）时，我会收到错误消息。

使用打印语句并在“加载” URL之后获取html，我发现没有html，它是空的（请参见下面的输出）。

class Fidelity:
    def __init__(self):
        self.url = 'https://eresearch.fidelity.com/eresearch/gotoBL/fidelityTopOrders.jhtml'
        self.options = Options()
        self.options.add_argument("--headless")
        self.options.add_argument("--window-size=1500,1000")
        self.driver = webdriver.Chrome(executable_path='.\\dependencies\\chromedriver.exe', options = self.options)
        print("init")

    def initiate_browser(self):
        self.driver.get(self.url)
        time.sleep(5)
        script = self.driver.execute_script("return document.documentElement.outerHTML")
        print(script)
        print("got url")

    def find_orders(self):
        wait = WebDriverWait(self.driver, 15)
        data= wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '[id*="t_trigger_TSLA"]'))) #ERROR ON THIS LINE

这是整个输出：

init
<html><head></head><body></body></html>
url
Traceback (most recent call last):
  File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 102, in <module>
    orders = scrape.find_tesla_orders()
  File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 75, in find_tesla_orders
    tesla = self.driver.find_element_by_xpath("//a[@href='https://qr.fidelity.com/embeddedquotes/redirect/research?symbol=TSLA']")
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
    'value': value})['value']
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@href='https://qr.fidelity.com/embeddedquotes/redirect/research?symbol=TSLA']"}
  (Session info: headless chrome=74.0.3729.169)
  (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Windows NT 10.0.17763 x86_64)

新错误，更新代码：

init
<html><head></head><body></body></html>
url
Traceback (most recent call last):
  File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 104, in <module>
    orders = scrape.find_tesla_orders()
  File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 76, in find_tesla_orders
    tesla = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '[id*="t_trigger_TSLA"]')))
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

我尝试通过google找到答案，但是所有建议均无效。其他人在某些网站上是否有此问题？任何帮助表示赞赏。

更新

不幸的是，此脚本仍然无法正常运行，即使没有使用headless选项，即使一切运行正常，webdriver在无头的情况下由于某些原因仍无法正确加载页面。

Answer 1

您是否尝试过使用用户代理？

我遇到了同样的错误。首先我做的是下载无头和普通的 HTML 源页面：

html = driver.page_source
file = open("foo.html","w")
file.write(html)
file.close()

无头模式的 HTML 源代码是一个简短的文件，几乎在末尾有这一行：The page cannot be displayed. Please contact the administrator for additional information. 但正常模式是预期的 HTML。

我通过添加用户代理解决了这个问题：

from fake_useragent import UserAgent
user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(executable_path = f"your_path",chrome_options=chrome_options)

Answer 2

添加显式等待。您还应该使用另一个定位器，当前一个匹配3个元素。元素具有唯一的id属性

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By

wait = WebDriverWait(self.driver, timeout)
data = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '[id*="t_trigger_TSLA"]')))

Answer 3

对于将来想解决此问题的任何人，有些网站只是无法使用chrome的无头选项正确加载。我认为没有办法解决此问题。只需使用其他浏览器（如firefox）即可。感谢user8426627。

Answer 4

我需要在不离开Google浏览器的情况下从同一控制台运行脚本，但是浏览器仍与我的程序一起运行

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("window-size=1920,1080")
print("complete")

driver = webdriver.Chrome('C:\proyectos\python-selenium\driver\chromedriver.exe')
driver.get('https://www.facebook.com/')

Python Selenium Chromedriver无法与--headless选项一起使用

更新

4 个答案: