Selenium页面源中缺少HTML元素，但是可以使用BeautifulSoup找到它们

时间：2019-06-05 14:17:42

标签： python selenium selenium-webdriver selenium-chromedriver

到目前为止，我已经看过this，this。

我正在尝试使用Selenium解析HTML源。为了使事情变得容易（或者我想），我从要解析的网页中提取了HTML，并将其放入本地HTML文件中。

BeautifulSoup可以毫无问题地查看HTML，但是Selenium出于某种原因却看不到它。

HTML：

<html><head>
<meta http-equiv="Cache-control" content="no-cache">
<title>NOTICE TO CORES USERS</title>
</head>
<body>
<center>
<b>

<h1>Welcome</h1>
Hours of Operation<br><br>
Monday-Friday 6:00am - 10:00pm<br>
Saturday 8:00am - 6:00pm<br>
Sunday 12:00 noon - 6:00pm<br>
<br>
<br>
<h4><font color="purple"><p><b><u>CORES ORACLE UPGRADE</u></b><br>
<br>
<font color="BLACK">
<font color="RED">Due to a recent technical upgrade, CORES is experiencing a number of issues. We are aware of these issues and our teams are working to resolve them. Corporate Registry will provide updates when available. Corporate Registry apologizes for any inconvenience.
<br>
<br>
<font color="RED">Effective February 3, 2019, <font color="BLACK"> Corporate Registry will send annual return reminders by email to corporations, non-profit organizations, limited liability partnerships, and cooperatives where there is an email address on record.<br>
<br>
Annual return reminders will be emailed about two weeks before the annual return is due.  The reminders will continue to be sent by regular mail
when there is no e-mail address on file or when there is a notice because the previous year's annual return has not been filed.  Directors of Alberta corporations will continue to receive copies of the outstanding annual return notice by regular mail.
<br>
<br>
</font></font></font></font></p><h4><font color="BLACK"><font color="RED"><font color="RED"><font color="BLACK"><font color="purple"><p><b><u>EXTENDED OUTAGE DATES</u></b><br>
<font color="RED"></font></p><p align="CENTRE"><font color="RED">
FULL DAY outages to allow for technical preventive maintenance are as follows:
<br>
<br>
<font color="BLACK">Sunday, May 12, 2019<br>
<font color="BLACK">Sunday, June 9, 2019<br>
<font color="BLACK">Sunday, July 14, 2019<br>
<font color="BLACK">Sunday, August 11, 2019<br>
<font color="BLACK">Sunday, September 8, 2019<br>
<font color="BLACK">Sunday, October 13, 2019<br>
<font color="BLACK">Sunday, November 10, 2019<br>
<font color="BLACK">Sunday, December 8, 2019<br>

</font></font></font></font></font></font></font></font></font></p><p align="CENTRE"></p><h5><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">Updated: April 30, 2019</font></font></font></font></font></h5><p></p><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">

<br>
<br>
<form action="cr_login.menu_frame" method="post">
<input type="hidden" name="p_default_menu" value="5">
<input type="hidden" name="p_system" value="CR">
<input type="hidden" name="p_accreditation" value="1">
<input type="hidden" name="p_spuid" value="30825">
<input type="hidden" name="p_userid" value="A02526">
<input type="submit" value="Continue">
</form>
</font></font></font></font></font></font></font></font></font></font></h4></font></h4></b></center><b><font color="purple"><font color="RED"><font color="RED"><font color="purple"><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">


</font></font></font></font></font></font></font></font></font></b></body></html>

Python：

from bs4 import BeautifulSoup

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

chrome_options = Options()
chrome_options.binary_location = '/usr/bin/google-chrome'
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome(chrome_options=chrome_options)
#driver.get('file:/c/Users/lanes/learning/product-repo/backend/functions/src/cores-scraper/oracle_upgrade.html')
driver.implicitly_wait(5)
driver.get('file:oracle_upgrade.html')

print('page source:', driver.page_source)

soup = BeautifulSoup(open('oracle_upgrade.html', 'r'), 'html.parser')

print('\nsoup:', soup)

driver.close()

输出：

page source: <html><head></head><body></body></html>

soup: <html><head>
<meta content="no-cache" http-equiv="Cache-control"/>
<title>NOTICE TO CORES USERS</title>
</head>
<body>
<center>
<b>
<h1>Welcome</h1>
Hours of Operation<br/><br/>
Monday-Friday 6:00am - 10:00pm<br/>
Saturday 8:00am - 6:00pm<br/>
Sunday 12:00 noon - 6:00pm<br/>
<br/>
<br/>
<h4><font color="purple"><p><b><u>CORES ORACLE UPGRADE</u></b><br/>
<br/>
<font color="BLACK">
<font color="RED">Due to a recent technical upgrade, CORES is experiencing a number of issues. We are aware of these issues and our teams are working to resolve them. Corporate Registry
will provide updates when available. Corporate Registry apologizes for any inconvenience.
<br/>
<br/>
<font color="RED">Effective February 3, 2019, <font color="BLACK"> Corporate Registry will send annual return reminders by email to corporations, non-profit organizations, limited liability partnerships, and cooperatives where there is an email address on record.<br/>
<br/>
Annual return reminders will be emailed about two weeks before the annual return is due.  The reminders will continue to be sent by regular mail
when there is no e-mail address on file or when there is a notice because the previous year's annual return has not been filed.  Directors of Alberta corporations will continue to receive copies of the outstanding annual return notice by regular mail.
<br/>
<br/>
</font></font></font></font></p><h4><font color="BLACK"><font color="RED"><font color="RED"><font color="BLACK"><font color="purple"><p><b><u>EXTENDED OUTAGE DATES</u></b><br/>
<font color="RED"></font></p><p align="CENTRE"><font color="RED">
FULL DAY outages to allow for technical preventive maintenance are as follows:
<br/>
<br/>
<font color="BLACK">Sunday, May 12, 2019<br/>
<font color="BLACK">Sunday, June 9, 2019<br/>
<font color="BLACK">Sunday, July 14, 2019<br/>
<font color="BLACK">Sunday, August 11, 2019<br/>
<font color="BLACK">Sunday, September 8, 2019<br/>
<font color="BLACK">Sunday, October 13, 2019<br/>
<font color="BLACK">Sunday, November 10, 2019<br/>
<font color="BLACK">Sunday, December 8, 2019<br/>
</font></font></font></font></font></font></font></font></font></p><p align="CENTRE"></p><h5><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">Updated: April 30, 2019</font></font></font></font></font></h5><p></p><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
<br/>
<br/>
<form action="cr_login.menu_frame" method="post">
<input name="p_default_menu" type="hidden" value="5"/>
<input name="p_system" type="hidden" value="CR"/>
<input name="p_accreditation" type="hidden" value="1"/>
<input name="p_spuid" type="hidden" value="30825"/>
<input name="p_userid" type="hidden" value="A02526"/>
<input type="submit" value="Continue"/>
</form>
</font></font></font></font></font></font></font></font></font></font></h4></font></h4></b></center><b><font color="purple"><font color="RED"><font color="RED"><font color="purple"><font color="RED"><font color="BLACK"><font color="BLACK"><font color="BLACK"><font color="PURPLE">
</font></font></font></font></font></font></font></font></font></b></body></html>

问题：

为什么Selenium无法像Soup一样“看到”体内的HTML？

2 个答案:

答案 0 :(得分：2)

我能够复制您的问题。

您被欺骗了，因为driver.get（）无法加载文件时，它不会真正返回错误。相反，driver.page_source将包含几乎为空的文档。我不确定您的文件位于何处，但是我认为您只是在想文件URI应该以file：//开头而不是file：

以下代码对我有用：

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('file:///Users/jimmy/src/stackoverflow/html-elements-missing-from-selenium-page-source-but-can-be-found-using-beautifu/oracle_upgrade.html')
print('page source:', driver.page_source)
driver.close()

您不应期望与输入完全相同的输出，因为chrome会为您“修复”您的html。例如，如果您忘记了，它将毫无礼貌地将其添加到源中。

答案 1 :(得分：1)

我无法使用以下方式重现您的问题：

enter image description here

所以我推荐

  • 使用如下pip升级到最新的selenium package版本：

    // click class 
public class Click
    {
        public string ID;
        public string Type;
        public string Time;

        public Click(string ID, string Type, string Time)
        {
            this.ID = ID;
            this.Type = Type;
            this.Time = Time;
        }
    }


// by button clicking calling this method:

 private void TableUpdate()
        {
            IDnum++;

            //test strings:
            var cl = new Click("asd", "sdad", "asds");


            ClicksGrid.Items.Add(cl);   
        }

  • 交叉检查您的Chrome和ChromeDriver版本-必须100％匹配

  • 或者您可以尝试通过XPath expression

    获取页面源 
    pip install --upgrade selenium

    enter image description here

相关问题
最新问题