Question

我正在尝试使用Selenium，通过点击此site中的“加载更多”按钮来加载结果页面。但是，由selenium加载的html页面的源代码未显示（加载）浏览时可以看到的实际产品。这是我的代码：

from selenium import webdriver      
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

#browser = webdriver.Firefox()#Chrome('./chromedriver.exe')
URL = "https://thekrazycouponlady.com/coupons-for/costco"
PATIENCE_TIME = 60
LOAD_MORE_BUTTON_XPATH = '//button[@class = "kcl-btn ng-scope"]/span' 
caps = DesiredCapabilities.PHANTOMJS
# driver = webdriver.Chrome(r'C:\Python3\selenium\webdriver\chromedriver_win32\chromedriver.exe')
caps["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
driver = webdriver.PhantomJS(r'C:\Python3\selenium\webdriver\phantomjs-2.1.1-windows\bin\phantomjs.exe',service_log_path=os.path.devnull,desired_capabilities=caps)
driver.get(URL)

while True:
    try:
        time.sleep(20)
        html = driver.page_source.encode('utf-8')
        print(html)
        loadMoreButton = driver.find_element_by_xpath(LOAD_MORE_BUTTON_XPATH)


        loadMoreButton.click()

    except Exception as e:
        print (e)
        break
print ("Complete")

driver.quit()

不确定是否可以在此处附加示例html文件以供参考。无论如何，问题是什么？我如何通过浏览器加载与硒完全相同的页面？

Answer 1

这可能是由于使用了PhantomJS，它不再维护了，并且从Selenium 3.8.1中弃用了。改用无头Chrome。

options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)

Selenium返回的HTML源与浏览器中查看的HTML源不同

1 个答案: