点击"显示更多" nytimes.com上的按钮与硒

时间:2016-05-20 08:26:17

标签: python css selenium selenium-webdriver web-scraping

我试图滚动浏览this页面。 向下滚动到页面底部后,我需要点击" SHOW MORE"按钮,但我无法用硒找到它。

self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
self.driver.find_element_by_class_name('button load-more-button').click()

但我得到InvalidSelectorException

3 个答案:

答案 0 :(得分:0)

您可以使用 find_element_by_css_selector

.find_element_by_css_selector("button.button.load-more-button").click()

如果您有多个css类,则无法使用find_element_by_class_name

代码如下所示:

In [22]: from selenium import webdriver


In [23]: dr = webdriver.PhantomJS()

In [24]: dr.get("http://www.nytimes.com/column/corner-office")

In [25]: btn = dr.find_element_by_css_selector("button.button.load-more-button") 

In [26]: btn
Out[26]: <selenium.webdriver.remote.webelement.WebElement at 0x7f50c34d7668>

In [27]: btn.click()

使用Firefox,设置 .maximize_window()也可以正常工作:

from selenium import webdriver


dr = webdriver.Firefox()
dr.maximize_window()
dr.get("http://www.nytimes.com/column/corner-office")

btn = dr.find_element_by_css_selector("button.load-more-button")
btn.click()

在Firefox和Chtorm上点击加载更多后,然后必须滚动以获取其余部分,在初始点击后按钮不存在:

dr = webdriver.Firefox()
dr.maximize_window()
dr.get("http://www.nytimes.com/column/corner-office")

btn = dr.find_element_by_css_selector("button.load-more-button")
btn.click()
print( len(dr.find_elements_by_xpath("//h2[@class='headline']")))
dr.execute_script("window.scrollTo(0, document.body.scrollHeight);")
print( len(dr.find_elements_by_xpath("//h2[@class='headline']")))

这给了你:

44
66

答案 1 :(得分:0)

使用以下代码:

elem = self.driver.find_element_by_css_selector(".button.load-more-button[aria-controls$='set-latest']");


//this line will scroll down to make element visible
self.driver.execute_script("window.scrollTo(" + elem.getLocation().x + "," +(elem.getLocation().y- 100) + ");");

elem.click();

答案 2 :(得分:-1)

可以使用ActionChains类,以防您遇到异常,其他元素会收到点击次数。

尝试下面的代码,它应该可以工作。

elem = self.driver.find_element_by_css_selector("button.button.load-more-button")

self.driver.ActionChains(driver).move_to_element(elem).click(elem).perform()