Question

我正在用硒刮一个网站。该网站使用无限滚动，因此我尝试使用selenium向下滚动并获得比第一次请求后提供的搜索结果更多的搜索结果。

我改编了以下优秀剧本：

crawl site that has infinite scrolling using python

from selenium import webdriver
import time

driver = webdriver.Firefox()
driver.implicitly_wait(30)
verificationErrors = []
accept_next_alert = True

driver.get(my_url)
driver.find_element_by_link_text("All").click()
for i in range(1,100):
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(4)
    html_source = driver.page_source
    data = html_source.encode('utf-8')

使用上面的网址打开网络浏览器。但是，我收到以下错误消息：

C:\Users\xxxx\Anaconda3\python.exe
C:/Users/xxxx/Documents/IdenSoft/unittest.py
Traceback (most recent call last):
File "C:/Users/xxxx/unittest.py", line 23, in <module>
driver.find_element_by_link_text("All").click()
File C:\Users\xxxx\Anaconda3\lib\sitepackages\selenium\webdriver\remote\webelement.py", line 77, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\xxxx\Anaconda3\lib\site
packages\selenium\webdriver\remote\webelement.py", line 494, in _execute
return self._parent.execute(command, params)
File "C:\xxxx\xxxx\Anaconda3\lib\site-  packages\selenium\webdriver\remote\webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "C:\Users\xxxx\Anaconda3\lib\site     packages\selenium\webdriver\remote\errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: 


Process finished with exit code 1

如何更改脚本以使其正常工作？

Answer 1

这一行有一个问题：driver.find_element_by_link_text("All").click()

试试这个：

from selenium import webdriver
import time
my_url = "https://aparat.com/khaledsss" #you can change this
#Open Browser
driver = webdriver.Firefox()
#Open Url
driver.get(my_url)
#for loop 
for i in range(1,100):
    # * With this line You can skip infinite scroll
    #scroll to Down
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    #Wait for 4 secend
    time.sleep(4)
# ... code ...

无限滚动 - 蟒蛇

1 个答案: