我试图在此页面上进行无限滚动,这是我的代码:
from selenium import webdriver
import time
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0")
driver = webdriver.Firefox(profile)
driver.get("http://www.quora.com/Programming-Languages/followers")
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work.
driver.execute_script("window.scrollTo(0,1000000);")
time.sleep(2)
所以当我运行它时,它会在进行任何滚动之前等待很多秒(有时超过1分钟),然后在下一次滚动之前再等待相同的时间。 代码似乎在其他页面上正常工作。 关于如何解决这个问题的任何想法?
当我尝试使用Chrome而不是firefox时,我收到以下错误:
driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
已添加到.py文件中。
Traceback (most recent call last):
File "ok.py", line 8, in <module>
driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session
'desiredCapabilities': desired_capabilities,
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
self._conn.request(method, parsed_url.path, body, headers)
File "/usr/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known
答案 0 :(得分:1)
切换到Chrome()
帮我解决了问题:
import time
from selenium import webdriver
followers_per_page = 18
driver = webdriver.Chrome()
driver.get("http://www.quora.com/Programming-Languages/followers")
# get the followers count
element = driver.find_element_by_class_name('count')
followers_count = int(element.text.replace('k', '000').replace('.', ''))
print followers_count
# scroll down the page iteratively with a delay
for _ in xrange(0, followers_count/followers_per_page + 1):
driver.execute_script("window.scrollTo(0, 0,1000000);")
time.sleep(2)
仅供参考,我使用了一种不同的方法:解析关注者的数量并计算每页的关注者,同时考虑到它一次加载18个关注者的事实。
我之前实际上曾在类似的quora问题上工作过,请参阅:
嗯,这不是我想到的第一件事。这就是故事。
问题是,有http://tch840195.tch.quora.com/up/chan5-8886/updates个网址的待处理请求需要几分钟才能完成。这就是让selenium认为页面没有完全加载的原因。事情变得越来越糟 - 这是一个每X秒发生一次的周期性事情。把它想象成长时间的汇集。
我已尝试使用Firefox
webdriver:
webdriver.load.strategy
偏好设置为unstable
network.http.response.timeout
,network.http.connection-timeout
和network.http.keep-alive.timeout
以及network.http.request.max-start-delay
偏好设置设置页面加载超时:
driver.set_page_load_timeout(3)
设置脚本超时:
driver.set_script_timeout(3)
调用window.stop();
希望它会停止活动请求:
driver.execute_script('window.stop();')
更新到最新的Firefox和selenium软件包版本
可能有效的另一个选项是以某种方式阻止对"slow url"的请求使用代理服务器并将firefox指向它,或者,如果可能的话,让Firefox知道将URL列入黑名单(可能通过延期)。
另请参阅内部多个解决方法的相关问题:
另见: