为什么selenium在执行此代码之前会等待很长时间?

时间:2014-10-08 13:20:16

标签: python firefox selenium selenium-webdriver web-scraping

我试图在此页面上进行无限滚动,这是我的代码:

from selenium import webdriver
import time

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0")
driver = webdriver.Firefox(profile)

driver.get("http://www.quora.com/Programming-Languages/followers")
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work.
    driver.execute_script("window.scrollTo(0,1000000);")
    time.sleep(2)

所以当我运行它时,它会在进行任何滚动之前等待很多秒(有时超过1分钟),然后在下一次滚动之前再等待相同的时间。 代码似乎在其他页面上正常工作。 关于如何解决这个问题的任何想法?

当我尝试使用Chrome而不是firefox时,我收到以下错误: driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')已添加到.py文件中。

Traceback (most recent call last):
  File "ok.py", line 8, in <module>
    driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__
    keep_alive=True)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session
    'desiredCapabilities': desired_capabilities,
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

1 个答案:

答案 0 :(得分:1)

切换到Chrome()帮我解决了问题:

import time
from selenium import webdriver

followers_per_page = 18

driver = webdriver.Chrome()
driver.get("http://www.quora.com/Programming-Languages/followers")

# get the followers count
element = driver.find_element_by_class_name('count')
followers_count = int(element.text.replace('k', '000').replace('.', ''))
print followers_count

# scroll down the page iteratively with a delay
for _ in xrange(0, followers_count/followers_per_page + 1):
    driver.execute_script("window.scrollTo(0, 0,1000000);")
    time.sleep(2)

仅供参考,我使用了一种不同的方法:解析关注者的数量并计算每页的关注者,同时考虑到它一次加载18个关注者的事实。

我之前实际上曾在类似的quora问题上工作过,请参阅:


嗯,这不是我想到的第一件事。这就是故事。

问题是,有http://tch840195.tch.quora.com/up/chan5-8886/updates个网址的待处理请求需要几分钟才能完成。这就是让selenium认为页面没有完全加载的原因。事情变得越来越糟 - 这是一个每X秒发生一次的周期性事情。把它想象成长时间的汇集。

我已尝试使用Firefox webdriver:

来解决问题
  • webdriver.load.strategy偏好设置为unstable
  • 设置network.http.response.timeoutnetwork.http.connection-timeoutnetwork.http.keep-alive.timeout以及network.http.request.max-start-delay偏好设置
  • 设置页面加载超时:

    driver.set_page_load_timeout(3)
    
  • 设置脚本超时:

    driver.set_script_timeout(3)
    
  • 调用window.stop();希望它会停止活动请求:

    driver.execute_script('window.stop();')
    
  • 更新到最新的Firefox和selenium软件包版本

可能有效的另一个选项是以某种方式阻止对"slow url"的请求使用代理服务器并将firefox指向它,或者,如果可能的话,让Firefox知道将URL列入黑名单(可能通过延期)。

另请参阅内部多个解决方法的相关问题:

另见: