Question

我需要使用javascript从几个网站解析新闻并使用 selenium + PhantomJS 。但是这些网站上有视频，对我来说没用，我根本不需要它们。（我得到了使用selenium + Chrome或selenium + Firefox的建议，但在解析过程中我不需要任何打开的窗口。）

根据网站的逻辑，这些视频开始自动播放，并在结束时异常http.client.RemoteDisconnected: Remote end closed connection without response抛出。

我认为它会引发因为我的互联网非常慢而且视频无法满载它。

我该如何避免这个问题？

selenium或PhantomJS中可能存在内容约束吗？

完整追溯：

File "viralnova/viralnova.py", line 101, in parse_viralnova
    _parse_post_link(postlinktest, driver)
  File "viralnova/viralnova.py", line 9, in _parse_post_link
    driver.get(post_link)
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 309, in get
    self.execute(Command.GET, {'url': url})
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 295, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 464, in execute
    return self._request(command_info[0], url, body=data)
  File "/Users/user/anaconda/envs/env/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 526, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 1346, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Users/user/anaconda/envs/env/lib/python3.6/urllib/request.py", line 1321, in do_open
    r = h.getresponse()
  File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/Users/user/anaconda/envs/env/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

代码在这里

def _parse_post_link(post_link, driver):
    try:
        driver.get(post_link)
    except Exception:
        return None

    post_page_soup = Soup(driver.page_source, "lxml")
    title = post_page_soup.find('div', attrs={'class': 'post-box-detail article'}).h2.text
    print(title)

def parse_viralnova(to_csv=True):
    driver = webdriver.PhantomJS("/Users/user/.phantomjsdriver/phantomjs")
    postlinktest = 'http://www.viralnova.com/restroom-design-fails/'
    _parse_post_link(postlinktest, driver)

Answer 1

如果它只是解析您所追求的文本内容，您可以考虑仅使用Python and BeautifulSoup。您不应该以这种方式在浏览器中触发任何内容，因为您根本不会使用它（您提到您不需要打开窗口），同时解决方案将更快地缺少浏览器开销。

如果你确实需要加载一些javascript，你也可以尝试使用dryscape。

Selenium python类型的内容加载约束

1 个答案: