Question

我在pythonanywhere.com上安装了dryscrape。然而session var无法加载任何网站，为什么？

import dryscrape
# as in demo: http://dryscrape.readthedocs.io/en/latest/usage.html#first-demonstration
dryscrape.start_xvfb() 

sess = dryscrape.Session()
sess.visit('https://www.pythonanywhere.com/')

结果错误：

sess.visit('https://www.pythonanywhere.com/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/igorsavinkin/.local/lib/python3.5/site-packages/dryscrape/session.py", line 33, in visit
    return self.driver.visit(self.complete_url(url))
  File "/home/igorsavinkin/.local/lib/python3.5/site-packages/webkit_server.py", line 235, in visit
    self.conn.issue_command("Visit", url)
  File "/home/igorsavinkin/.local/lib/python3.5/site-packages/webkit_server.py", line 520, in issue_command
    return self._read_response()
  File "/home/igorsavinkin/.local/lib/python3.5/site-packages/webkit_server.py", line 530, in _read_response
    raise InvalidResponseError(msg)
webkit_server.InvalidResponseError: {"class":"InvalidResponseError","message":"Unable to load URL: https://www.pythonanywhere.com/ because 
of error loading https://www.pythonanywhere.com/: Unknown error"}

无论我从whitelisted进行会话访问的网站是什么，问题都是一样的。

我读过有关dryscrape installation prerequisits的信息：

在安装dryscrape之前，您需要安装一些依赖于它的软件：


Qt，QtWebKit

LXML

PIP

xvfb_（仅在没有其他X服务器可用时才需要）

因此，Qt和QtWebKit都不属于pythoneverywhere的默认模块......

当我尝试安装它时，结果是错误（与QtWebKit相同）

$ pip install --user Qt
Collecting Qt
  Could not find a version that satisfies the requirement Qt (from versions: )
No matching distribution found for Qt

dryscrape设置文件，setup.py：

from distutils.core import setup, Command

setup(name='dryscrape',
      version='0.9.1',
      description='a lightweight Javascript-aware, headless web scraping library for Python',
      author='Niklas Baumstark',
      author_email='niklas.baumstark@gmail.com',
      license='MIT',
      url='https://niklasb.github.com/dryscrape',
      packages=['dryscrape', 'dryscrape.driver'],
      requires=['webkit_server', 'lxml'],
      )

任何帮助都很明显......

Answer 1

PythonAnywhere dev here - 遗憾的是dryscrape依赖于WebKit，而WebKit并不适用于我们的虚拟化系统。如果您需要使用可以处理JavaScript的浏览器进行网页抓取，您可以使用selenium和Firefox - 那里more information on our blog。但请注意，我们只有Firefox版本17 - 更新的问题与WebKit存在相同的问题。

Dryscrape会话无法加载任何网站

1 个答案: