Question

我试图将Spider部署到Scrapy云中，但是我反复遇到需求问题，甚至导入了io，re，shlex和PyPDF2等常用模块。我正在使用Python 3.5。我的scrapinghub.yml文件包含以下几行：

projects:
    default: 358310
stacks:
    default: scrapy:1.3-py3
requirements:
    file: requirements.txt

我的requirements.txt文件包含以下几行：

io
re
shlex
PyPDF2==1.26.0

这是我得到的错误：

Login succeeded
Building an image:
Step 1/12 : FROM scrapinghub/scrapinghub-stack-scrapy:1.3-py3
[91m# Executing 2 build triggers...
[0m
Step 1/1 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST PIP_INDEX_URL $PIP_INDEX_URL
 ---> Using cache
Step 1/1 : RUN test -n $APT_PROXY && echo 'Acquire::http::Proxy \"$APT_PROXY\";'     >/etc/apt/apt.conf.d/proxy
 ---> Using cache
 ---> 8d2af0ecc1ce
Step 2/12 : ENV PYTHONUSERBASE /app/python
 ---> Using cache
 ---> c5bc537289c7
Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/
 ---> Using cache
 ---> 210ce92ef42e
Step 4/12 : ADD run-pipcheck /usr/local/bin/
 ---> Using cache
 ---> 2d0a46143fa4
Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck
 ---> Using cache
 ---> a2eefa41c642
Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck
 ---> Using cache
 ---> f3f5b5c713e3
Step 7/12 : ADD requirements.txt /app/requirements.txt
 ---> Using cache
 ---> e6417a1c3fea
Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python
 ---> Using cache
 ---> 6720be2ef393
Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt
 ---> Running in 8d8694511a74
Collecting io (from -r /app/requirements.txt (line 1))
[91m  Could not find a version that satisfies the requirement io (from -r /app/requirements.txt (line 1)) (from versions: )
[0m
[91mNo matching distribution found for io (from -r /app/requirements.txt (line 1))
[0m
[91mYou are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[0m
{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}

我要去哪里错了？顺便说一句，我安装了最新版本的pip（与错误消息指出的相反）。

Answer 1

我知道，

io，re，shlex是内部库，而不是包。它们不应位于需求文件中。

无法部署到Scrapy云，即使导入通用模块也存在问题

1 个答案: