Question

我正在尝试在Dockerfile中构建一个Python项目。我想缓存依赖项，然后在以后使用该缓存，如下所示：

RUN pip3 download -d "/pth/to/downloaded/files" -r /temp/requirements.txt -c /temp/constraints.txt

# much later on in the Dockerfile:

RUN pip3 install --download-cache="/pth/to/downloaded/files" -r requirements.txt -c constraints.txt

问题：假设pip3 download命令是正确的，当我查看--download-cache的输出时，我不再看到pip3 --help选项-是否有可以与pip3 install一起使用的新选项可供参考由pip3 download生成的依赖项缓存？

现在我收到此错误：

用法：pip install [选项]

没有这样的选项：--download-cache

Answer 1

在点子版本8中，--download-cache选项已删除，因为默认情况下它现在使用缓存。因此，您根本不需要指定此选项。我不确定pip download -d <dir>选项的用途是什么，但是显然它没有在目标目录中创建缓存。您也可以省去-d <dir>选项。

以下Dockerfile有效：

FROM python:3.7
COPY constraints.txt requirements.txt ./
RUN pip3 download -d .pipcache -r requirements.txt -c constraints.txt
COPY test.txt ./
RUN pip3 install -r requirements.txt -c constraints.txt

如果将--cache-dir <dir>和download和install命令同时添加到{strong>两者，它也将正常工作。因此，以下Dockerfile也适用：

FROM python:3.7
COPY constraints.txt requirements.txt ./
RUN pip3 download --cache-dir ./tmp/pipcache -r requirements.txt -c constraints.txt
COPY test.txt ./
RUN pip3 install --cache-dir ./tmp/pipcache -r requirements.txt -c constraints.txt

示例输出（pep8中只有pylint和requirements.txt）：

首次运行：

Sending build context to Docker daemon  5.632kB
Step 1/5 : FROM python:3.7
 ---> a4cc999cf2aa
Step 2/5 : COPY constraints.txt requirements.txt ./
 ---> 411eaa3d36ff
Step 3/5 : RUN pip3 download -r requirements.txt -c constraints.txt
 ---> Running in 6b489df74137
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Downloading https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl (41kB)
  Saved /pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Downloading https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl (765kB)
  Saved /pylint-2.3.1-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
  Saved /mccabe-0.6.1-py2.py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl (193kB)
  Saved /astroid-2.2.5-py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl (42kB)
  Saved /isort-4.3.19-py2.py3-none-any.whl
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Downloading https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (49kB)
  Saved /lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
  Saved /wrapt-1.11.1.tar.gz
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
  Saved /six-1.12.0-py2.py3-none-any.whl
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Downloading https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl (736kB)
  Saved /typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Successfully downloaded pep8 pylint mccabe astroid isort lazy-object-proxy wrapt six typed-ast
Removing intermediate container 6b489df74137
 ---> 8ac3be432c58
Step 4/5 : COPY test.txt ./
 ---> 5cac20851967
Step 5/5 : RUN pip3 install -r requirements.txt -c constraints.txt
 ---> Running in 394847f09e9b
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Using cached https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Using cached https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Using cached https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc
Successfully built wrapt
Installing collected packages: lazy-object-proxy, six, wrapt, typed-ast, astroid, isort, mccabe, pep8, pylint
Successfully installed astroid-2.2.5 isort-4.3.19 lazy-object-proxy-1.4.1 mccabe-0.6.1 pep8-1.7.1 pylint-2.3.1 six-1.12.0 typed-ast-1.3.5 wrapt-1.11.1
Removing intermediate container 394847f09e9b
 ---> 68e65a214a32
Successfully built 68e65a214a32
Successfully tagged test:latest

第二次运行（更改test.txt后触发Docker层4和5的重建）

Sending build context to Docker daemon  5.632kB
Step 1/5 : FROM python:3.7
 ---> a4cc999cf2aa
Step 2/5 : COPY constraints.txt requirements.txt ./
 ---> Using cache
 ---> 411eaa3d36ff
Step 3/5 : RUN pip3 download -r requirements.txt -c constraints.txt
 ---> Using cache
 ---> 8ac3be432c58
Step 4/5 : COPY test.txt ./
 ---> 7ab5814153b7
Step 5/5 : RUN pip3 install -r requirements.txt -c constraints.txt
 ---> Running in 501da787ab07
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Using cached https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Using cached https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Using cached https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc
Successfully built wrapt
Installing collected packages: typed-ast, six, wrapt, lazy-object-proxy, astroid, isort, mccabe, pep8, pylint
Successfully installed astroid-2.2.5 isort-4.3.19 lazy-object-proxy-1.4.1 mccabe-0.6.1 pep8-1.7.1 pylint-2.3.1 six-1.12.0 typed-ast-1.3.5 wrapt-1.11.1
Removing intermediate container 501da787ab07
 ---> b377fe561e97
Successfully built b377fe561e97
Successfully tagged test:latest

注意：official documentation很有帮助。

Answer 2

正如 wovano 所述，从PIP 8.0.0到--download-cache选项已被删除。 pip默认使用缓存。

要重新使用已下载/缓存的软件包，可以使用以下策略：

使用--dest as described here将软件包下载到自定义目标目录。

$ pip3 download --dest "$DEST_DIR" ...

使用$DEST_DIR as described here从本地目录--find-links安装以前下载的软件包。

$ pip3 install --find-links "file://${DEST_DIR}" ...

总结：

FROM python:3

ENV PIP_DOWNLOAD_CACHE "/var/custom-pip-download-dir"

COPY requirements1.txt ./requirements1.txt
RUN pip3 download -r requirements1.txt --dest "$PIP_DOWNLOAD_CACHE"

COPY requirements2.txt ./requirements2.txt
RUN pip3 install -r requirements2.txt --find-links "file://${PIP_DOWNLOAD_CACHE}"

Answer 3

（此处为点子维护者）

在现代pip版本（> 6.0）中，如果可访问默认缓存目录（请参见documentation），pip将自动缓存其正在安装的软件包。

只要您能以某种方式重新填充此缓存，pip便会在可能时使用它们（尽管在网络可用性较差时它不会退回给它）。

另一种选择（我个人更喜欢）是使用pip下载一步下载文件，以便Docker可以自己缓存内容。下一步将使用pip install --no-index和--find-links进行“离线”安装。

$ pip download --dest=downloaded-packages project
$ pip install --no-index --find-links=downloaded-packages project

编辑：

Python打包用户指南，也包含与此相关的指南。它有一个类似的建议（尽管它建议使用pip wheel而不是pip download，这是一个很好的建议）：https://packaging.python.org/guides/index-mirrors-and-caches/#caching-with-pip

Answer 4

正确地缓存python依赖性意味着在构建Dockerfile时，它会重新使用install python依赖性。我试图通过使用下面的DockerFile来构建Docker。

requirements.txt

Django==2.1.7

DockerFile

FROM python:3
ENV PYTHONUNBUFFERED 1
ADD requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt

构建docker映像

docker build .

首次构建docker

第二次构建docker。默认情况下，它使用缓存中的依赖项。

如果您对Requirements.txt进行任何更改或库版本更改，则它不使用缓存。并安装新的库版本。

谢谢。

如何正确缓存Python依赖项

4 个答案: