具有所有依赖项的Python分布式zip

时间:2019-05-27 09:59:50

标签: python dependencies distributed

我正在尝试创建一个具有诸如requests之类的依赖项的简单项目,并使用以下命令创建一个分布式zip文件:python3.6 setup.py bdist --format=zip。在dist zip文件中,还应该使用以下bash脚本创建package.zip:

#!/bin/bash
pip install -r requirements.txt --target ./packages
if [[ -d packages ]]; then
    cd packages
    find . -name "*.pyc" -delete
    find . -name "*.egg-info" | xargs rm -rf
    zip -9mrv packages.zip .
    mv packages.zip ..
    cd ..
    rm -rf packages
fi

最终的zip文件包含以下文件:


Archive:  dist/test_python_zip-1.0.linux-x86_64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2019-05-27 12:37   src/
        0  2019-05-27 12:37   __pycache__/
        0  2019-05-27 12:37   commands/
      187  2019-05-27 12:37   test_python_zip-1.0-py3.6.egg-info
        0  2019-05-22 19:18   __init__.py
      222  2019-05-27 12:34   __main__.py
        0  2019-05-27 12:37   src/commands/
        0  2019-05-27 12:37   src/commands/__pycache__/
       28  2019-05-22 17:47   src/commands/buy.py
       18  2019-05-22 19:02   src/commands/__init__.py
      246  2019-05-22 18:39   src/commands/hello.py
       30  2019-05-22 17:47   src/commands/start.py
      125  2019-05-27 12:37   src/commands/__pycache__/__init__.cpython-36.pyc
      337  2019-05-27 12:37   src/commands/__pycache__/hello.cpython-36.pyc
      214  2019-05-27 12:37   src/commands/__pycache__/start.cpython-36.pyc
      210  2019-05-27 12:37   src/commands/__pycache__/buy.cpython-36.pyc
       95  2019-05-27 12:37   __pycache__/__init__.cpython-36.pyc
      403  2019-05-27 12:37   __pycache__/__main__.cpython-36.pyc
        0  2019-05-27 12:37   commands/__pycache__/
       28  2019-05-22 17:47   commands/buy.py
       18  2019-05-22 19:02   commands/__init__.py
      246  2019-05-22 18:39   commands/hello.py
       30  2019-05-22 17:47   commands/start.py
      121  2019-05-27 12:37   commands/__pycache__/__init__.cpython-36.pyc
      333  2019-05-27 12:37   commands/__pycache__/hello.cpython-36.pyc
      210  2019-05-27 12:37   commands/__pycache__/start.cpython-36.pyc
      206  2019-05-27 12:37   commands/__pycache__/buy.cpython-36.pyc
---------                     -------
     3307                     27 files

setup.py文件的内容为:

from distutils.command.bdist_dumb import bdist_dumb
from distutils.core import setup


class custom_bdist_dumb(bdist_dumb):

    def reinitialize_command(self, name, **kw):
        cmd = bdist_dumb.reinitialize_command(self, name, **kw)
        if name == 'install':
            cmd.install_lib = '/'
        return cmd


if __name__ == '__main__':
    setup(
        # our custom class override
        cmdclass={'bdist_dumb': custom_bdist_dumb},
        name='test_python_zip',
        version="1.0",
        py_modules=['__main__'],
        packages=['lib', 'src/commands'])

项目树如下:


.
├── build
│   ├── bdist.linux-x86_64
│   ├── lib
│   │   ├── commands
│   │   │   ├── buy.py
│   │   │   ├── hello.py
│   │   │   ├── __init__.py
│   │   │   └── start.py
│   │   ├── __init__.py
│   │   ├── __main__.py
│   │   └── src
│   │       └── commands
│   │           ├── buy.py
│   │           ├── hello.py
│   │           ├── __init__.py
│   │           └── start.py
│   └── lib.linux-x86_64-2.7
│       └── commands
│           ├── buy.py
│           ├── hello.py
│           ├── __init__.py
│           └── start.py
├── dist
│   └── test_python_zip-1.0.linux-x86_64.zip
├── __main__.py
├── packages.sh
├── packages.zip
├── __pycache__
├── requirements.txt
├── setup.py
├── src
│   ├── commands
│   │   ├── buy.py
│   │   ├── hello.py
│   │   ├── hello.pyc
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── __pycache__
│   │   └── start.py
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── packages.zip
│   └── test_python_zip.egg-info
│       ├── dependency_links.txt
│       ├── PKG-INFO
│       ├── SOURCES.txt
│       └── top_level.txt
└── test_python_zip.egg-info
    ├── dependency_links.txt
    ├── PKG-INFO
    ├── SOURCES.txt
    └── top_level.txt

最后,主要 .py文件的内容为:

import sys

if not ('packages.zip' in sys.path):
    sys.path.insert(0, 'packages.zip')
print(sys.version)
print("\n \n")
print(sys.path)
from src.commands import hello


def main():
    hello.call()


if __name__ == '__main__':
    main()

这里的问题是没有从package.zip文件中加载requirements依赖项,如果未安装,则出现以下错误:

3.7.3 (default, Apr  3 2019, 05:39:12)
[GCC 8.3.0]



['lib', 'dist/test_python_zip-1.0.linux-x86_64.zip', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/yborisov/.local/lib/python3.7/site-packages', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages']
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "dist/test_python_zip-1.0.linux-x86_64.zip/__main__.py", line 8, in <module>
  File "dist/test_python_zip-1.0.linux-x86_64.zip/src/commands/hello.py", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

应该是什么问题,我该如何解决?

1 个答案:

答案 0 :(得分:0)

我没有在最终的zip中使用 package.zip ,而是使用了 package 文件夹,该文件夹移至 src 软件包下。我在内容中添加了 MANIFEST.in

include  packages __main__.py
recursive-include  src/commands *

setup.py更改为

from distutils.command.bdist_dumb import bdist_dumb
from distutils.core import setup

from setuptools import find_packages


class CustomBdistDumb(bdist_dumb):

    def reinitialize_command(self, name, **kw):
        cmd = bdist_dumb.reinitialize_command(self, name, **kw)
        if name == 'install':
            cmd.install_lib = '/'
        return cmd


setup(
    # our custom class override
    cmdclass={'bdist_dumb': CustomBdistDumb},
    name='test_python_zip',
    version="1.0",
    package_data={
        '': ['packages']
    },
    py_modules=['__main__'],
    packages=find_packages(),
)

request 模块直接从 packages 文件夹中加载:

from src.packages import requests

URL = "https://api.ipify.org/?format=json"

def call():
    # sending get request and saving the response as response object
    r = requests.get(url=URL)

    # extracting data in json format
    data = r.json()
    print(data)