包装有numpy和测试套件

时间：2013-10-10 13:40:22

标签： python numpy fortran distutils f2py

简介

免责声明：我对使用distutils的python包装非常陌生。到目前为止，我只是把所有东西都藏进了模块，手工打包并在此基础上开发。我之前从未写过setup.py文件。

我有一个Fortran模块，我想在我的python代码中使用numpy。我认为最好的方法是f2py，因为它包含在numpy中。为了自动化构建过程，我想使用distutils和相应的numpy增强功能，其中包括f2py包装器的便捷功能。

我不明白我应该如何组织文件，以及如何包含我的测试套件。

我想要的是使用./setup.py进行构建，安装，测试和开发的可能性。

我的目录结构如下：

volterra
├── setup.py
└── volterra
    ├── __init__.py
    ├── integral.f90
    ├── test
    │   ├── __init__.py
    │   └── test_volterra.py
    └── volterra.f90

setup.py文件包含：

def configuration(parent_package='', top_path=None):
    from numpy.distutils.misc_util import Configuration
    config = Configuration('volterra', parent_package, top_path)
    config.add_extension('_volterra',
                         sources=['volterra/integral.f90', 'volterra/volterra.f90'])
    return config


if __name__ == '__main__':
    from numpy.distutils.core import setup
    setup(**configuration(top_path='').todict())

运行./setup.py build后，我得到了。

build/lib.linux-x86_64-2.7/
└── volterra
    └── _volterra.so

既不包含__init__.py文件，也不包括测试。

问题

  • 是否真的有必要将路径添加到扩展的每个源文件中？ （即volterra/integral.f90）我不能给出一个参数，在volterra/寻找东西吗？ top_path和package_dir参数无法解决问题。
  • 目前，__init__.py文件未包含在构建中。那是为什么？
  • 如何在此设置中运行测试？
  • 在这样的环境中进行开发的最佳工作流程是什么？对于我做的每一次更改，我都不想安装我的包。当您需要编译某些扩展模块时，如何在源目录中进行开发？

2 个答案:

答案 0 :(得分：2)

这是我制作的项目中的setup.py。我发现找出setup.py /包装是令人沮丧的，没有可靠的答案，绝对不是pythonic，只有一个而且只有一种明显的方法可以做某事。希望这会有所帮助。

您可能会发现有用的要点是：

  • find_packages可以消除包含大量文件的苦差事或者搞乱生成清单。
  • package_data，可让您轻松指定要包含的非.py文件
  • install_requires / tests_require

如果您还没有找到distribute_setup.py的来源，则需要找到它。

  
      
  • 是否真的有必要将路径添加到每个源文件中   延期？ （即.terterra / integral.f90）我不能给出参数   这说，在volterra中找东西/？ top_path和package_dir   参数不起作用。
    •   
  • 目前， init .py文件不是   包含在构建中。那是为什么？
    •   

希望find_packages()能解决这两个问题。我的包装经验不多，但我还没有回到手工包装。

  
      
  • 如何在此运行我的测试   设置？
    •   

我认为这可能是一个不同的问题，许多答案取决于你如何进行测试。也许你可以单独问它？

作为旁注，我的印象是标准是将您的测试目录放在顶层。即volterra/volterra和volterra/tests。

  
      
  • 在这样的情况下进行开发的最佳工作流程是什么？   环境？我不想为每一个安装我的包   改变我做。如何在源目录中进行开发   你需要编译一些扩展模块吗？
    •   

这也许值得另一个问题。我不明白为什么你需要为每一次更改安装你的软件包。如果要上传软件包，只需将其安装在开发系统上（测试安装除外），直接从开发副本开始。也许我错过了一些东西，因为我不使用编译的扩展。

以下是示例

try:
    from setuptools import setup, find_packages
except ImportError:
    from distribute_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages


setup(
    # ... other stuff
    py_modules=['distribute_setup'],
    packages=find_packages(),
    package_data={'': ['*.png']},  # for me to include anything with png
    install_requires=['numpy', 'treenode', 'investigators'],
    tests_require=['mock', 'numpy', 'treenode', 'investigators'],
)

答案 1 :(得分：2)

这是一个适合我的setup.py：

# pkg - A fancy software package
# Copyright (C) 2013  author (email)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see http://www.gnu.org/licenses/gpl.html.
"""pkg: a software suite for 

Hey look at me I'm a long description
But how long am I?

"""

from __future__ import division, print_function

#ideas for setup/f2py came from:
#    -numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
#    -winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
#    -needing to use 
#        import setuptools; from numpy.distutils.core import setup, Extension: 
#        http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
#    -wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
#    -numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
#    -manifest files in disutils: 
#        'distutils doesn't properly update MANIFEST. when the contents of directories change.'
#        https://github.com/numpy/numpy/blob/master/setup.py         
#    -if things are not woring try deleting build, sdist, egg directories  and try again: 
#        https://stackoverflow.com/a/9982133/2530083 2013-11-07
#    -getting fortran extensions to be installed in their appropriate sub package
#        i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])" 
#        Note that sources is a list even if one file: 
#        http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
#    -install fortran source files into their appropriate sub-package 
#        i.e. "package_data={'': ['*.f95','*.f90']}# Note it's a dict and list":
#        https://stackoverflow.com/a/19373744/2530083 2013-11-07
#    -Chapter 9 Fortran Programming with NumPy Arrays: 
#        Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
#    -Hitchhikers guide to packaging :
#        http://guide.python-distribute.org/
#    -Python Packaging: Hate, hate, hate everywhere : 
#        http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
#    -How To Package Your Python Code: 
#        http://www.scotttorborg.com/python-packaging/
#    -install testing requirements: 
#        https://stackoverflow.com/a/7747140/2530083 2013-11-07

import setuptools
from numpy.distutils.core import setup, Extension
import os
import os.path as osp

def readme(filename='README.rst'):
    with open('README.rst') as f:
        text=f.read()
    f.close()
    return text

def get_package_data(name, extlist):
    """Return data files for package *name* with extensions in *extlist*"""
    #modified slightly from taken from http://code.google.com/p/winpython/source/browse/setup.py 2013-11-7
    flist = []
    # Workaround to replace os.path.relpath (not available until Python 2.6):
    offset = len(name)+len(os.pathsep)
    for dirpath, _dirnames, filenames in os.walk(name):
        for fname in filenames:            
            if not fname.startswith('.') and osp.splitext(fname)[1] in extlist:
#                flist.append(osp.join(dirpath, fname[offset:]))
                flist.append(osp.join(dirpath, fname))
    return flist

DOCLINES = __doc__.split("\n")
CLASSIFIERS = """\
Development Status :: 1 - Planning
License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering
"""

NAME = 'pkg'
MAINTAINER = "me"
MAINTAINER_EMAIL = "me@me.com"
DESCRIPTION = DOCLINES[0]
LONG_DESCRIPTION = "\n".join(DOCLINES[2:])#readme('readme.rst')
URL = "http://meeeee.mmemem"
DOWNLOAD_URL = "https://github.com/rtrwalker/geotecha.git"
LICENSE = 'GNU General Public License v3 or later (GPLv3+)'
CLASSIFIERS = [_f for _f in CLASSIFIERS.split('\n') if _f]
KEYWORDS=''
AUTHOR = "me"
AUTHOR_EMAIL = "me.com"
PLATFORMS = ["Windows"]#, "Linux", "Solaris", "Mac OS-X", "Unix"]
MAJOR = 0
MINOR = 1
MICRO = 0
ISRELEASED = False
VERSION = '%d.%d.%d' % (MAJOR, MINOR, MICRO)

INSTALL_REQUIRES=[]
ZIP_SAFE=False
TEST_SUITE='nose.collector'
TESTS_REQUIRE=['nose']

DATA_FILES = [(NAME, ['LICENSE.txt','README.rst'])]
PACKAGES=setuptools.find_packages()
PACKAGES.remove('tools')

PACKAGE_DATA={'': ['*.f95','*f90']}               
ext_files = get_package_data(NAME,['.f90', '.f95','.F90', '.F95'])
ext_module_names = ['.'.join(osp.splitext(v)[0].split(osp.sep)) for v in ext_files]
EXT_MODULES = [Extension(name=x,sources=[y]) for x, y in zip(ext_module_names, ext_files)]      


setup(
    name=NAME,
    version=VERSION,
    maintainer=MAINTAINER,
    maintainer_email=MAINTAINER_EMAIL,
    description=DESCRIPTION,
    long_description=LONG_DESCRIPTION,
    url=URL,
    download_url=DOWNLOAD_URL,
    license=LICENSE,
    classifiers=CLASSIFIERS,
    author=AUTHOR,
    author_email=AUTHOR_EMAIL,
    platforms=PLATFORMS,
    packages=PACKAGES,
    data_files=DATA_FILES,
    install_requires=INSTALL_REQUIRES,
    zip_safe=ZIP_SAFE,
    test_suite=TEST_SUITE,
    tests_require=TESTS_REQUIRE,
    package_data=PACKAGE_DATA,    
    ext_modules=EXT_MODULES,
    )

要安装，请在命令行中使用：

python setup.py install
python setup.py clean --all

我似乎唯一的问题是一个小问题。当我查看我的包的site-packages时，它安装在egg文件夹C:\Python27\Lib\site-packages\pkg-0.1.0-py2.7-win32.egg\pkg中。我看到的大多数其他软件包都有一个与egg文件夹分开的C:\Python27\Lib\site-packages\pkg文件夹。有谁知道如何分离？

对于测试，安装后，我在命令行输入以下内容：

nosetests package_name -v

尝试调查python setup.py develop（Python setup.py develop vs install），以便在每次更改后都不必安装软件包。

正如我在代码中评论的那样，我发现以下内容非常有用：

相关问题
最新问题