我正在编写一个链接C ++库的Python扩展，我正在使用cmake来帮助构建过程。这意味着现在，我知道如何捆绑它的唯一方法，我必须先用cmake编译它们才能运行setup.py bdist_wheel。必须有更好的方法。
我想知道是否有可能（或任何人尝试过）调用CMake作为setup.py ext_modules构建过程的一部分？我猜有一种方法来创建一个东西的子类但我不知道在哪里看。
我正在使用CMake，因为它为我提供了更多控制，可以根据我的需要构建具有复杂构建步骤的c和c ++库扩展。另外，我可以使用findPythonLibs.cmake中的PYTHON_ADD_MODULE（）命令直接使用cmake构建Python扩展。我只希望这一步都是一步。
答案 0 :(得分：15)
您基本上需要做的是覆盖build_ext中的setup.py命令类，并在命令类中注册它。在build_ext的自定义impl中，配置并调用cmake以配置然后构建扩展模块。不幸的是，官方文档对于如何实现自定义distutils命令非常简洁（参见Extending Distutils）;我发现直接研究命令代码会更有帮助。例如，以下是build_ext command的源代码。
我准备了一个简单的项目，它由一个C扩展foo和一个python模块spam.eggs组成：
so-42585210/
├── spam
│ ├── __init__.py # empty
│ ├── eggs.py
│ ├── foo.c
│ └── foo.h
├── CMakeLists.txt
└── setup.py
这些只是我为测试安装脚本而编写的一些简单存根。
spam/eggs.py（仅用于测试库调用）：
from ctypes import cdll
import pathlib
def wrap_bar():
foo = cdll.LoadLibrary(str(pathlib.Path(__file__).with_name('libfoo.dylib')))
return foo.bar()
spam/foo.c：
#include "foo.h"
int bar() {
return 42;
}
spam/foo.h：
#ifndef __FOO_H__
#define __FOO_H__
int bar();
#endif
CMakeLists.txt：
cmake_minimum_required(VERSION 3.10.1)
project(spam)
set(src "spam")
set(foo_src "spam/foo.c")
add_library(foo SHARED ${foo_src})
这就是魔术发生的地方。当然，还有很大的改进空间 - 如果需要，您可以将其他选项传递给CMakeExtension课程（有关扩展的更多信息，请参阅Building C and C++ Extensions），通过以下方式配置CMake选项setup.cfg覆盖方法initialize_options和finalize_options等。
import os
import pathlib
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext as build_ext_orig
class CMakeExtension(Extension):
def __init__(self, name):
# don't invoke the original build_ext for this special extension
super().__init__(name, sources=[])
class build_ext(build_ext_orig):
def run(self):
for ext in self.extensions:
self.build_cmake(ext)
super().run()
def build_cmake(self, ext):
cwd = pathlib.Path().absolute()
# these dirs will be created in build_py, so if you don't have
# any python sources to bundle, the dirs will be missing
build_temp = pathlib.Path(self.build_temp)
build_temp.mkdir(parents=True, exist_ok=True)
extdir = pathlib.Path(self.get_ext_fullpath(ext.name))
extdir.mkdir(parents=True, exist_ok=True)
# example of cmake args
config = 'Debug' if self.debug else 'Release'
cmake_args = [
'-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + str(extdir.parent.absolute()),
'-DCMAKE_BUILD_TYPE=' + config
]
# example of build args
build_args = [
'--config', config,
'--', '-j4'
]
os.chdir(str(build_temp))
self.spawn(['cmake', str(cwd)] + cmake_args)
if not self.dry_run:
self.spawn(['cmake', '--build', '.'] + build_args)
os.chdir(str(cwd))
setup(
name='spam',
version='0.1',
packages=['spam'],
ext_modules=[CMakeExtension('spam/foo')],
cmdclass={
'build_ext': build_ext,
}
)
构建项目的轮子，安装它。测试库已安装：
$ pip show -f spam
Name: spam
Version: 0.1
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages
Requires:
Files:
spam-0.1.dist-info/DESCRIPTION.rst
spam-0.1.dist-info/INSTALLER
spam-0.1.dist-info/METADATA
spam-0.1.dist-info/RECORD
spam-0.1.dist-info/WHEEL
spam-0.1.dist-info/metadata.json
spam-0.1.dist-info/top_level.txt
spam/__init__.py
spam/__pycache__/__init__.cpython-36.pyc
spam/__pycache__/eggs.cpython-36.pyc
spam/eggs.py
spam/libfoo.dylib
从spam.eggs模块运行包装函数：
$ python -c "from spam import eggs; print(eggs.wrap_bar())"
42
答案 1 :(得分：6)
我想对此做出自己的回答，作为对霍夫林描述的一种补充。
谢谢，因为您的回答使我朝着自己的存储库以几乎相同的方式编写设置脚本的方向前进。
编写此答案的主要动机是试图“粘合”缺失的部分。 OP没有说明正在开发的C / C ++ Python模块的性质；我想先说清楚以下步骤适用于C / C ++ cmake构建链，该链创建多个.dll / .so文件以及一个预编译的*.pyd / { {1}}文件以及一些需要放置在脚本目录中的常规so文件。
所有这些文件在执行cmake build命令后直接实现。很有趣。不建议以这种方式构建setup.py。
因为setup.py意味着您的脚本将成为软件包/库的一部分，并且必须通过库部分声明需要构建的.py文件，并列出源和包含目录，没有一种直观的方式告诉setuptools，发生在.dll中的一次对cmake -b的调用所产生的库，脚本和数据文件都应该放在各自的位置。更糟糕的是，如果您想让setuptools跟踪此模块并且该模块可以完全卸载，这意味着用户可以卸载它，并在需要时清除所有跟踪信息。
我正在为其编写setup.py的模块是bpy，相当于build_ext / .pyd相当于将搅拌器构建为python模块，如下所述：
https://wiki.blender.org/wiki//User:Ideasman42/BlenderAsPyModule（更好的说明，但现在无效链接） http://www.gizmoplex.com/wordpress/compile-blender-as-python-module/（说明可能更糟，但似乎仍然在线）
您可以在github上查看我的存储库：
https://github.com/TylerGubala/blenderpy
这是我编写此答案的动机，希望能帮助其他尝试完成类似任务的人，而不是丢掉他们的cmake构建链，或者更糟糕的是，必须维护两个单独的构建环境。抱歉，这很抱歉。
用自己的类扩展.so类，该类不包含source或libs属性的条目
用自己的类扩展setuptools.Extension类，该类具有一个自定义方法，该方法执行我必需的构建步骤（git，svn，cmake，cmake --build）
用我自己的类扩展了setuptools.commands.build_ext.build_ext类（y，distutils.command.install_data.install_data ...但是似乎没有等效的setuputils），以标记所构建的二进制库在setuptools的记录创建过程中（installed-files.txt），例如
库将被记录，并使用distutils
命令pip
uninstall package_name也将在本机运行，并且
可用于提供源代码的预编译版本
用我自己的类扩展py setup.py bdist_wheel类，这将确保已构建的库从其生成的构建文件夹移至setuptools期望它们进入的文件夹中（在Windows上，它将放置setuptools.command.install_lib.install_lib文件位于bin / Release文件夹中，而不是setuptools期望的位置）
用我自己的类扩展.dll类，以便将脚本文件复制到正确的目录（Blender期望2.79或任何目录位于脚本位置）
执行构建步骤后，将那些文件复制到一个已知目录中，setuptools将其复制到我的环境的site-packages目录中。此时，其余的setuptools和distutils类可以接管写入installed-files.txt记录，并且可以完全删除！
这里是一个样本，或多或少是从我的存储库中获取的，但为了更清晰地进行了修剪而进行了修剪（您可以随时访问存储库并亲自查看它）
setuptools.command.install_scripts.install_scripts以这种方式编写from distutils.command.install_data import install_data
from setuptools import find_packages, setup, Extension
from setuptools.command.build_ext import build_ext
from setuptools.command.install_lib import install_lib
from setuptools.command.install_scripts import install_scripts
import struct
BITS = struct.calcsize("P") * 8
PACKAGE_NAME = "example"
class CMakeExtension(Extension):
"""
An extension to run the cmake build
This simply overrides the base extension class so that setuptools
doesn't try to build your sources for you
"""
def __init__(self, name, sources=[]):
super().__init__(name = name, sources = sources)
class InstallCMakeLibsData(install_data):
"""
Just a wrapper to get the install data into the egg-info
Listing the installed files in the egg-info guarantees that
all of the package files will be uninstalled when the user
uninstalls your package through pip
"""
def run(self):
"""
Outfiles are the libraries that were built using cmake
"""
# There seems to be no other way to do this; I tried listing the
# libraries during the execution of the InstallCMakeLibs.run() but
# setuptools never tracked them, seems like setuptools wants to
# track the libraries through package data more than anything...
# help would be appriciated
self.outfiles = self.distribution.data_files
class InstallCMakeLibs(install_lib):
"""
Get the libraries from the parent distribution, use those as the outfiles
Skip building anything; everything is already built, forward libraries to
the installation step
"""
def run(self):
"""
Copy libraries from the bin directory and place them as appropriate
"""
self.announce("Moving library files", level=3)
# We have already built the libraries in the previous build_ext step
self.skip_build = True
bin_dir = self.distribution.bin_dir
# Depending on the files that are generated from your cmake
# build chain, you may need to change the below code, such that
# your files are moved to the appropriate location when the installation
# is run
libs = [os.path.join(bin_dir, _lib) for _lib in
os.listdir(bin_dir) if
os.path.isfile(os.path.join(bin_dir, _lib)) and
os.path.splitext(_lib)[1] in [".dll", ".so"]
and not (_lib.startswith("python") or _lib.startswith(PACKAGE_NAME))]
for lib in libs:
shutil.move(lib, os.path.join(self.build_dir,
os.path.basename(lib)))
# Mark the libs for installation, adding them to
# distribution.data_files seems to ensure that setuptools' record
# writer appends them to installed-files.txt in the package's egg-info
#
# Also tried adding the libraries to the distribution.libraries list,
# but that never seemed to add them to the installed-files.txt in the
# egg-info, and the online recommendation seems to be adding libraries
# into eager_resources in the call to setup(), which I think puts them
# in data_files anyways.
#
# What is the best way?
# These are the additional installation files that should be
# included in the package, but are resultant of the cmake build
# step; depending on the files that are generated from your cmake
# build chain, you may need to modify the below code
self.distribution.data_files = [os.path.join(self.install_dir,
os.path.basename(lib))
for lib in libs]
# Must be forced to run after adding the libs to data_files
self.distribution.run_command("install_data")
super().run()
class InstallCMakeScripts(install_scripts):
"""
Install the scripts in the build dir
"""
def run(self):
"""
Copy the required directory to the build directory and super().run()
"""
self.announce("Moving scripts files", level=3)
# Scripts were already built in a previous step
self.skip_build = True
bin_dir = self.distribution.bin_dir
scripts_dirs = [os.path.join(bin_dir, _dir) for _dir in
os.listdir(bin_dir) if
os.path.isdir(os.path.join(bin_dir, _dir))]
for scripts_dir in scripts_dirs:
shutil.move(scripts_dir,
os.path.join(self.build_dir,
os.path.basename(scripts_dir)))
# Mark the scripts for installation, adding them to
# distribution.scripts seems to ensure that the setuptools' record
# writer appends them to installed-files.txt in the package's egg-info
self.distribution.scripts = scripts_dirs
super().run()
class BuildCMakeExt(build_ext):
"""
Builds using cmake instead of the python setuptools implicit build
"""
def run(self):
"""
Perform build_cmake before doing the 'normal' stuff
"""
for extension in self.extensions:
if extension.name == 'example_extension':
self.build_cmake(extension)
super().run()
def build_cmake(self, extension: Extension):
"""
The steps required to build the extension
"""
self.announce("Preparing the build environment", level=3)
build_dir = pathlib.Path(self.build_temp)
extension_path = pathlib.Path(self.get_ext_fullpath(extension.name))
os.makedirs(build_dir, exist_ok=True)
os.makedirs(extension_path.parent.absolute(), exist_ok=True)
# Now that the necessary directories are created, build
self.announce("Configuring cmake project", level=3)
# Change your cmake arguments below as necessary
# Below is just an example set of arguments for building Blender as a Python module
self.spawn(['cmake', '-H'+SOURCE_DIR, '-B'+self.build_temp,
'-DWITH_PLAYER=OFF', '-DWITH_PYTHON_INSTALL=OFF',
'-DWITH_PYTHON_MODULE=ON',
f"-DCMAKE_GENERATOR_PLATFORM=x"
f"{'86' if BITS == 32 else '64'}"])
self.announce("Building binaries", level=3)
self.spawn(["cmake", "--build", self.build_temp, "--target", "INSTALL",
"--config", "Release"])
# Build finished, now copy the files into the copy directory
# The copy directory is the parent directory of the extension (.pyd)
self.announce("Moving built python module", level=3)
bin_dir = os.path.join(build_dir, 'bin', 'Release')
self.distribution.bin_dir = bin_dir
pyd_path = [os.path.join(bin_dir, _pyd) for _pyd in
os.listdir(bin_dir) if
os.path.isfile(os.path.join(bin_dir, _pyd)) and
os.path.splitext(_pyd)[0].startswith(PACKAGE_NAME) and
os.path.splitext(_pyd)[1] in [".pyd", ".so"]][0]
shutil.move(pyd_path, extension_path)
# After build_ext is run, the following commands will run:
#
# install_lib
# install_scripts
#
# These commands are subclassed above to avoid pitfalls that
# setuptools tries to impose when installing these, as it usually
# wants to build those libs and scripts as well or move them to a
# different place. See comments above for additional information
setup(name='my_package',
version='1.0.0a0',
packages=find_packages(),
ext_modules=[CMakeExtension(name="example_extension")],
description='An example cmake extension module',
long_description=open("./README.md", 'r').read(),
long_description_content_type="text/markdown",
keywords="test, cmake, extension",
classifiers=["Intended Audience :: Developers",
"License :: OSI Approved :: "
"GNU Lesser General Public License v3 (LGPLv3)",
"Natural Language :: English",
"Programming Language :: C",
"Programming Language :: C++",
"Programming Language :: Python",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: Implementation :: CPython"],
license='GPL-3.0',
cmdclass={
'build_ext': BuildCMakeExt,
'install_data': InstallCMakeLibsData,
'install_lib': InstallCMakeLibs,
'install_scripts': InstallCMakeScripts
}
)
之后，构建python模块就像运行setup.py一样简单，它将运行构建并生成输出文件。
建议您为慢速互联网上的用户或不想从源代码构建的用户制作一个转轮。为此，您将需要安装py setup.py软件包（wheel）并通过执行py -m pip install wheel产生车轮分布，然后像其他任何软件包一样使用py setup.py bdist_wheel上传