Packaging Native Extensions with Poetry

Poetry

Poetry is a modern dependency management and packaging tool for use with python libraries.

Because poetry has its own package building process, it can be difficult to determine how to package libraries that make use of native extensions or have other install-time logic that would traditionally be included in the project’s setup.py file.

Below, I’ll describe an approach based on this github issue that can be used to integrate native extensions into poetry’s custom build system.

Customizing Poetry’s Build Process

To modify the build process, you just need to make a few changes to the pyproject.toml, and create a python file used by poetry to modify its call to setuptools.setup.

pyproject.toml

To customize the build process, you need to add the following to your pyproject.toml:

• Add the build key to the [tool.poetry] section of your pyproject.toml

  • The value should be the filename of a python file that will contain the custom build keys.
  • Typically, this file is called build.py and is placed in the project root. See below for more detail.

[tool.poetry]
name = "my_pkg"
version = "0.1.0"
description = "My Package with C++ Extensions"
# ... Other keys in [tool.poetry] ...
build = "build.py"  # <-------------------------------

[build-system]
build-backend = "poetry.masonry.api"
requires = ["poetry>=0.12", "wheel", "setuptools-cpp"]

# ... Other sections ..

• Add the [build-system] section with keys:

  • build-backend = "poetry.masonry.api"

  • requires, a list containing at least "poetry>=0.12"

    • If you require other dependencies to build your extensions, you should also add them here

[tool.poetry]
name = "my_pkg"
version = "0.1.0"
description = "My Package with C++ Extensions"
# ... Other keys in [tool.poetry] ...
build = "build.py"

[build-system]  # <-----------------------------------
build-backend = "poetry.masonry.api"
requires = ["poetry>=0.12", "wheel", "setuptools-cpp"]

# ... Other sections ..

build.py

In pyproject.toml, we added a reference to a python file to be used by poetry’s build process.

This file should define a function called build, which accepts a dict containing the keyword arguments that will be be passed to setuptools.setup. The build function should then modify this dict in place as appropriate for your build process.

Here is an example build.py that could be used to build C++ extensions using my setuptools-cpp package:

# build.py

from typing import Any, Dict

from setuptools_cpp import CMakeExtension, ExtensionBuilder, Pybind11Extension

ext_modules = [
    # A basic pybind11 extension in <project_root>/src/ext1:
    Pybind11Extension(
        "my_pkg.ext1", ["src/ext1/ext1.cpp"], include_dirs=["src/ext1/include"]
    ),
    # An extension with a custom <project_root>/src/ext2/CMakeLists.txt:
    CMakeExtension(f"my_pkg.ext2", sourcedir="src/ext2"),
]


def build(setup_kwargs: Dict[str, Any]) -> None:
    setup_kwargs.update(
        {
            "ext_modules": ext_modules,
            "cmdclass": dict(build_ext=ExtensionBuilder),
            "zip_safe": False,
        }
    )

To build native extensions, you generally need to add the key "ext_modules" with a value that is a list of subclasses of setuptools.Extension. You also need to add "zip_safe": False to ensure a platform-specific wheel is created.

Note that when including native extensions, you may want to build (and publish) prebuilt wheels for your package. Otherwise, consumers of your package will need to build from source, which can add challenges with compilation-related dependencies.

If you publish pre-built wheels, be aware that they are platform-specific and must be built separately for each platform you intend to support (e.g., win, macosx, manylinux).