Distributing a CFFI Project Redux

29 June 2015

A little over six months ago, I wrote about how to enable sane distribution of CFFI. That previous post contained a number of work arounds and hacks to deal with a single design decision in CFFI, namely that it would implicitly invoke a compiler to compile your module and that was a core part of the API. A little over a month ago, CFFI 1.0 was released which offered new APIs which changed that assumption and offered better integration with setuptools. There are still a few things to keep in mind while writing a CFFI using module to enable easy and sane distribution, however it is now much easier to do so.

I’m going to adapt the original examples from my previous post to use the new APIs so we can see how it’s changed, and what the new best way of distributing a CFFI based project is.

Minimal Example

Here is a minimal example of using CFFI to be able to call the printf function from Python:

# This file should be saved as example_build.py
from cffi import FFI

ffi = FFI()

ffi.cdef(
    """
    int printf(const char *format, ...);
    """
)

ffi.set_source(
    "_example",  # This is the name of the import that this will build.
    """
    #include <stdio.h>
    """
)


if __name__ == '__main__':
    ffi.compile()

# This file should be saved as example.py
from _example import ffi, lib

if __name__ == "__main__":
    lib.printf(b"Hi There!\n")

This example works, and if you save both files into your current directory you can verify it by running:

$ python example_build.py
$ python example.py
Hi There!

This works because when you first execute example_build.py it will construct a FFI object at the module scope, and then execute the compile() method on that FFI object. This will cause CFFI to compile the _example.so module which is a standard Python extension module that you can simply import. This can let you quickly and easily write simple modules with a minimal amount of overhead.

Packaging our Example Project

Now that we have a simple example.py file we can package this up so that we can distribute it to other people. We’ll use a simple setup.py taken from the CFFI docs with some slight modifications to fit our project:

from setuptools import setup

setup(
    name="example",
    version="0.1",
    py_modules=["example"],
    setup_requires=["cffi>=1.0.0"],
    install_requires=["cffi>=1.0.0"],
    cffi_modules=["example_build.py:ffi"],
)

Now that we have our setup.py we can go ahead and create a sdist using the command python setup.py sdist which will give us example-0.1.tar.gz in the dist/ folder. We can even publish it to PyPI and then let other users install it using pip install example!

Right about here is where my previous post started to layer more and more hacks ontop of everything in order to restore some sanity to distributing. The good news is CFFI 1.0 fixed all of this and we’re already done! People installing this distribution will require a few system dependencies like the Python development headers and libffi and its development headers however there is no longer a need for all of the layers of monkeypatching and hacks.

The one really subtle thing I would point out here that isn’t obvious in our example, is that you should not install the build scripts. When you’re simply shipping a single .py file (such as in the example) then you can handle this by simply not adding the example_build.py (or whatever name your script has) to the py_modules list. However if you’re instead packaging an importable package (e.g. modules inside of a directory) then you would instead want do something like this:

# This should be saved as _cffi_build/example_build.py
from cffi import FFI

ffi = FFI()

ffi.cdef(
    """
    int printf(const char *format, ...);
    """
)

ffi.set_source(
    "_example",  # This is the name of the import that this will build.
    """
    #include <stdio.h>
    """
)

# This should be saved as example/__init__.py
from example._example import ffi, lib

if __name__ == "__main__":
    lib.printf(b"Hi There!\n")

Then, we can have a setup.py that looks something like:

from setuptools import find_packages, setup

setup(
    name="example",
    version="0.1",
    packages=find_packages(exclude=["_cffi_build", "_cffi_build.*"]),
    setup_requires=["cffi>=1.0.0"],
    install_requires=["cffi>=1.0.0"],
    ext_package="example",
    cffi_modules=["_cffi_build/example_build.py:ffi"],
)

This will have the same outcome as the first example, you’ll get the example project installed without installing the build script.

Bonus: “Better” setup_requires

Sadly, a better CFFI still doesn’t solve the issues around setup.py and setuptools, particularly that the setup.py as written above will install CFFI and all of its dependencies for any invocation of setup.py, even just for printing out the usage information with python setup.py --help. The setup_requires dependencies exist there to allow CFFI to introduce the cffi_modules keyword, however setuptools doesn’t know in which cases you actually want to install the setup_requires and in which cases they are superflous, so it just always installs them.

We can limit this so that setuptools will only install CFFI if required, however it requires adding more logic to our setup.py. This isn’t strictly required though users may appreciate being able to query information from the setup.py without downloading and installing CFFI.

To do this we’ll create a function that will inspect the arguments that setup.py was called with and determine if any of them are invoking something which will require CFFI in setup_requires. This function can then add additional keyword arguments to the setup() function call depending on if we need CFFI in the setup_requires or not.

This will create a setup.py that looks like:

import sys

from distutils.command.build import build
from setuptools import setup
from setuptools.command.install import install


SETUP_REQUIRES_ERROR = (
    "Requested setup command that needs 'setup_requires' while command line "
    "arguments implied a side effect free command or option."
)

NO_SETUP_REQUIRES_ARGUMENTS = [
    "-h", "--help",
    "-n", "--dry-run",
    "-q", "--quiet",
    "-v", "--verbose",
    "-v", "--version",
    "--author",
    "--author-email",
    "--classifiers",
    "--contact",
    "--contact-email",
    "--description",
    "--egg-base",
    "--fullname",
    "--help-commands",
    "--keywords",
    "--licence",
    "--license",
    "--long-description",
    "--maintainer",
    "--maintainer-email",
    "--name",
    "--no-user-cfg",
    "--obsoletes",
    "--platforms",
    "--provides",
    "--requires",
    "--url",
    "clean",
    "egg_info",
    "register",
    "sdist",
    "upload",
]


class DummyCFFIBuild(build):
    def run(self):
        raise RuntimeError(SETUP_REQUIRES_ERROR)


class DummyCFFIInstall(install):
    def run(self):
        raise RuntimeError(SETUP_REQUIRES_ERROR)


def keywords_with_side_effects(argv):
    def is_short_option(argument):
        """Check whether a command line argument is a short option."""
        return len(argument) >= 2 and argument[0] == '-' and argument[1] != '-'

    def expand_short_options(argument):
        """Expand combined short options into canonical short options."""
        return ('-' + char for char in argument[1:])

    def argument_without_setup_requirements(argv, i):
        """Check whether a command line argument needs setup requirements."""
        if argv[i] in NO_SETUP_REQUIRES_ARGUMENTS:
            # Simple case: An argument which is either an option or a command
            # which doesn't need setup requirements.
            return True
        elif (is_short_option(argv[i]) and
              all(option in NO_SETUP_REQUIRES_ARGUMENTS
                  for option in expand_short_options(argv[i]))):
            # Not so simple case: Combined short options none of which need
            # setup requirements.
            return True
        elif argv[i - 1:i] == ['--egg-base']:
            # Tricky case: --egg-info takes an argument which should not make
            # us use setup_requires (defeating the purpose of this code).
            return True
        else:
            return False

    if all(argument_without_setup_requirements(argv, i)
           for i in range(1, len(argv))):
        return {
            "cmdclass": {
                "build": DummyCFFIBuild,
                "install": DummyCFFIInstall,
            }
        }
    else:
        return {
            "setup_requires": ["cffi>=1.0.0"],
            "cffi_modules": ["example_build.py:ffi"],
        }


setup(
    name="example",
    version="0.1",
    py_modules=["example"],
    install_requires=["cffi>=1.0.0"],
    **keywords_with_side_effects(sys.argv)
)