source_files_and_compilation.rst 9.96 KB
Newer Older
Robert Bradshaw's avatar
Robert Bradshaw committed
1 2 3 4 5 6 7 8 9 10 11 12
.. highlight:: cython

.. _compilation:

****************************
Source Files and Compilation
****************************

Cython source file names consist of the name of the module followed by a
``.pyx`` extension, for example a module called primes would have a source
file named :file:`primes.pyx`.

13 14 15 16 17 18 19
Cython code, unlike Python, must be compiled.  This happens in two stages:

  * A ``.pyx`` file is compiled by Cython to a ``.c`` file.

  * The ``.c`` file is compiled by a C compiler to a ``.so`` file (or a
    ``.pyd`` file on Windows)

Robert Bradshaw's avatar
Robert Bradshaw committed
20
Once you have written your ``.pyx`` file, there are a couple of ways of turning it
21 22 23 24 25 26 27 28 29
into an extension module.

The following sub-sections describe several ways to build your
extension modules, and how to pass directives to the Cython compiler.

Compiling from the command line
===============================

One way is to compile it manually with the Cython
Robert Bradshaw's avatar
Robert Bradshaw committed
30 31 32 33 34 35 36 37 38 39 40 41
compiler, e.g.:

.. sourcecode:: text

    $ cython primes.pyx

This will produce a file called :file:`primes.c`, which then needs to be
compiled with the C compiler using whatever options are appropriate on your
platform for generating an extension module. For these options look at the
official Python documentation.

The other, and probably better, way is to use the :mod:`distutils` extension
Jakub Wilk's avatar
Jakub Wilk committed
42
provided with Cython. The benefit of this method is that it will give the
Robert Bradshaw's avatar
Robert Bradshaw committed
43 44
platform specific compilation options, acting like a stripped down autotools.

45

Robert Bradshaw's avatar
Robert Bradshaw committed
46 47 48 49 50 51 52 53 54 55
Basic setup.py
===============
The distutils extension provided with Cython allows you to pass ``.pyx`` files
directly to the ``Extension`` constructor in your setup file.

If you have a single Cython file that you want to turn into a compiled
extension, say with filename :file:`example.pyx` the associated :file:`setup.py`
would be::

    from distutils.core import setup
56
    from Cython.Build import cythonize
Robert Bradshaw's avatar
Robert Bradshaw committed
57 58

    setup(
59
        ext_modules = cythonize("example.pyx")
mathbunnyru's avatar
mathbunnyru committed
60
    )
Robert Bradshaw's avatar
Robert Bradshaw committed
61 62 63 64 65 66 67 68 69

To understand the :file:`setup.py` more fully look at the official
:mod:`distutils` documentation. To compile the extension for use in the
current directory use:

.. sourcecode:: text

    $ python setup.py build_ext --inplace

70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227
Configuring the C-Build
------------------------

If you have include files in non-standard places you can pass an
``include_path`` parameter to ``cythonize``::

    from distutils.core import setup
    from Cython.Build import cythonize

    setup(
        name="My hello app",
        ext_modules=cythonize("src/*.pyx", include_path=[...]),
    )

Often, Python packages that offer a C-level API provide a way to find
the necessary include files, e.g. for NumPy::

    include_path = [numpy.get_include()]

.. note::

    Using memoryviews or importing NumPy with ``import numpy`` does not mean that
    you have to add the path to NumPy include files. You need to add this path only
    if you use ``cimport numpy``.

Despite this, you will still get warnings like the
following from the compiler, because Cython is using a deprecated Numpy API::

   .../include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

For the time being, it is just a warning that you can ignore.

If you need to specify compiler options, libraries to link with or other
linker options you will need to create ``Extension`` instances manually
(note that glob syntax can still be used to specify multiple extensions
in one line)::

    from distutils.core import setup
    from distutils.extension import Extension
    from Cython.Build import cythonize

    extensions = [
        Extension("primes", ["primes.pyx"],
            include_dirs=[...],
            libraries=[...],
            library_dirs=[...]),
        # Everything but primes.pyx is included here.
        Extension("*", ["*.pyx"],
            include_dirs=[...],
            libraries=[...],
            library_dirs=[...]),
    ]
    setup(
        name="My hello app",
        ext_modules=cythonize(extensions),
    )

Note that when using setuptools, you should import it before Cython as
setuptools may replace the ``Extension`` class in distutils.  Otherwise,
both might disagree about the class to use here.

Note also that if you use setuptools instead of distutils, the default
action when running ``python setup.py install`` is to create a zipped
``egg`` file which will not work with ``cimport`` for ``pxd`` files
when you try to use them from a dependent package.
To prevent this, include ``zip_safe=False`` in the arguments to ``setup()``.

If your options are static (for example you do not need to call a tool like
``pkg-config`` to determine them) you can also provide them directly in your
.pyx or .pxd source file using a special comment block at the start of the file::

    # distutils: libraries = spam eggs
    # distutils: include_dirs = /opt/food/include

If you cimport multiple .pxd files defining libraries, then Cython
merges the list of libraries, so this works as expected (similarly
with other options, like ``include_dirs`` above).

If you have some C files that have been wrapped with Cython and you want to
compile them into your extension, you can define the distutils ``sources``
parameter::

    # distutils: sources = helper.c, another_helper.c

Note that these sources are added to the list of sources of the current
extension module.  Spelling this out in the :file:`setup.py` file looks
as follows::

    from distutils.core import setup
    from Cython.Build import cythonize
    from distutils.extension import Extension

    sourcefiles = ['example.pyx', 'helper.c', 'another_helper.c']

    extensions = [Extension("example", sourcefiles)]

    setup(
        ext_modules=cythonize(extensions)
    )

The :class:`Extension` class takes many options, and a fuller explanation can
be found in the `distutils documentation`_. Some useful options to know about
are ``include_dirs``, ``libraries``, and ``library_dirs`` which specify where
to find the ``.h`` and library files when linking to external libraries.

.. _distutils documentation: https://docs.python.org/extending/building.html

Sometimes this is not enough and you need finer customization of the
distutils :class:`Extension`.
To do this, you can provide a custom function ``create_extension``
to create the final :class:`Extension` object after Cython has processed
the sources, dependencies and ``# distutils`` directives but before the
file is actually Cythonized.
This function takes 2 arguments ``template`` and ``kwds``, where
``template`` is the :class:`Extension` object given as input to Cython
and ``kwds`` is a :class:`dict` with all keywords which should be used
to create the :class:`Extension`.
The function ``create_extension`` must return a 2-tuple
``(extension, metadata)``, where ``extension`` is the created
:class:`Extension` and ``metadata`` is metadata which will be written
as JSON at the top of the generated C files. This metadata is only used
for debugging purposes, so you can put whatever you want in there
(as long as it can be converted to JSON).
The default function (defined in ``Cython.Build.Dependencies``) is::

    def default_create_extension(template, kwds):
        if 'depends' in kwds:
            include_dirs = kwds.get('include_dirs', []) + ["."]
            depends = resolve_depends(kwds['depends'], include_dirs)
            kwds['depends'] = sorted(set(depends + template.depends))

        t = template.__class__
        ext = t(**kwds)
        metadata = dict(distutils=kwds, module_name=kwds['name'])
        return ext, metadata

In case that you pass a string instead of an :class:`Extension` to
``cythonize()``, the ``template`` will be an :class:`Extension` without
sources. For example, if you do ``cythonize("*.pyx")``,
the ``template`` will be ``Extension(name="*.pyx", sources=[])``.

Just as an example, this adds ``mylib`` as library to every extension::

    from Cython.Build.Dependencies import default_create_extension

    def my_create_extension(template, kwds):
        libs = kwds.get('libraries', []) + ["mylib"]
        kwds['libraries'] = libs
        return default_create_extension(template, kwds)

    ext_modules = cythonize(..., create_extension=my_create_extension)

.. note::

    If you Cythonize in parallel (using the ``nthreads`` argument),
    then the argument to ``create_extension`` must be pickleable.
    In particular, it cannot be a lambda function.

228

Robert Bradshaw's avatar
Robert Bradshaw committed
229 230 231
Multiple Cython Files in a Package
===================================

232 233 234 235 236 237 238 239 240 241
To automatically compile multiple Cython files without listing all of them
explicitly, you can use glob patterns::

    setup(
        ext_modules = cythonize("package/*.pyx")
    )

You can also use glob patterns in :class:`Extension` objects if you pass
them through :func:`cythonize`::

242
    extensions = [Extension("*", ["*.pyx"])]
243 244 245 246 247

    setup(
        ext_modules = cythonize(extensions)
    )

Robert Bradshaw's avatar
Robert Bradshaw committed
248 249 250 251

Pyximport
===========

252 253 254 255 256
Cython is a compiler.  Therefore it is natural that people tend to go
through an edit/compile/test cycle with Cython modules.  :mod:`pyximport`
simplifies this process by executing the "compile" step at need during
import.  For instance, if you write a Cython module called :file:`foo.pyx`,
with Pyximport you can import it in a regular Python module like this::
Robert Bradshaw's avatar
Robert Bradshaw committed
257 258 259 260 261 262 263 264


    import pyximport; pyximport.install()
    import foo

Doing so will result in the compilation of :file:`foo.pyx` (with appropriate
exceptions if it has an error in it).

265 266 267 268
If you would always like to import Cython files without building them specially,
you can also add the first line above to your :file:`sitecustomize.py`.
That will install the hook every time you run Python.  Then you can use
Cython modules just with simple import statements, even like this:
Robert Bradshaw's avatar
Robert Bradshaw committed
269 270 271 272 273

.. sourcecode:: text

    $ python -c "import foo"

274 275 276 277 278
Note that it is not recommended to let :mod:`pyximport` build code
on end user side as it hooks into their import system.  The best way
to cater for end users is to provide pre-built binary packages in the
`wheel <https://wheel.readthedocs.io/>`_ packaging format.

279
To have more information of :mod:`pyximport`, please refer to :ref:`pyximport`.