Python is a powerful programming language with extensive library support. But what does one do when needing to integrate with a platform specific C or C++ component that has no native python support? There are two options. Completely rewrite the functionality in python or create a python extension. Either option can be painful and prone to errors. Enter Cython. It’s like the peanut butter and the jelly to the extension sandwich.
In this post, I will attempt to describe how excellent the Cython language is and will liken it to a PBJ sandwich. Like a PBJ sandwich, Cython code is easy to construct, satisfying to use, does not require exotic ingredients, and can be prepared by almost anyone. I will construct a simple python extension using Cython to show how delicious it is.
Cython
A python extension is code that adds a builtin module to python. Extension modules can do three things that are not possible directly in python: they can implement new built-in object types, can call C library functions, and can invoke system calls. The traditional way of constructing a python extension is to learn about the internal workings of the python C api and then use it. The Cython language sort circuits this process by allowing developers to use a python like scripting dialect to create an extension. It makes writing C or C++ extensions for the python language as easy as python itself.
The Bread
A typical PBJ sandwich has bread. In the case of a python extension, the C or C++ code and the python code that uses the extension is like the bread in a BBJ sandwich. For the sake of this post we will use a contrived trivial C++ class. In real world use cases, the C or C++ code is usually not trivial since such components could be easily implemented in python and there is usually a module already built. We will also use a contrived python script to exercise the python extension.
The Peanut Butter
This is the yummy layer of protein that gives substance to a PBJ sandwich. Cython is like the peanut butter. It provides yummy goodness that automates much of the grunt work in creating a python extension. When using Cython one has to be aware of the three Cython file types:
- “pxd” – Definition files that carry a .pxd suffix
- “pyx” – Implementation files that carry a .pyx suffix
- Include files which carry a .pxi suffix
To create a python extension to a C++ class, all one needs to do is provide an implementation for two of the file types: a "pxd" and a "pyx".
PXD Files
The "pxd" file works like a C header file – it contains the Cython declarations (and sometimes code sections) which are only meant for inclusion by Cython modules. A "pxd" file is imported into a pyx module by using the cimport keyword. Put another way, the "pxd" file defines the mapping interface between the native component (written in C or C++) and a python shim generated by cython. I will gloss over the actual Cython syntax but it is very expressive. In the example "pxd" file the AFancyWidget is declared with its methods GetName and IsCorrect. The methods GetName and IsCorrect can each throw exceptions. Example 1 illustrates a simple "pxd" file.
Example 1
# Copyright (C) 2017 Art and Logic.
#
# The Cython declarations that generate a python class wrapper around
# the AFancyWidget C++ class.
# Allow the use of std::string type
from libcpp.string cimport string
# The actual wrapper for the C++ based class AFancyWidget
cdef extern from "fancywidget.h":
cdef cppclass AFancyWidget:
AFancyWidget() except +
string GetName() except +
bint IsCorect(string value) except +
PYX Files
The "pyx" file provides the implementation of the extension using the declaration in the "pxd" file. This is the class that is used within your python code. There are two things to note in the "pyx" file:
- The use of “new” when instantiating AFancyWidget
- The C like syntax for declaring AFnacyWidget*
Cython allows developers to use a python like syntax to express the class interface of the extension class while also using a C like syntax to declare the C or C++ elements. Example 2 illustrates a simple "pyx" file.
Example 2
# distutils: language=c++
# distutils: sources=fancyWidget.cpp
# !!!...
# The above two comments need to be the first two comments in this file
# ...!!!
#
# Copyright (C) 2017 Art and Logic.
#
# The Cython code that generates a native python class that wraps the C++
# AFancyWidget class.
# This pulls in the AFancyWidget declaration from the pxd file
from widget cimport AFancyWidget
from libcpp.string cimport string
cdef class AFancyWidget:
# This is the python class that proxies the calls into the C++ class.
# This class can be used by native Python code as any other Python class
# would.
# Use composition for the C++ based class instance
cdef AFancyWidget* helper
def __cinit__(self):
# Create an instance of the AFancyWidget C++ class on the heap
self.helper = new AFancyWidget()
def __dealloc__(self):
# Destroy the heap allocated C++ class
del self.helper
def GetName():
return self.helper.GetName()
def IsCorrect(self, string value):
return self.helper.IsCorrect(value)
The Jelly
This is the sweet stuff that enhances the flavor of the PBJ. Without it you just have peanut butter and bread. The Cython build process is like the jelly. There are several options available to build the Cython extension. The mechanism used for building an extension will depend on the requirements of the execution environment and the extension. For this example I use cythonize with distutils as the mechanism for building the extension. What is so sweet about this process (pun intended), is that this is the same installation mechanism that other python modules use. To build simply issue the following command within the same directory as the setup.py file.
python setup.py build_ext --inplace
Compiler and linker options can be specified within the Extension declaration but our simple example doesn’t require such settings. Example 3 illustrates a simple setup.py file using cythonize.
Example 3
# Copyright (C) 2017 Art and Logic.
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
extensions = [
Extension("widget", ["fancyWidget.pyx"],
language="c++",
extra_compile_args=["-std=c++11"],
),
]
setup(
name = "widgets",
version = "0.1",
ext_modules = cythonize(extensions),
)
Putting it Together
The complete example can be found on github for those who want to try Cython. I wanted to show how easy it is to use Cython so I didn’t dive into the many features is has. Cython has many powerful features. I urge the reader to head to the documentation for more complete coverage of Cython features.
