Department of Physics

Need for Speed –

Python meets C/C++
Scientific Programming with Python
Christian Elsasser

Based partially on a talk by Stéfan van der Walt This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 License.

September 6, 2017 Page 1

Department of Physics

Python is nice, but by construction slow . . .

[xkcd]

September 6, 2017 Python meets C/C++ Page 2

Department of Physics

. . . why not therefore interfacing it with C/C++

(or something similar, e.g. if you don’t feel too young to use Fortran)

Easy to use/Flexibility

Cython → Performance improvement

Python
SWIG → Interfaces
boost::python
ctypes

...
C/C++
Fortran

Speed/Complexity
September 6, 2017 Python meets C/C++ Page 3
Department of Physics

. . . why not therefore interfacing it with C/C++

(or something similar, e.g. if you don’t feel too young to use Fortran)

Easy to use/Flexibility

Cython → Performance improvement

Python
SWIG → Interfaces
boost::python
ctypes

...
C/C++
Fortran

Speed/Complexity
September 6, 2017 Python meets C/C++ Page 3
Department of Physics

. . . why not therefore interfacing it with C/C++

(or something similar, e.g. if you don’t feel too young to use Fortran)

Easy to use/Flexibility

Cython → Performance improvement

Python
SWIG → Interfaces
boost::python
ctypes

...
C/C++
Fortran

Speed/Complexity
September 6, 2017 Python meets C/C++ Page 3
Department of Physics

A Few Technical Remarks

If you want to follow directly the code used in the lecture
I Download the code from the course homepage (Lecture 5)
I Start the virtual environment
$ . venv/bin/activate (from the home directory)
I Create a kernel for the notebook with the virtual environment
$ python3 -m ipykernel install --user --name=ve3
I Unzip the file
$ tar zxvf material_Python_C_lec.tar.gz
I Enter the created directory
$ cd material_Python_C_lec
I . . . and start the notebook
$ ipython3 notebook

September 6, 2017 Python meets C/C++ Page 4

Department of Physics

Table of Contents

I Introduction
I Why should I care?
I When should I consider something else?
I cython – a hybrid programming language/compiler
I Speed-up examples
I Standard Template Library
I Classes
I Exceptions
I SWIG (and other wrappers)

September 6, 2017 Python meets C/C++ Page 5

Department of Physics

C++ on one Slide www.cplusplus.com and www.learncpp.com

I C++ is an (if not the) object-oriented programming language (like Python)

I including inheritance (like Python does in a slightly different way)
I . . . operator overloading (like Python)
I It has a rich variety of libraries (like Python)
I It can raise exceptions (like Python)
I It requires declaration of variables (not like Python)
I It is (usually) a compiled language! (not like Python)
⇒ C++ and Python share a lot of similarities!
C is just the non-object-oriented version of C++ (minus some other missing features, e.g.
exceptions)

September 6, 2017 Python meets C/C++ Page 6

Department of Physics

A Few Words of Warning

Bad code stays bad code! – Better clean Do not expect miracles! – You have to
it up than trying to overpaint it! master two languages!

September 6, 2017 Python meets C/C++ Page 7

Department of Physics

C keeps Python running . . .

I CPython is the standard implementation of the Python interpreter written in C.
I The Python C API (application programming interface) allows to build C libraries that can be
imported into Python (https://docs.python.org/3/c-api/) . . .
I . . . and looks like this:

Pure Python
»»» a = [1,2,3,4,5,6,7,8]
»»» sum(a)
36

September 6, 2017 Python meets C/C++ Page 8

Department of Physics

. . . but takes a lot of the fun out of Python

Python C can understand
sum_list(PyObject *list) {
int i, n;
long total = 0;
PyObject *item;
n = PyList_Size(list);
if (n < 0)
return -1; /* Not a list */
for (i = 0; i < n; i++) {
item = PyList_GetItem(list, i); /* Can’t fail */
if (!PyInt_Check(item)) continue; /* Skip non-integers */
total += PyInt_AsLong(item);
}
return total;
}

September 6, 2017 Python meets C/C++ Page 9

Department of Physics

C/C++ in Python: Not a New Thing

NumPy’s C API
ndarray typedef struct PyArrayObject {
PyObject_HEAD
char *data;
int nd;
npy_intp *dimensions;
npy_intp *strides;
PyObject *base;
PyArray_Descr *descr;
int flags;
PyObject *weakreflist;
} PyArrayObject ;

⇒ Several Python “standard” libraries are using C/C++ to speed things up

September 6, 2017 Python meets C/C++ Page 10

Department of Physics

Cython – An easy way to get C-enhanced compiled Python code

(http://cython.org)
I Hybrid programming language combining Python and an interface for using C/C++ routines.
I . . . or a static compiler for Python allowing to write C/C++ extensions for Python and heavily
optimising this code.
I It is a successor of the Pyrex language.
⇒ Every valid Python statement is also valid when using cython.
⇒ Code needs to be compiled → Time!
I Translates you “C-enhanced” Python code into C/C++ code using the C API
Cython (v0.25.2) understands Python 3, and also most of the features of C++11

September 6, 2017 Python meets C/C++ Page 11

Department of Physics

Requirements: Cython package and a C compiler

I cython
The latest version can be downloaded from http://cython.org.
I C/C++ compiler, e.g. gcc/g++/clang (or for Windows: mingw)
Mille viae ducunt hominem per saecula ad compilorem!

Linux: usually already installed

(Ubuntu/Debian: sudo apt-get install build-essential)

MacOS X: XCode command line tools

Windows: Download of MinGW from http:// mingw.org and install it

September 6, 2017 Python meets C/C++ Page 12

Department of Physics

Benchmark One: Fibonacci series

Fibonacci (Pure Python)

def fib(n):
a,b = 1,1
for i in range(n):
a,b = a+b,a
return a

September 6, 2017 Python meets C/C++ Page 13

Department of Physics

Benchmark One: Fibonacci series

Fibonacci (Cython)
def fib(int n):
cdef int i,a,b
a,b = 1,1
for i in range(n):
a,b = a+b,a
return a

I Type declaration (cdef) ⇒ Python/Cython knows what to expect

September 6, 2017 Python meets C/C++ Page 13

Department of Physics

Benchmark One: Fibonacci series

Fibonacci (Cython)
def fib(int n):
cdef int i,a,b
a,b = 1,1
for i in range(n):
a,b = a+b,a
return a

I Type declaration (cdef) ⇒ Python/Cython knows what to expect

I A few (simple) modifications can easily change the CPU time by a factor of O(100)

September 6, 2017 Python meets C/C++ Page 13

Department of Physics

Compiling Cython Code (The hard way)

cython ‘gcc‘ ‘gcc‘ 1. Compile Cython code to C/C++ code

cython/cython3 <name>.pyx
.c 2. Create object files
.pyx .o .so
.cpp gcc -O2 -fPIC
-I<path_to_python_include> -c
<name>.c -o <name>.o
3. Compile shared object (i.e. library)
gcc [options]
-L<path_to_python_library>
lib <name>.o -o <name>.so
I If using C++ code, cython needs the
option -+ and gcc → g++
Shared object (<name>.so) can be imported into I options are for MacOS X -bundle
Python with import name -undefined dynamic_lookup and
for Debian -shared
September 6, 2017 Python meets C/C++ Page 14
Department of Physics

Compiling Cython Code (The hard way)

cython ‘gcc‘ ‘gcc‘ 1. Compile Cython code to C/C++ code

cython/cython3 <name>.pyx
.c 2. Create object files
.pyx .o .so
.cpp gcc -O2 -fPIC
-I<path_to_python_include> -c
<name>.c -o <name>.o
3. Compile shared object (i.e. library)
gcc [options]
-L<path_to_python_library>
lib <name>.o -o <name>.so
I If using C++ code, cython needs the
option -+ and gcc → g++
Shared object (<name>.so) can be imported into I options are for MacOS X -bundle
Python with import name -undefined dynamic_lookup and
for Debian -shared
September 6, 2017 Python meets C/C++ Page 14
Department of Physics

Compiling Cython Code (The hard way)

cython ‘gcc‘ ‘gcc‘ 1. Compile Cython code to C/C++ code

cython/cython3 <name>.pyx
.c 2. Create object files
.pyx .o .so
.cpp gcc -O2 -fPIC
-I<path_to_python_include> -c
<name>.c -o <name>.o
3. Compile shared object (i.e. library)
gcc [options]
-L<path_to_python_library>
lib <name>.o -o <name>.so
I If using C++ code, cython needs the
option -+ and gcc → g++
Shared object (<name>.so) can be imported into I options are for MacOS X -bundle
Python with import name -undefined dynamic_lookup and
for Debian -shared
September 6, 2017 Python meets C/C++ Page 14
Department of Physics

Compiling Cython Code (The hard way)

cython ‘gcc‘ ‘gcc‘ 1. Compile Cython code to C/C++ code

cython/cython3 <name>.pyx
.c 2. Create object files
.pyx .o .so
.cpp gcc -O2 -fPIC
-I<path_to_python_include> -c
<name>.c -o <name>.o
3. Compile shared object (i.e. library)
gcc [options]
-L<path_to_python_library>
lib <name>.o -o <name>.so
I If using C++ code, cython needs the
option -+ and gcc → g++
Shared object (<name>.so) can be imported into I options are for MacOS X -bundle
Python with import name -undefined dynamic_lookup and
for Debian -shared
September 6, 2017 Python meets C/C++ Page 14
Department of Physics

Compiling Cython Code (The hard way)

cython ‘gcc‘ ‘gcc‘ 1. Compile Cython code to C/C++ code

cython/cython3 <name>.pyx
.c 2. Create object files
.pyx .o .so
.cpp gcc -O2 -fPIC
-I<path_to_python_include> -c
<name>.c -o <name>.o
3. Compile shared object (i.e. library)
gcc [options]
-L<path_to_python_library>
lib <name>.o -o <name>.so
I If using C++ code, cython needs the
option -+ and gcc → g++
Shared object (<name>.so) can be imported into I options are for MacOS X -bundle
Python with import name -undefined dynamic_lookup and
for Debian -shared
September 6, 2017 Python meets C/C++ Page 14
Department of Physics

Compiling Cython Code (The easy way)

Support via the distutils (distribution utilities) package in building and installing Python modules
⇒ applicable for cython
setup.py
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize([<list of .pyx files>],
language="c++" # optional
)
)

Command python setup.py build_ext --inplace creates for each .pyx file a .c/.cpp file,
compiles it to an executable (in the build directory of the corresponding OS/architecture/Python
version) and compiles a .so file (or a .pxd if you are using Windows)
Further options for cythonize via help explorable

September 6, 2017 Python meets C/C++ Page 15

Department of Physics

How Performant is My Code?

cython -a/--annotate <name>.pxy → additional HTML file

I bad performance → yellow marking

I allows to investigate code and to learn about performance tuning

I Not every yellow part can be improved!

September 6, 2017 Python meets C/C++ Page 16
Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

September 6, 2017 Python meets C/C++ Page 17

Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

Integrate
from math import sin,exp
def f(double x):
return sin(x)*exp(-x)
def integrate(double a,double b,int N):
cdef double dx,s
cdef int i
dx = (b-a)/N
s = 0.0
for i in range(N):
s += f(a+(i+0.5)*dx)
return s*dx
September 6, 2017 Python meets C/C++ Page 17
Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

Python layer (expensive) C layer (cheap)

integrate(a,b,N)
‘_pyx_integrate’(a,b,N)
for (i=0; i<N; i++)
f(x)
‘_pyx_f’(x)
sum updated

September 6, 2017 Python meets C/C++ Page 17

Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

Integrate
from math import sin,exp
cdef double f(double x):
return sin(x)*exp(-x)
def integrate(double a,double b,int N):
cdef double dx,s
cdef int i
dx = (b-a)/N
s = 0.0
for i in range(N):
s += f(a+(i+0.5)*dx)
return s*dx
September 6, 2017 Python meets C/C++ Page 17
Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

Integrate
from math import sin,exp
cpdef double f(double x):
return sin(x)*exp(-x)
def integrate(double a,double b,int N):
cdef double dx,s
cdef int i
dx = (b-a)/N
s = 0.0
for i in range(N):
s += f(a+(i+0.5)*dx)
return s*dx
September 6, 2017 Python meets C/C++ Page 17
Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

Integrate
from libc.math cimport sin,exp
cpdef double f(double x):
return sin(x)*exp(-x)
def integrate(double a,double b,int N):
cdef double dx,s
cdef int i
dx = (b-a)/N
s = 0.0
for i in range(N):
s += f(a+(i+0.5)*dx)
return s*dx
September 6, 2017 Python meets C/C++ Page 17
Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607

Integrate
from libc.math cimport sin,exp
cpdef double f(double x):
return sin(x)*exp(-x)
def integrate(double a,double b,int N):
cdef double dx,s
cdef Py_ssize_t i
dx = (b-a)/N
s = 0.0
for i in range(N):
s += f(a+(i+0.5)*dx)
return s*dx
September 6, 2017 Python meets C/C++ Page 17
Department of Physics

Benchmark Two: Numerical Integration

Integral of f (x) = sin x · e−x between 0 and π
⇒ Exact result: (e−π + 1)/2 = 0.521607
I Return values of function can be specified via the key word cdef
I cpdef ⇒ function also transparent to Python itself (no performance penalty)

I C/C++ library can be imported via from libc/libcpp.<module> cimport <name> (see
later)
I Using C++ functions can lead to a huge speed-up
I Try to do as much as you can in the C-layer

I Already huge speed-up when levaraging numpy and its vectorisation

September 6, 2017 Python meets C/C++ Page 17

Department of Physics

You are here!

September 6, 2017 Python meets C/C++ Page 18

Department of Physics

STL Containers
An often used feature of C++ are the Standard Template Library containters (e.g. std::vector,
std::map, etc.)
Object holders with specific memory access structure, e.g.
I std::vector allows to access any element
I std::list only allows to access elements via iteration
I std::map represents an associative container with a key and a mapped values

September 6, 2017 Python meets C/C++ Page 19

Department of Physics

STL Containers
An often used feature of C++ are the Standard Template Library containters (e.g. std::vector,
std::map, etc.)
. . . and Cython knows how to treat them!

Python −→ C++ −→ Python

iterable → std::vector → list

iterable → std::list → list
iterable → std::set → set
iterable (len 2) → std::pair → tuple (len 2)
dict → std::map → dict
bytes → std::string → bytes

September 6, 2017 Python meets C/C++ Page 19

Department of Physics

STL Containers
An often used feature of C++ are the Standard Template Library containters (e.g. std::vector,
std::map, etc.)
A few remarks!
I iterators (e.g. it) can be used ⇒ dereferencing with dereference(it) and
incrementing/decrementing with preincrement (i.e. ++it), postincrement (i.e. it++),
predecrement (i.e. --it) and postdecrement (i.e. it--) from cython.operator
I Be careful with performance! ⇒ performance lost due to shuffling of data
I More indepth information can be found directly in the corresponding sections of the cython
code https://github.com/cython/cython/tree/master/Cython/Includes/libcpp
I C++11 containters (like std::unordered_map) are partially implemented

September 6, 2017 Python meets C/C++ Page 19

Department of Physics

Exceptions/Errors
In terms of exception and error handling three different cases need to be considered:
I Raising of a Python error in cython code ⇒ return values make it impossible to raise
properly Python errors (Warning message, but continuing)
I Handling of error codes from pure C functions
I Raising of a C++ exception in C++ code used in cython ⇒ C++ exception terminates – if
not caught – program

September 6, 2017 Python meets C/C++ Page 20

Department of Physics

Errors in Python

Python Error
cpdef int raiseError():
raise RuntimeError("A problem")
return 1

⇒ Just prints a warning (and worse gives an ambigious return value)

September 6, 2017 Python meets C/C++ Page 21

Department of Physics

Errors in Python

Python Error
cpdef int raiseError():
raise RuntimeError("A problem")
return 1
⇒ Just prints a warning (and worse gives an ambigious return value)

Python Error
cpdef int raiseError() except *:
raise RuntimeError("A problem")
return 1

⇒ Propagates the RuntimeError

September 6, 2017 Python meets C/C++ Page 21

Department of Physics

Errors in C
C does not know exceptions like Python or C++. If errors should be caught, it is usually done via
dedicated return values of functions which cannot appear in a regular function call.
Use the except statement to tell cython about this value

C Error
cpdef int raiseException() except -1:
return -1

⇒ allows to indicate error codes from C ⇒ raises SystemError

September 6, 2017 Python meets C/C++ Page 22

Department of Physics

Exceptions in C++

[xkcd]

In cython this is also true for C++ exceptions!

Cython is not able to deal with C++ exceptions in a try’n’except clause!
⇒ But caption in cython and translation to Python exceptions/errors is possible!

September 6, 2017 Python meets C/C++ Page 23

Department of Physics

Exceptions in C++
. . . and how to tackle them!
I cdef <C++ function>() except + C++ −→ Python
⇒ translates a C++ exception into a Python bad_alloc → MemoryError
error according to the right-hand scheme bad_cast → TypeError
I cdef <C++ function>() except domain_error → ValueError
invalid_argument → ValueError
+<Python Error> e.g. MemoryError ⇒
ios_base::failure → IOError
translates every thrown C++ exception into out_of_range → IndexError
a MemoryError overflow_error → OverflowError
range_error → ArithmeticError
I cdef <C++ function>() except
underflow_error → ArithmeticError
+<function raising Python error> ⇒ (all others) → RuntimeError
runs the indicated function if the C++
function throws any exception. If <function
raising Python error> does not raise an
error, a RuntimeError will be raised.
September 6, 2017 Python meets C/C++ Page 23
Department of Physics

Classes
Classes are a common feature of Python and C++
There are two aspects when dealing with cython:
I Defining classes containing C++ code in cython
I C++ classes integrated into Python

September 6, 2017 Python meets C/C++ Page 24

Department of Physics

Defining Classes in Cython

Let’s go back to the integration examples

Integrate with classes

cdef class Integrand:
cpdef double evaluate(self,double x) except *:
raise NotImplementedError()
cdef class SinExpFunction(Integrand):
cpdef double evaluate(self,double x):
return sin(x)*exp(-x)
def integrate(Integrand f,double a,double b,int N):
...
s += f.evaluate(a+(i+0.5)*dx)
...

Cython does not know @abstractmethod from the module abc!

September 6, 2017 Python meets C/C++ Page 25

Department of Physics

Defining Classes in Cython

Let’s go back to the integration examples

Integrate with classes

class Poly(Integrand):
def evaluate(self,double x):
return x*x-3*x

integrate(Poly(),0.0,2.0,1000)

⇒ Speed lost with respect to definition in cython, but still faster than a pure Python
implementation

September 6, 2017 Python meets C/C++ Page 25

Department of Physics

Integration of C++ Classes in Cython – Possible but cumbersome

Starting point: .cpp/.h file for class Rectangle defined in a namespace shapes
1. Expose it to Cython by delaring the class structure and method signatures
2. Integrating it into Cython either via direct usage or by defining a wrapper class

rect.pyx (File to expose C++ class to cython)

# distutils: language = c++
# distutils: sources = Rectangle.cpp
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Rectangle:
Rectangle(int, int, int, int) except +
int x0, y0, x1, y1
int getLength()
int getHeight()
int getArea()
void move(int, int)

September 6, 2017 Python meets C/C++ Page 26

Department of Physics

Automatic Wrappers
. . . since not everybody likes to write lines of error-prone code
I SWIG
I boost::python
I ctypes
I ...

Goal: creating compilable C/C++ code

based on the Python C API

September 6, 2017 Python meets C/C++ Page 27

Department of Physics

SWIG
SWIG: Simplified Wrapper and Interface Generator
I Generic Wrapper for C/C++ to script-like languages
I R
I Perl
I Ruby
I Tcl
I PHP5
I Java
I . . . and Python
I Pretty old – created in 1995 by Dave Beazley
I Current version is 3.0.12

September 6, 2017 Python meets C/C++ Page 28

Department of Physics

SWIG – in a Nutshell

swig ‘gcc‘ ‘gcc‘ 1. Create python wrapper and

necessary C files
..h .c
swig -c++ -python <name>.i
.cxx .o .so
.c .cpp 2. Create object files based on output
from the wrapper plus native C/C++
code
3. Compile shared object (i.e. library)
.i .py
Normally step 2 and 3 can be
lib combined with via Distutils setup.py
python setup.py build_ext
--inplace
Moduel (<name>.py) can be imported into Python
with import name ⇒ Shared object needs different
name
September 6, 2017 Python meets C/C++ Page 29
Department of Physics

SWIG – in a Nutshell

swig ‘gcc‘ ‘gcc‘ 1. Create python wrapper and

necessary C files
..h .c
swig -c++ -python <name>.i
.cxx .o .so
.c .cpp 2. Create object files based on output
from the wrapper plus native C/C++
code
3. Compile shared object (i.e. library)
.i .py
Normally step 2 and 3 can be
lib combined with via Distutils setup.py
python setup.py build_ext
--inplace
Moduel (<name>.py) can be imported into Python
with import name ⇒ Shared object needs different
name
September 6, 2017 Python meets C/C++ Page 29
Department of Physics

SWIG – in a Nutshell

swig ‘gcc‘ ‘gcc‘ 1. Create python wrapper and

necessary C files
..h .c
swig -c++ -python <name>.i
.cxx .o .so
.c .cpp 2. Create object files based on output
from the wrapper plus native C/C++
code
3. Compile shared object (i.e. library)
.i .py
Normally step 2 and 3 can be
lib combined with via Distutils setup.py
python setup.py build_ext
--inplace
Moduel (<name>.py) can be imported into Python
with import name ⇒ Shared object needs different
name
September 6, 2017 Python meets C/C++ Page 29
Department of Physics

SWIG – in a Nutshell

swig ‘gcc‘ ‘gcc‘ 1. Create python wrapper and

necessary C files
..h .c
swig -c++ -python <name>.i
.cxx .o .so
.c .cpp 2. Create object files based on output
from the wrapper plus native C/C++
code
3. Compile shared object (i.e. library)
.i .py
Normally step 2 and 3 can be
lib combined with via Distutils setup.py
python setup.py build_ext
--inplace
Moduel (<name>.py) can be imported into Python
with import name ⇒ Shared object needs different
name
September 6, 2017 Python meets C/C++ Page 29
Department of Physics

SWIG – in a Nutshell

swig ‘gcc‘ ‘gcc‘ 1. Create python wrapper and

necessary C files
..h .c
swig -c++ -python <name>.i
.cxx .o .so
.c .cpp 2. Create object files based on output
from the wrapper plus native C/C++
code
3. Compile shared object (i.e. library)
.i .py
Normally step 2 and 3 can be
lib combined with via Distutils setup.py
python setup.py build_ext
--inplace
Moduel (<name>.py) can be imported into Python
with import name ⇒ Shared object needs different
name
September 6, 2017 Python meets C/C++ Page 29
Department of Physics

SWIG – The interface file

Main configuration with interface (.i) files
I tells which (header) file(s) contains the Interface file
C/C++ code to wrap %module geom // name of the module
I defines some special data types ...
(e.g. std::vector<...>) // things swig should know about
%include "Shape.h"
I handles some additional configuration %include "Rectangle.h"
(e.g. exception/error translation) ...
// things that should be put into the
header of the wrapper file (.c/.cxx)
%{
#include "Shape.h"
#include "Rectangle.h"
%}

September 6, 2017 Python meets C/C++ Page 30

Department of Physics

SWIG – The Distutils file

Distutils (setup.py)
from distutils.core import setup, Extension
extension_mod = Extension("_<name>",
["<name_wrap>.cxx",
"<source1>.cpp",
"<source2>.cpp","..."],
language="c++")
setup(name = "_<name>", ext_modules=[extension_mod])

I To be build extension needs a different name than the module set up by switch ⇒ Avoid
name conflicts
I Language option only for C++
I python setup.py build_ext --inplace
September 6, 2017 Python meets C/C++ Page 31
Department of Physics

A Few Remarks about SWIG

I SWIG ≈ performance loss with respect to cython
I If SWIG works: ,
I If it does not: /
I . . . and therefore you can lose a lot of time with special problems
I It is not always optimal to expose the whole class to Python

September 6, 2017 Python meets C/C++ Page 32

Department of Physics

Conclusion
I Interfacing Python with C/C++ is – or
better – can be a way to create powerful
code
I cython and SWIG are two nice tools to
do so
I . . . but always make the interfacing
maintainable/useful/etc. i.e. not a British
train door
I And it’s all about finding the sweet spot!

September 6, 2017 Python meets C/C++ Page 33

Department of Physics

The Sweet Spot!

Time spent

Pure code optimal Compiled code optimal

Python + C/C++

Python
Compilation

Code executed per compilation

September 6, 2017 Python meets C/C++ Page 34

Department of Physics

The End!

[xkcd]
September 6, 2017 Python meets C/C++ Page 35
Department of Physics

References

1. Stéfan van der Walt, Speeding up scientific Python code using Cython, Advanced Scientific
Programming in Python, 2013 (Zurich) & 2014 (Split)
2. Stefan Behnel et al., Cython tutorial, Proceedings of the 8th Python in Science Conference (SciPy 2009)
⇒ based on older cython version, but the main reference of cython
3. Dave Beazley, Swig Master Class, PyCon’2008
4. http://docs.cython.org/src/tutorial/
5. http://www.swig.org

September 6, 2017 Python meets C/C++ Page 36

Department of Physics

Backup
Department of Physics

Fortran meets Python

The f2py compiler (http://docs.scipy.org/doc/numpy-dev/f2py/) offers – in a similar way as
cython – the possibility to generate extension modules for Python from Fortran code.
f2py -c -m <module name> <fortran file>.f/.f90 -I<path to python header file>
builds from the code in <fortran file>.f/.f90 a importable module (i.e. shared object)
<module name>.so
Fortran modules and subroutines are exposed to Python on time of the import of the built module.
The compilation can also be split into a first step generating a signature file, which is in a second
step compiled into the extension module

September 6, 2017 Python meets C/C++ Page 38

Department of Physics

Exceptions in C++
Examples
Two C++ functions void raiseException() and void raiseBadAlloc() defined in
except_cy.h

Exception Example 1
cdef extern from ’except_cy.h’
cdef void raiseException() except +
def tryIt():
try:
raiseException()
except RuntimeError as e:
print(e)

⇒ OK as raiseException() throws a std::exception → RuntimeError

September 6, 2017 Python meets C/C++ Page 39

Department of Physics

Exceptions in C++
Examples
Two C++ functions void raiseException() and void raiseBadAlloc() defined in
except_cy.h

Exception Example 2
cdef extern from ’except_cy.h’
cdef void raiseException() except +MemoryError
def tryIt():
try:
raiseException()
except RuntimeError as e:
print(e)

⇒ Not OK as raiseException() throws a std::exception which is explicitly transformed into a

MemoryError

September 6, 2017 Python meets C/C++ Page 39

Department of Physics

Exceptions in C++
Examples
Two C++ functions void raiseException() and void raiseBadAlloc() defined in
except_cy.h

Exception Example 3
cdef extern from ’except_cy.h’
cdef void raiseBadAlloc() except +
def tryIt():
try:
raiseBadAlloc()
except RuntimeError as e:
print(e)

⇒ Not OK as raiseBadAlloc() throws a std::bad_alloc which is transformed into a

MemoryError

September 6, 2017 Python meets C/C++ Page 39

Department of Physics

Exceptions in C++
Examples
Two C++ functions void raiseException() and void raiseBadAlloc() defined in
except_cy.h

Exception Example 4
cdef extern from ’except_cy.h’
cdef void raiseBadAlloc() except +
def tryIt():
try:
raiseBadAlloc()
except MemoryError as e:
print(e)

⇒ OK as raiseBadAlloc() throws a std::bad_alloc which is transformed into a MemoryError

September 6, 2017 Python meets C/C++ Page 39

Department of Physics

Exceptions in C++
Examples
Two C++ functions void raiseException() and void raiseBadAlloc() defined in
except_cy.h

Exception Example 5
cdef void raise_py_error() except *:
raise MemoryError("Problem")
cdef extern from ’except_cy.h’:
cdef void raiseBadAlloc() except +raise_py_error
def tryIt():
try:
raiseBadAlloc()
except MemoryError as e:
print(e)

⇒ OK as raise_py_error() throws an error

September 6, 2017 Python meets C/C++ Page 39
Department of Physics

Exceptions in C++
Examples
Two C++ functions void raiseException() and void raiseBadAlloc() defined in
except_cy.h

Exception Example 6
cdef void raise_py_error() except *:
pass
cdef extern from ’except_cy.h’:
cdef void raiseBadAlloc() except +raise_py_error
def tryIt():
try:
raiseBadAlloc()
except MemoryError as e:
print(e)

⇒ Not OK as no error is thrown by raise_py_error()

September 6, 2017 Python meets C/C++ Page 39
Department of Physics

Integration of C++ Classes

Assuming a C++ class Rectangle

Rectangle.h
namespace shapes {
class Rectangle {
public:
int x0, y0, x1, y1;
Rectangle(int x0, int y0, int x1, int y1);
∼Rectangle(); // destructor
int getLength();
int getHeight();
int getArea();
void move(int dx, int dy);
};
}

September 6, 2017 Python meets C/C++ Page 40

Department of Physics

Integration of C++ Classes

Assuming a C++ class Rectangle

Rectangle.cpp
#include "Rectangle.h"
#include <iostream>
using namespace shapes;
Rectangle::Rectangle(int X0, int Y0, int X1, int Y1) {
x0 = X0;
y0 = Y0;
x1 = X1;
y1 = Y1;
std::cout « "Here I am" « std::endl;}
Rectangle::∼Rectangle() {
std::cout « "Byebye" « std::endl;}
...

September 6, 2017 Python meets C/C++ Page 40

Department of Physics

Integration of C++ Classes

Now exposing it to cython
rect.pyx
# distutils: language = c++
# distutils: sources = Rectangle.cpp
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Rectangle:
Rectangle(int, int, int, int) except +
int x0, y0, x1, y1
int getLength()
int getHeight()
int getArea()
void move(int, int)

September 6, 2017 Python meets C/C++ Page 40

Department of Physics

Integration of C++ Classes

. . . and using it!
Either in further cython code!
rect.pyx
def tryIt():
cdef Rectangle* r
try:
r = new Rectangle(1,2,3,4)
print("My length is: %f"%r.getLength())
print("My first x-coordinate is: %f"%r.x0)
finally:
del r

September 6, 2017 Python meets C/C++ Page 40

Department of Physics

Integration of C++ Classes

. . . and using it!
Or for creating a Python (wrapper) class!
rect.pyx
cdef class PyRectangle:
cdef Rectangle *thisptr
def __cinit__(self, int x0, int y0, int x1, int y1):
self.thisptr = new Rectangle(x0, y0, x1, y1)
def __dealloc__(self):
del self.thisptr
def getLength(self):
return self.thisptr.getLength()
def getHeight(self):
return self.thisptr.getHeight()
...

September 6, 2017 Python meets C/C++ Page 40

Department of Physics

Special features: STL Stuff with SWIG

I Dedicated interface files need to be integrated when running SWIG
I . . . and templates for each containers + each content need to be defined

Interface file
...
%include "std_vector.i"
%include "std_string.i"
...
%template(dVector) std::vector<double>;
%template(rectVector) std::vector<Rectangle*>;
...

September 6, 2017 Python meets C/C++ Page 41

Department of Physics

Special features: Exceptions with SWIG

Interface file
...
%include "exception.i"
...
%exceptionclass ShapeError;
%exception *::whine {
try {
$action
} catch(ShapeError & e) {
ShapeError *ecopy = new ShapeError(e);
PyObject *err = SWIG_NewPointerObj(ecopy, SWIGTYPE_p_ShapeError, 1);
PyErr_SetObject(SWIG_Python_ExceptionType(SWIGTYPE_p_ShapeError), err);
SWIG_fail;
}
}
September 6, 2017 Python meets C/C++ Page 42
Department of Physics

Special features: Overloading

Cython deals the usual way with overloaded methods in C++:
rect.pyx works
cdef extern from "Rectangle.h" namespace "shapes":
...
void move(int, int)
void move(int)
but it cannot happen in a Python wrapper class:
rect.pyx does not work
cdef class PyRectangle:
...
def move(self,dx,dy):
return self.thisptr.move(dx,dy)
def move(self,d):
return self.thisptr.move(d)

September 6, 2017 Python meets C/C++ Page 43

Department of Physics

Special features: Inheritance

As in Python C++ classes can inherit from parent classes including overriding of methods

C++ classes
class Shape {
public:
...
void virtual printInfo(); // Prints "Shape"
};
class Rectangle : public Shape {
public:
...
void printInfo(); // Prints "Rectangle"
};

September 6, 2017 Python meets C/C++ Page 44

Department of Physics

Special features: Inheritance

Cython can also deal with this feature, but there are two points to keep in mind:
1. If parent class is also exposed to cython, no redefinition of overridden methods is required
(and also allow → mis-interpreted as overloading)

C++ classes
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Shape:
Shape() except +
void printInfo()
cdef cppclass Rectangle(Shape):
Rectangle(int, int, int, int) except +
...
void printInfo() # causes problems
...

September 6, 2017 Python meets C/C++ Page 44

Department of Physics

Special features: Inheritance

2. The inheritance can only be transported into wrapper classes if child classes have the same
set of methods as the mother class
C++ classes
cdef class PyObject:
cdef Object* thisptr
def __cinit__(self):
self.thisptr = new Object()
def __dealloc__(self):
del self.thisptr
def printInfo(self):
self.thisptr.printInfo()
cdef class PyRectangle(PyObject):
def __cinit__(self,int x0,int y0,int x1,int y1):
self.thisptr = new Rectangle(x0,y0,x1,y1)

September 6, 2017 Python meets C/C++ Page 44

Department of Physics

Special features: Operator Overloading

C++ as well as Python offers the potential to define operators for objects.
Example with Rectangles:

A
B*A B

September 6, 2017 Python meets C/C++ Page 45

Department of Physics

Special features: Operator Overloading

C++ code
Rectangle operator*(Rectangle& rhs){
return Rectangle(x0,y0,rhs.x1,rhs.y1);
};

rect.pyx
# to expose it to cython
Rectangle operator*(Rectangle)

# in the wrapper class

def __mul__(PyRectangle lhs,PyRectangle rhs):
res = PyRectangle(0,0,0,0)
res.thisptr[0] = lhs.thisptr[0]*rhs.thisptr[0] # ptr deref
return res
September 6, 2017 Python meets C/C++ Page 45
Department of Physics

Arrays
Arrays in cython are usually treated via typed memoryviews (e.g. double[:,:] means a
two-dimensional array of doubles, i.e. compatible with e.g. np.ones((3,4)))
Further you can specify which is the fastest changing index by :1, e.g.
I double[::1,:,:] is a F-contiguous three-dimensional array
I double[:,:,::1] is a C-contiguous three-dimensional array
I double[:,::1,:] is neither F- nor C-contiguous
For example a variable double[:,::1] a has as NumPy arrays variables like shape and size
and the elements can be accessed by a[i,j]
But be aware: NumPy is already heavily optimised, so do not to reinvent the wheel!

September 6, 2017 Python meets C/C++ Page 46