Wrapping C Files
Wrapping C Files
3 (2008)
Computer Science
Institute of Information & Mathematical Sciences
Massey University at Albany, Auckland, New Zealand
Email: [email protected]
At some point of time many Python developers at least in computational
science will face the situation that they want to interface some natively
compiled library from Python. For binding native code to Python by now a
larger variety of tools and technologies are available. This paper focuses on
wrapping shared C libraries, using Python's default Ctypes. Particularly tools
to ease the process (by using code generation) and some best practises will be
stressed. The paper will try to tell a stepbystep story of the wrapping and
development process, that should be transferable to similar problems.
Keywords:
Introduction
One of the grand fundamentals in software engineering is to use the tools that are
best suited for a job, and not to prematurely decide on an implementation. That is
often easier said than done, in the light of some complimentary requirements (e. g.
rapid/easy implementation vs. needed speed of execution or vs. low level access to
hardware).
extending
or
embedding
Ctypes
As an example the creation of a wrapper for the Little CMS colour management
library [3] is outlined. The library oers excellent features, and ships with ocial
Python bindings (using
SWIG
(incompleteness, un-Pythonic API, complex to use, etc.). So out of need and frustration the initial steps towards alternative Python bindings were undertaken.
An alternative would be to x or improve the bindings using
SWIG,
or to use
one of a variety of binding tools. The eld has been limited to tools that are widely
in use today within the community, and that are promising to be future proof as
well as not overly complicated to use. These are the contestants with (very brief )
notes for use cases that suit their particular strengths:
Use
Use
Ctypes
Boost.Python
that also reects the object oriented nature of your native code, including
inheritance into Python, etc.
Use
cython
[7], if you want to easily speed up and migrate code from Python
SWIG
[4], if you want to wrap your code against several dynamic lan-
Use
guages.
Of course, wrapper code can be written manually, in this case directly using
Ctypes
Ctypes
should be familiar with this package when attempting to undertake serious library
wrapping. The
Ctypes tutorial
and
Ctypes reference
an excellent starting point for this. For extensive libraries and robustness towards
an evolving API, code generation proved to be a good approach over manual editing.
Boost.Python
Boost.Python
Py++
[8] (for
Ctypes
CtypesLib's h2xml.py
as well as for
) and
[2]
and xml2py.py.
Three main reasons have inuenced the decision to approach this project using
ctypes:
Ctypes
relieves one from installing a number of development tools, and the library
wrapper can be approached in a platform independent way.
The next section of this paper will rst introduce a simple C example.
This
example is later migrated to Python code through the various incarnations of the
Python wrapper throughout the paper. Sect. 3 introduces how to facilitate the C
library code from Python, in this case through code generation.
Sect. 4 explains
how to rene the generated code to meet the desired functionality of the wrapper.
The library is anything but Pythonic, so Sect. 5 explains an object oriented Faade
API for the library that features qualities we love.
This paper only outlines some interesting fundamentals of the wrapper building
process. Please refer to the source code for more precise details [9].
The Example
The sample code (listing in Fig. 1) aims to convert image data from device dependent
colour information to a standardised colour space.
a device specic characterisation of a Hewlett Packard ScanJet (in the ICC prole
HPSJTW.ICM). The output is in the standard conformant sRGB output colour space
as it is used for the majority of displays on computers. For this a built-in prole
from
LittleCMS
is used.
For the
input prole the characterisation is read from a le (line 8), and a built in output
prole is used (line 9). The transformation object is set up using the proles (lines
1113), specifying the colour encoding in the in- and output as well as some further
parameters not worth discussing here. In the for loop (lines 1521) the image data is
transformed line by line, operating on the number of pixels used per line (necessary
as array rows are often padded).
The goal is to provide a suitable and easy to use API to perform the same task
in Python.
Code Generation
Ctypes
is not particularly
dicult. The tutorial, project web site and documentation on the wiki introduce
this concept quite well.
wrapping can be tedious and error prone, as well as hard to keep consistent with
the library in case of changes. This is especially true when the library is maintained
by someone else. Therefore, it is advisable to generate the wrapper code.
Thomas Heller, the author of
CtypesLib
Ctypes
parts, the parser (for header les) and the code generator.
3.1
The C header les are parsed by the tool h2xml. In the background it uses GCCXML,
a GCC compiler that parses the code and generates an XML tree representation.
Therefore, usually the same compiler that builds the binary of the library can be
used to analyse the sources for the code generation. Alternative parsers often have
problems determining a 100 % proper interpretation of the code. This is particularly
true in the case of C code containing pre-processor macros, which can commit
massively complex things.
#include "lcms.h"
3
4
5
6
int correctColour(void) {
cmsHPROFILE inProfile, outProfile;
cmsHTRANSFORM myTransform;
int i;
8
9
11
12
13
15
16
17
18
19
20
21
23
24
25
cmsDeleteTransform(myTransform);
cmsCloseProfile(inProfile);
cmsCloseProfile(outProfile);
27
28
return 0;
}
Figure 1: Example in C using the
3.2
LittleCMS
library directly.
In the next stage the parser tree in XML format is taken to generate the binding
code in Python using
Ctypes.
ator can be congured in its actions by means of switches passed to it. Of particular
interest here are the
-k
and the
-r
to include in the output. In this case the #defines, functions, structure and union
denitions are of interest, yielding
matically. The
-r
-kdfs.
symbols to generate code for. The full argument list is shown in the listing in Fig. 2
(lines 1115). The generated code is written to a Python module, in this case _lcms.
It is made private by convention (leading underscore) to indicate that it is
be used or modied directly.
not
to
3.3
Both h2xml and xml2py are Python scrips. Therefore, the generation process can be
automated in a simple generator script. This makes all steps reproducible, documents the used settings, and makes the process robust towards evolutionary (smaller)
changes in the C API. A largely simplied version is in the listing of Fig. 2.
1
2
3
5
6
7
8
h2xml.main([h2xml.py, header_path,
-c,
-o,
%s.xml % header_basename])
10
11
12
13
14
15
never
be edited manually.
be necessary to achieve the desired functionality (see Sect. 4), automation becomes
essential to yield reproducible results. Due to some shortcomings (see Sect. 4) of
the generated code however, some editing was necessary. This modication has also
been integrated into the generator script to fully remove the need of manual editing.
Ctypes
platform independent way needs to be patched into the generated code. A function
in the code generator reads the whole generated module _lcms and writes it back to
the le system, and in the course replacing three lines from the beginning of the le
with the code snippet from the listing in Fig. 3.
1A
monkey patch is a way to extend or modify the runtime code of dynamic languages without
http://en.wikipedia.org/wiki/Monkey_patch
1
2
4
5
_libraries = {}
_libraries[/usr/lib/liblcms.so.1] = _setup._init()
1
2
import ctypes
from ctypes.util import find_library
4
5
6
7
8
9
10
class Structure(ctypes.Structure):
def __repr__(self):
"""Print fields of the object."""
res = []
for field in self._fields_:
res.append(%s=%s % (field[0], repr(getattr(self, field[0]))))
return %s(%s) % (self.__class__.__name__, , .join(res))
12
13
def _init():
return ctypes.cdll.LoadLibrary(find_library(lcms))
Figure 4: Extract from module _setup.py.
4.1
Further modications are less invasive. For this, the C API is rened into a module
everything
SWIG
LittleCMS
). The wrapped
C API can be used from Python (see Sect. 4.2). Although, it still requires closing,
freeing or deleting from the code after use, and c_lcms objects/structures do not
feature methods for operations. This shortcoming will be solved later.
4.2
c lcms
Example
The wrapped raw C API in Python behaves in exactly the same way, it is just
implemented in Python syntax (listing in Fig. 5).
3
4
5
def correctColour():
inProfile = cmsOpenProfileFromFile(HPSJTW.ICM, r)
outProfile = cmsCreate_sRGBProfile()
7
8
9
11
12
13
14
15
16
18
19
20
cmsDeleteTransform(myTransform)
cmsCloseProfile(inProfile)
cmsCloseProfile(outProfile)
Figure 5: Example using the basic API of the c_lcms module.
A Pythonic API
To create the usual pleasant batteries included feeling when working with code
in Python, another module littlecms was manually created, implementing the
API. This high level object oriented Faade takes care of the internal handling of
tedious and error prone operations. It also performs sanity checking and automatic
detection for certain crucial parameters passed to the C API. This has drastically
reduced problems with the low level nature of the underlying C library.
5.1
littlecms
Using
Example
Transform classes.
Redundant passing of information (e. g. the in- and output colour spaces) is
determined within the Transform constructor from information available in the
Profile objects.
Uses
NumPy
further custom types. On these data array types and shapes can be automatically matched up.
The number of pixels for each scan line placed in yourInBuffer can usually be
detected automatically.
PIL
Several sanity checks prevent clashes of erroneously passed buer sizes, shapes,
[11] library.
3
4
5
6
def correctColour():
inProfile = Profile(HPSJTW.ICM)
outProfile = Profile(colourSpace=PT_RGB)
myTransform = Transform(inProfile, outProfile)
8
9
10
Conclusion
Binding pure C libraries to Python is not very dicult, and the skills can be mastered
in a rather short time frame.
even towards certain changes in the evolving C API without the need of very time
consuming manual tracking of all changes.
vital to be able to automate the mechanical processes: Beyond the outlined code
generation in this paper, an important role comes to automated code integrity testing
(here: using
PyUnit
Unfortunately, as
CtypesLib
Epydoc
[13]).
CtypesLib.
CtypesLib
develop a x for the code generator as it failed to generate code for #defined oating
point constants. The patch has been reported to the author and is now in the source
code repository. Also patching into the generated source code for overriding some
features and manipulating the library loading code can be considered as being less
than elegant.
Library wrapping as described in this paper was performed on version 1.16 of the
LittleCMS
library. While writing this paper the author has moved to the now stable
version 1.17. Adapting the Python wrapper to this code base was a matter of about
15 minutes of work.
The main task was xing some unit tests due to rounding
LittleCMS
The
good day of modications, even though some substantial changes were made to the
API. But even for this case only very little amounts of new code had to be written.
Overall, it is foreseeable that this type of library wrapping in the Python world
will become more and more ubiquitous, as the tools for it mature. But already at the
present time one does not have to fear the process. The time spent initially setting
up the environment will be easily saved over all projects phases and iterations. It
will be interesting to see
well.
Ctypes
Ctypes
and
Py++
References
[1]
Yakovenko,
Py++
Project,
http://www.language-binding.net/
10
[9] G. K. Kloss, Source Code: Automatic C Library Wrapping Ctypes from the
Trenches,