Mixed Language Programming

Types of Languages Mixings

Embedding Python Using Python code from code in a different language. Extending Python Using code in a different language from Python.

We

are going to focus only on extending Python in this presentation.

Why Mixing Python with Another Language?

Extending\Embedding Python:
To

spare the development cycle of a code that was already written once (code reuse).

Extending Python:
To improve Python code performance. True multithreading support. Type-safe language -> Less bugs (?).

Other reasons:
To

save testing time by testing low-level code using Python high-level tests. Other languages are more mature:

Less bugs in the language. Mission-specific tools that ease development.

Python-C Integration Using ctypes

ctypes

Functionality:
Gives

access to the OS specific DLLs via Python. Allows calling C language functions in userdefined DLLs\shared libraries from Python.

Pros:
No

need to recompile the DLL (wrap libraries in pure Python). Availability: External module up to V2.4. From V2.5 part of the Python standard library.

Loading a DLL

cdecl calling convention (most of the times):

dll

= ctypes.cdll.LoadLibrary(<dll_name_str>)

stdcall calling convention (on Windows):

dll

= ctypes.windll.LoadLibrary(<dll_name_str>)

On Windows, some DLLs are being loaded automatically:

Kernel32.dll

is at ctypes.windll.kernel32 Msvcrt.dll is at ctypes.cdll.msvcrt

Fundamental Data Types

ctypes type c_char
c_int c_ulong c_float c_double c_char_p

C type char
int unsigned long float double char * (NULL terminated) void *

Python type 1 character string int/long int/long float float string or None int\long or None

c_void_p

Fundamental Data Types (II)

An int data type:

i

= ctypes.c_int(3) # or i=ctypes.c_int(), to get an init val of 0. i.value = -99

A string data type:

s

= "Hello, World" c_s = ctypes.c_char_p(s) # or c_s = ctypes.c_char_p(), to an init val of NULL. c_s.value = "Hi, there"

Strings and Buffers

Assigning a new value to instances of the pointer types c_char_p and c_void_p changes the memory location they point to, not the contents of the memory block. If you need mutable memory blocks, use create_string_buffer():

buff

= ctypes.create_string_buffer("Hello", 10) # create a 10 byte buffer buff.value = "Hi"

Use the raw attribute to get the entire buffer contents. ctypes.sizeof() == sizeof() in C

Accessing Global Variables

Suppose that in your DLL, Test.dll, you have the following exported global variable declaration:
__declspec(dllexport)

double some_var;

To access it from ctypes, you must call the in_dll() method of the variables proper ctype, with the variables containing DLL and name:
test_dll

= ctypes.CDLL(Test.dll) test_dll_var = ctypes.c_double.in_dll(test_dll, "some_var") test_dll_var.value = 3.14

Calling Functions

You can call every exported function (i.e., __declspec(dllexport)-ed function on Windows) in your DLL. All Python types except integers, strings, and unicode strings have to be wrapped in their corresponding ctypes type:
libc

= ctypes.cdll.msvcrt printf = libc.printf printf("An int %d, a double %f\n", 1234, c_double(3.14))

None is used as the NULL pointer:

libc.time(None)

# System time in seconds since

the UNIX epoch

Windows Specific Issues

You must specify (for example) GetModuleHandleA or GetModuleHandleW explicitly, and then call it with normal or unicode strings respectively. ctypes tries to protect you from calling functions with the wrong number of arguments or the wrong calling convention (raises ValueError).
It

does this by examining the stack after the function returns, so although an error is raised the function has been called.

ctypes uses SEH to prevent crashes from general protection faults due to invalid argument values (raises WindowsError):
>>>

windll.kernel32.GetModuleHandleA(32)

Specifying Functions Prototypes

Specifying the required argument types can be done by the argtypes attribute - a sequence of ctypes types:
strchr

= libc.strchr strchr.argtypes = [ctypes.c_char_p, ctypes.c_char]

By default, functions are assumed to return the C int type. Other return types can be specified by setting the restype attribute:
strchr.restype

= ctypes.c_char_p strchr("abcdef", "d") # Would now return def, and not

Passing Parameters By Reference

The byref() function is used to pass parameters by reference:

f

= ctypes.c_float() # f = 0.0 s = ctypes.create_string_buffer(10) # create a 10 byte empty buffer. libc.sscanf("Input", "%f %s", ctypes.byref(f), s)

pointer() does a lot more work, so it is faster to use byref() if you don't need the pointer object in Python itself.

Structures and Unions

Structures and unions must derive from the Structure and Union base classes. They must contain a _fields_ attribute: a list of 2tuples, containing a field name and a field type. The field type must be a ctypes type. >>> class POINT(ctypes.Structure): ... _fields_ = [("x", ctypes.c_int), ("y", ctypes.c_int)] >>> point = POINT(10, 20) >>> print point.x, point.y

Bit Fields in Structures and Unions

Bit fields are only possible for integer fields. The bit width is specified as the third item in the _fields_ tuples. In C: struct {int word1 :16; int word2 : 16;} DoubleWord; In Python: >>> class DoubleWord(ctypes.Structure): ... _fields_ = [(word1", ctypes.c_int, 16), ... (word2", ctypes.c_int, 16)]

Arrays and Sizes

The way to create array types is by multiplying a data type with the array length. >>> TenIntegers = ctypes.c_int * 10 >>> ii = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) # You can also write: ii = TenIntegers(), and get an array of zeroes. >>> for i in ii: print i

Pointers

Pointer instances are created by calling the pointer() function on a ctypes type:
i = ctypes.c_int(42) pi = ctypes.pointer(i)

pi have a contents attribute which returns the object to which the pointer points (the i object, above):
i2

= ctypes.c_int(99) pi.contents = i2

It is also possible to use indexes to change the pointed value, but you must know what you're changing, just as in C:
pi[0]

= 22

# i2.value is now 22.

Pointer Types and NULL Pointers

A pointer type is being created via the POINTER() factory function:

PI

= ctypes.POINTER(ctypes.c_int) pi = PI(ctypes.c_int(42))

Calling the pointer type without an argument creates a NULL pointer:

null_ptr

= PI()

As already mentioned, to set a POINTER type field to NULL, you can assign None.

Type Checking and Casting

Usually, only instances of the specified type are accepted. cast() takes two parameters, a ctypes object that can be converted to a pointer of some kind, and a ctypes pointer type. It returns an instance of the second argument, which references the same memory block as the first argument:
a

= (ctypes.c_byte * 4)() byte array. pi = ctypes.cast(a, PI) array to an int ptr.

# Creates a zeroed 4# Converts the

Array instances are auto-casted to the compatible pointer types (ii to PI, for example).

Forward Declarations
We can write a linked list on C like this: struct node; // forward declaration. struct { void *data; struct node *next; } node; In ctypes, we can define the cell class and set the _fields_ attribute after the class statement: >>> class node(ctypes.Structure): ... pass >>> node._fields_ = [(data", ctypes.c_void_p), ("next", ctypes.POINTER(node))]

Callback Functions

You can pass Python functions as C callbacks. The CFUNCTYPE factory function creates types for callback functions using the cdecl calling convention. On Windows, the WINFUNCTYPE factory function creates types for callback functions using the stdcall calling convention. Both of these factory functions are called with the result type as first argument, and the callback functions expected argument types as the remaining arguments.

Callback Functions An example

>>> from ctypes import c_int, POINTER >>> IntArray5 = c_int * 5 >>> ia = IntArray5(5, 1, 7, 33, 99) >>> qsort = libc.qsort >>> qsort.restype = None # A void return value. >>> CMPFUNC = CFUNCTYPE(c_int, POINTER(c_int), POINTER(c_int)) >>> def py_cmp_func(a, b): ... return a.contents b.contents >>> qsort(ia, len(ia), ctypes.sizeof(ctypes.c_int), CMPFUNC(py_cmp_func)) # ia is now sorted

Callback Functions Warnings

Make sure you keep references to CFUNCTYPE objects as long as they are used from C code.
Otherwise,

they may be garbage collected, crashing your program when a callback is made.

The result type of callback functions is limited to the predefined fundamental types.
ctypes

1.0.1 (included in Python 2.5) behaves strangely if you use other types.

Function Pointers In Python

In-order to call a function pointer from Python, you need to wrap it with the appropriate callback type. If a function pointer is defined as:
typedef

int (*MyFuncPtr)(void); = ctypes.CFUNCTYPE(ctypes.c_int)

Then the equivalent ctypes type is:

MyFuncPtr

Use WINFUNCTYPE, instead of CFUNCTYPE for the stdcall pointer:

int (__stdcall *MyStdFuncPtr)(void); You can also use this type as a struct field, function return type, etc.
typedef

Auto-Generated Wrapper Code

You can try the ctypeslib code-generator that parses the header file of the DLL and generate the wrapper code from it (currently unstable).

Alternatives For ctypes

SWIG Boost.Python Many more

Using
Pyrex Weave\PyInline and

the Python\C API directly

many more:

SIP Cython PyCXX Etc.

Python-.NET Integration Using Python For .NET

Why Extending Python with .NET Code?

Managed code -> no resource leaks. An extensive runtime library (BCL)

Windows Forms ASP .NET ADO .NET

GUI developing is simpler on the Visual Studio .NET Designer. Cross-Platform:

.NET Framework Mono DotGNU

.NET 3.0 is installed on Windows Vista and Windows Server 2008.

Availability:

Python For .NET (PythonNET)

An integration of the C Python engine with the .NET runtime. Functionality:

Allows

you to use CLR services from Python. Allows you to call user-defined code written in any language that targets the CLR (C#, Managed C++, VB.NET, etc.).

Assembly Loading

Before you can access a namespace in an assembly, you have to load it. Loading an assembly:
import

clr clr.AddReference("System.Windows.Forms") .dll prefix required

# No

Assemblies are loaded from:

The

application directory. The PYTHONPATH (= sys.path). The GAC.

-> To ensure that you can load an assembly, put the directory containing the assembly in sys.path.

Accessing CLR Namespaces

Once their assembly was loaded ,CLR namespaces are treated as Python packages:
from

System.Windows.Forms import Form

You must import the clr module BEFORE accessing the CLR namespaces, even if their containing assembly is already\automatically loaded.
import

clr from System import String, Int32 from System.Collections import *

Using Classes

Python for .NET allows you to use any nonprivate classes, structs, interfaces, enums or delegates from Python. To create an instance of a managed class, you call one of its public constructors:

from System.Collections import Hashtable table = Hashtable() table["key 1"] = "value 1

You can get and set fields and properties of CLR objects just as if they were regular attributes. You can subclass managed classes in Python, though members of the Python subclass are not visible to .NET code:

import System.Windows.Forms as WinForms class HelloApp(WinForms.Form): # Subclass Implementation

Using Methods

All public and protected methods of CLR objects are accessible to Python:
from

System import Environment env_vars = Environment.GetEnvironmentVariables() for env_var in env_vars: print env_var.Key, env_var.Value

Static methods may be called either through the class or through an instance of the class (As above, for Environment).

Type Conversions

Elemental Python types (string, int, long, etc.) convert automatically to compatible managed equivalents (String, Int32, etc.) and vice-versa. All strings returned from the CLR are returned as Unicode. Types that do not have a Pythonic equivalent are exposed as instances of managed classes or structs (System.Decimal, for example).

Using Generics

When running under.NET runtime 2.0+, you can use generic types. A generic type must be bound to create a concrete type before it can be instantiated:
from System.Collections.Generic import my_dict = Dictionary[String, Int32]()

Dictionary

my_dict[a]

=1

You can also pass a subset of Python types that directly correspond to .NET types:
my_dict

= Dictionary[str, int]()

This also works when explicitly selecting generic methods or specific versions of overloaded methods and constructors.

Overloaded and Generic Methods

There are cases where it is desirable to select a particular method overload explicitly. Methods of CLR objects have an "__overloads__" attribute that can be used for this purpose:
from

System import Console Console.WriteLine.__overloads__[bool](true)

Generic methods may be bound at runtime using this syntax directly on the method:
someobject.SomeGenericMethod[str]("10")

Other Features

If a managed object implements one or more indexers, you can call the indexer using standard Python indexing syntax. You can raise and catch managed exceptions just the same as you would pure-Python exceptions. Managed arrays behave like standard Python sequence. Multidimensional arrays support indexing one would use in C#:

arr = System.Array.CreateInstance(String, 3, 2, 4) # Creating a 3 dimensions 3*2*4 array of strings. arr[1, 1, 1] = abc

Managed objects that implement the IEnumerable interface (=Collections) can be iterated over using the standard iteration Python idioms.

Getting Help

The docstring of a CLR method (__doc__) can be used to view the signature of the method, including overloads:
print

Environment.GetFolderPath.__doc__

You can also use the Python help method to inspect a managed class:
help(Environment)

(Explicit) Delegates And Events

A delegate type can be instantiated and passed a callable Python object to get a delegate instance: >>> def my_handler(source, args): print 'my_handler called!' >>> from System import AssemblyLoadEventHandler, AppDomain >>> deleg = AssemblyLoadEventHandler(my_handler) >>> AppDomain.CurrentDomain.AssemblyLoad += deleg # event+=deleg -> use deleg as an event handler. >>> from System.Drawing import Point # The delegate would be called. You can also call it directly: deleg(None, None). You can add callable objects to a delegate instance using: deleg+=method.

(Implicit) Delegates And Events

You do not have to pass an explicitly instantiated delegate instance to an event: >>> class SimpleApp(WinForms.Form): def __init__(self): self.button = WinForms.Button() self.button.Click += self.button_Click # register the event handler (unregister it with -=) self.Controls.Add(self.button) def button_Click(self, sender, args): WinForms.MessageBox.Show(This button was clicked-on!) >>> WinForms.Application.Run(SimpleApp())

The Boxing Problem

An example:
points = System.Array.CreateInstance(Point, points[0].X = 1 # Wont work as you expect!

3)

In C#, the compiler knows that Point is a value type and change the value in place. On Python, the setattr (.X = 1) changes the state of the boxed value, not the original unboxed value It is still 0! Handling the boxing problem:
point = points[0] point.X = 1 points[0] = point

The rule in Python is: "the result of any attribute or item access is a boxed value.

The Alternative: IronPython

An implementation of Python running on .NET. Same syntax as Python For .NET. Pros:
Managed

code -> no resource leaks. Better .NET interoperability. Has (unofficial) support from Microsoft.

Cons:
Cant

use neither Python modules written in C (like ctypes), nor P/Invoke (directly). Not every Python module is implemented (os) or behave the same (re)

Not every Python script would work on it.