10/11/2019 Learning Python: Unit 16, Embedding

16. Embedding Python in C/C++

♦ C/C++ runs Python code via API calls

♦ For dynamic system customization

Embedding topics

♦ Calling objects
♦ Running code strings
♦ Registration techniques
♦ Other topics: errors, tools, etc.

General embedding concepts

Embedded code forms

♦ Code strings
● Running expressions, statements

♦ Callable objects
● Calling functions, classes, methods

♦ Code files
https://learning-python.com/class/Workbook/unit16.htm 1/23
10/11/2019 Learning Python: Unit 16, Embedding

● Importing modules, executing scripts

Embedded code sources

Source Description
Modules fetching code by importing modules
Text files fetching code from simple text files
Registration letting Python pass code to a C extension
HTML tags extracting code from web pages
Databases fetching code from a database table
Processes receiving code over sockets
Construction building Python code in C at runtime
And so on system registries, etc.

Common code sources

♦ Callable objects… modules, registration

♦ Code strings… files, registration, HTML,…

♦ Code files… files, modules, scripts

Data communication techniques

Code form Technique Mode

Objects function arguments In/out
Objects function return values Out
Strings expression results Out
Strings, Objects global module-level variables In/out
https://learning-python.com/class/Workbook/unit16.htm 2/23
10/11/2019 Learning Python: Unit 16, Embedding

Strings, Objects C extension get/set functions In/out

Strings, Objects files, stdin/stdout, sockets,… In/out

Running simple code strings

♦ ‘Py_Initialize’ initializes Python libraries

♦ ‘PyRun_SimpleString’ runs Python statements

♦ Runs in module ‘__main__’, no result from code

♦ Python code uses ‘environ’ extension module too

file: main1.c
#include <Python.h>

main(argc, argv)
int argc;
char **argv;
{
/* This is the simplest embedding mode. */
/* Other API functions return results, */
/* accept namespace arguments, allow */
/* access to real Python objects, etc. */
/* Strings may be precompiled for speed. */

Py_Initialize(); /* init python */

PyRun_SimpleString("print 'Hello embedded world!'");

/* use C extension module above */

PyRun_SimpleString("from environ import *");
PyRun_SimpleString(
"for i in range(4):\n"
"\tprint i,\n"
"\tprint 'Hello, %s' % getenv('USER')\n\n" );

PyRun_SimpleString("print 'Bye embedded world!'");

}

https://learning-python.com/class/Workbook/unit16.htm 3/23
10/11/2019 Learning Python: Unit 16, Embedding

Running the C program

% main1
Hello embedded world!
0 Hello, mlutz
1 Hello, mlutz
2 Hello, mlutz
3 Hello, mlutz
Bye embedded world!

Same as typing at “>>>” prompt:

print 'Hello embedded world!'
from environ import *
for i in range(4):
print i,
print 'Hello, %s' % getenv('USER')
print 'Bye embedded world!

Building programs that embed Python

♦ Link with Python library (1), and your main()

♦ Add any libs referenced by Python lib (Modules/Setup)

♦ Older scheme (1.4): 4 Python .a libs + 2 Python .o files

file: Makefile.main1
# this file builds a C executable that embeds Python
# assuming no external module libs must be linked in
# works on Linux with a custom Python build tree

PY = /home/mark/python1.5.2-ddjcd/Python-1.5.2
PYLIB = $(PY)/libpython1.5.a
https://learning-python.com/class/Workbook/unit16.htm 4/23
10/11/2019 Learning Python: Unit 16, Embedding

PYINC = -I$(PY)/Include -I$(PY)

basic1: basic1.o
cc basic1.o $(PYLIB) -g -export-dynamic -lm -ldl -o basic1

basic1.o: basic1.c
cc basic1.c -c -g $(PYINC)

Calling objects and methods

The example task

♦ Make an instance of a Python class in a module file
♦ Call a method of that instance by name
♦ Python equivalent:
import <module>
object = <module>.<class>()
result = object.<method>(..args..)

Python API tools to be used

Tool Description
PyObject* The type of a generic Python object
in C
PyImport_ImportModule Imports a Python module, much
like the Python ‘import’
PyObject_GetAttrString Performs attribute qualifications:
‘object.name’, like ‘getattr’
PyEval_CallObject Calls any callable object (class,
function, method), like ‘apply’
Py_BuildValue Converts C data to Python form,
based on a format string
PyArg_Parse Converts Python data to C form,
based on a format string
Py_DECREF Release ownership of object
passed to C from the API
https://learning-python.com/class/Workbook/unit16.htm 5/23
10/11/2019 Learning Python: Unit 16, Embedding

The task implementation

file: module.py (on $PYTHONPATH)

class klass:
def method(self, x, y):
return "brave %s %s" % (x, y) # run me from C

file: objects1.c
#include <Python.h>
#include <stdio.h>

main() {
char *arg1="sir", *arg2="robin", *cstr;
PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;

/* instance = module.klass() */
Py_Initialize();
pmod = PyImport_ImportModule("module");
pclass = PyObject_GetAttrString(pmod, "klass");
Py_DECREF(pmod);

pargs = Py_BuildValue("()");
pinst = PyEval_CallObject(pclass, pargs);
Py_DECREF(pclass);
Py_DECREF(pargs);

/* result = instance.method(x,y) */
pmeth = PyObject_GetAttrString(pinst, "method");
Py_DECREF(pinst);
pargs = Py_BuildValue("(ss)", arg1, arg2);
pres = PyEval_CallObject(pmeth, pargs);
Py_DECREF(pmeth);
Py_DECREF(pargs);

PyArg_Parse(pres, "s", &cstr); /* convert to C */

printf("%s\n", cstr);
Py_DECREF(pres);
}

% objects1
brave sir robin

With full error checking (!)

file: objects1err.c
#include <Python.h>
#include <stdio.h>
#define error(msg) do { printf("%s\n", msg); exit(1); } while (1)

https://learning-python.com/class/Workbook/unit16.htm 6/23
10/11/2019 Learning Python: Unit 16, Embedding
main() {
char *arg1="sir", *arg2="robin", *cstr;
PyObject *pmod, *pclass, *pargs, *pinst, *pmeth, *pres;

/* instance = module.klass() */
Py_Initialize();
pmod = PyImport_ImportModule("module");
if (pmod == NULL)
error("Can't load module");
pclass = PyObject_GetAttrString(pmod, "klass");
Py_DECREF(pmod);
if (pclass == NULL)
error("Can't get module.klass");

pargs = Py_BuildValue("()");
if (pargs == NULL) {
Py_DECREF(pclass);
error("Can't build arguments list");
}
pinst = PyEval_CallObject(pclass, pargs);
Py_DECREF(pclass);
Py_DECREF(pargs);
if (pinst == NULL)
error("Error calling module.klass()");

/* result = instance.method(x,y) */
pmeth = PyObject_GetAttrString(pinst, "method");
Py_DECREF(pinst);
if (pmeth == NULL)
error("Can't fetch klass.method");
pargs = Py_BuildValue("(ss)", arg1, arg2);
if (pargs == NULL) {
Py_DECREF(pmeth);
error("Can't build arguments list");
}
pres = PyEval_CallObject(pmeth, pargs);
Py_DECREF(pmeth);
Py_DECREF(pargs);
if (pres == NULL)
error("Error calling klass.method");

if (!PyArg_Parse(pres, "s", &cstr)) /* convert to C */

error("Can't convert klass.method result");
printf("%s\n", cstr);
Py_DECREF(pres);
}

The easy way: an extended API

♦ Higher-level interface from “Programming Python”

♦ Automates errors, reference counts, conversions

♦ Supports reloading and debugging embedded code

♦ Link with Python libs/objects + “pyembed*” files

https://learning-python.com/class/Workbook/unit16.htm 7/23
10/11/2019 Learning Python: Unit 16, Embedding

file: objects2.c
#include <stdio.h>
#include "pyembed.h"

main () {
int failflag;
PyObject *pinst;
char *arg1="sir", *arg2="robin", *cstr;

failflag =
Run_Function("module", "klass", /* module.klass() */
"O", &pinst, "()") /* result, args */
||
Run_Method(pinst, "method", /* pinst.method() */
"s", &cstr, /* result fmt/ptr */
"(ss)", arg1, arg2); /* args fmt/values*/

printf("%s\n", (!failflag) ? cstr : "Can't call objects");

Py_XDECREF(pinst);
}

% objects2
brave sir robin

Running strings: results & name-spaces

♦ Runs: [ upper('spam') + '!' ] in string module

♦ Expression result comes back as a Python object
♦ ‘PyModule_GetDict’: module’s name-space dictionary
♦ ‘PyRun_String’: runs a code string in name-spaces
♦ Parse-mode flag: ‘Py_eval_input’ = expression

file codestring1.c
#include <Python.h> /* standard API defs */

main() {
/* error checking omitted! */
char *cstr;
PyObject *pstr, *pmod, *pdict;
Py_Initialize();

/* result = string.upper('spam') + '!' */

pmod = PyImport_ImportModule("string"); /* namespace */
https://learning-python.com/class/Workbook/unit16.htm 8/23
10/11/2019 Learning Python: Unit 16, Embedding

pdict = PyModule_GetDict(pmod);
pstr = PyRun_String("upper('spam') + '!'",
Py_eval_input, pdict, pdict);

/* convert result to C */
PyArg_Parse(pstr, "s", &cstr);
printf("%s\n", cstr);
Py_DECREF(pmod);
Py_DECREF(pstr); /* free exported objects */
}

% codestring1
SPAM!

The easy way, part II: extended API

♦ Automates code strings, object calls, and methods

♦ Also supports running strings without modules

file: codestring2.c
#include "pyembed.h"
#include <stdio.h>

main() {
char *cstr;
int err =
Run_Codestr(
PY_EXPRESSION, /* expr|stmt? */
"upper('spam') + '!'", "string", /* code,module */
"s", &cstr); /* expr result */
printf("%s\n", (!err) ? cstr : "Can't run string");
}

% codestring2
SPAM!

Other code string possibilities

♦ Making new dictionaries for string name-spaces

https://learning-python.com/class/Workbook/unit16.htm 9/23
10/11/2019 Learning Python: Unit 16, Embedding

♦ Fetching strings from files, modules, HTML, databases

♦ Exporting C extension functions for communication
♦ Global variables for communication: ‘copy-in-copy-out’

Copy-in-copy-out

validate.py

# use QUANTITY, ORDERTYPE

# set QUANTITY, MESSAGES
[. . .]

C code…

/* set input vars */

char *string = . . .
Set_Global("validate", "QUANTITY", "i", quantity);
Set_Global("validate", "ORDERTYPE", "s", ordertype);

/* run code string */

status = Run_Codestr(PY_STATEMENT,
string, "validate", "", NULL);
if (status == -1)
PyErr_Print(); /* stack traceback */
else {
/* fetch output vars */
Get_Global("validate", "QUANTITY", "i", &quantity);
Get_Global("validate", "MESSAGES", "s", &messages);
}

Making namespace dictionaries

♦ Shamelessly stolen from “Programming Python”

♦ Same as using built-in ‘eval’/‘exec’, ‘{}’, ‘X[key]’:
dict = {}
dict['Y'] = 2
exec 'X = 99' in dict, dict
exec 'X = X+Y' in dict, dict
print dict['X'] # => 101

https://learning-python.com/class/Workbook/unit16.htm 10/23
10/11/2019 Learning Python: Unit 16, Embedding

file: basic4.c
#include <Python.h>

main() {
int cval;
PyObject *pdict, *pval;
Py_Initialize();

/* make a new namespace */

pdict = PyDict_New();
PyDict_SetItemString(pdict, "__builtins__",
PyEval_GetBuiltins());
/* dict['Y'] = 2 */
PyDict_SetItemString(pdict, "Y", PyInt_FromLong(2));

PyRun_String("X = 99", Py_file_input, pdict, pdict);

PyRun_String("X = X+Y", Py_file_input, pdict, pdict);

/* fetch dict['X'] */
pval = PyDict_GetItemString(pdict, "X");
PyArg_Parse(pval, "i", &cval); /* convert to C */
printf("%d\n", cval); /* result=101 */
Py_DECREF(pdict);
}

Registering Python objects and strings

♦ A strategy for code location/source

♦ Python passes code to a C extension
♦ C saves the code and runs it later
♦ May register objects or code strings
♦ Works best if Python is ‘on top’

Components: extending + embedding

♦ A C extension module
https://learning-python.com/class/Workbook/unit16.htm 11/23
10/11/2019 Learning Python: Unit 16, Embedding

● Exports a registration function to Python (‘setHandler’)

♦ A C event routing function

● Calls the registered Python object in response to events

♦ A Python client program

● Registers functions, triggers events

Registration implementation
file: cregister.c
#include <Python.h>
#include <stdlib.h>

/***********************************************/
/* 1) code to route events to Python object */
/* note that we could run strings here instead */
/***********************************************/

static PyObject *Handler = NULL; /* Python object in C */

void Route_Event(char *label, int count)

{
char *cres;
PyObject *args, *pres;

/* call Python handler */

args = Py_BuildValue("(si)", label, count);
pres = PyEval_CallObject(Handler, args);
Py_DECREF(args);

if (pres != NULL) {
/* use and decref handler result */
PyArg_Parse(pres, "s", &cres);
printf("%s\n", cres);
Py_DECREF(pres);
}
}

/*****************************************************/
/* 2) python extension module to register handlers */
/* python imports this module to set handler objects */
/*****************************************************/

static PyObject *
Register_Handler(PyObject *self, PyObject *args)
{
/* save Python callable object */
Py_XDECREF(Handler); /* called before? */
PyArg_Parse(args, "O", &Handler); /* one argument? */
https://learning-python.com/class/Workbook/unit16.htm 12/23
10/11/2019 Learning Python: Unit 16, Embedding

Py_XINCREF(Handler); /* add reference */

Py_INCREF(Py_None); /* None=success */
return Py_None;
}

static PyObject *
Trigger_Event(PyObject *self, PyObject *args)
{
/* let Python simulate event caught by C */
static count = 0;
Route_Event("spam", count++);
Py_INCREF(Py_None);
return Py_None;
}

static struct PyMethodDef cregister_methods[] = {

{"setHandler", Register_Handler}, /* name, addr */
{"triggerEvent", Trigger_Event},
{NULL, NULL}
};

void initcregister() /* this is called by Python */

{ /* on first "import cregister" */
(void) Py_InitModule("cregister", cregister_methods);
}

Python client program

♦ Combines extending and embedding

♦ Extending: Python encloses C
♦ Embedding: C calls Python on events
♦ C may also call embedded python code to register

file: register.py

# handle an event, return a result (or None)

def function1(label, count):

return "%s number %i..." % (label, count)

def function2(label, count):

return label * count

# register handlers, trigger events

import cregister
https://learning-python.com/class/Workbook/unit16.htm 13/23
10/11/2019 Learning Python: Unit 16, Embedding

cregister.setHandler(function1)
for i in range(3):
cregister.triggerEvent() # simulate events caught by C

cregister.setHandler(function2)
for i in range(3):
cregister.triggerEvent() # routes events to function2

% python register.py
spam number 0...
spam number 1...
spam number 2...
spamspamspam
spamspamspamspam
spamspamspamspamspam

Registration tradeoffs

♦ +: Granularity
Associating actions with objects without external files

♦ -: Application structure
Requires Python on top, or extra embedding logic

♦ -: Complexity
Might require extra coding step to register code

♦ -: Reloading code
Difficult to reload without going through modules

Building the example

Solaris
# add the extension modules as dynamically linked modules;
# when first imported, they are located in a dir on
# $PYTHONPATH, and loaded into the process.

cregister.so: cregister.c
$(Cc) cregister.c $(CFLAGS) -DDEBIG -KPIC -o cregister.o
ld -G cregister.o -o $@

environ.so: environ.c
$(Cc) environ.c $(CFLAGS) -KPIC -o environ.o
https://learning-python.com/class/Workbook/unit16.htm 14/23
10/11/2019 Learning Python: Unit 16, Embedding

ld -G environ.o -o $@

Linux
PY = /home/mark/python1.5.2-ddjcd/Python-1.5.2
PYINC = -I$(PY)/Include -I$(PY)
cregister.so: cregister.c
gcc cregister.c -g $(PYINC) -fpic -shared

Accessing C variables in Python

♦ Using an extension module

import cvars # import C variable wrapper module
cvars.setX(24) # set C's X from Python
print cvars.getY() # fetch C's Y from Python

♦ Using an extension type

import cvars # import C variable wrapper module
c = cvars.interace() # make interface object instance
c.X = (24) # set attributes -> C variable
print c.Y # get attributes -> C variable

♦ Using copy-in-copy-out
1. C copies X and Y values to variables in module M
2. C runs embedded code in module M’s name-space
3. C fetches the final values of the Python variables

C API equivalents in Python

♦ Python may be embedded in C or Python
♦ See section “Dynamic coding” for Python tools

C API tool Python tool

PyImport_ImportModule import
PyEval_CallObject apply
PyRun_String exec/eval
PyObject_GetAttrString getattr

https://learning-python.com/class/Workbook/unit16.htm 15/23
10/11/2019 Learning Python: Unit 16, Embedding

Running code files from C

♦ ‘PyRun_File’, ‘PyRun_SimpleFile’
♦ Module imports and reloads
♦ system, popen, fork/exec

Precompiling strings into byte-code

● Modules are compiled once, on first import

● Raw strings compiled when run, unless precompile

♦ Compile to byte code object

PyCodeObject*
Py_CompileString(char *string,
char *filename, int parsemode);

♦ Run byte-code object

PyObject*
PyEval_EvalCode(PyCodeObject *code,
PyObject *globalnamesdict,
PyObject *localnamesdict);

Embedding under C++

● Python is coded in portable ANSI C

https://learning-python.com/class/Workbook/unit16.htm 16/23
10/11/2019 Learning Python: Unit 16, Embedding

● Normal C++ → C mixing rules apply

● No special considerations when embedding

♦ Python header files

● Automatically wrapped in extern “C”, and may be included in C++
programs freely

♦ Python libraries
● Don’t need to be recompiled by the C++ compiler to link: API entry
points are all extern “C”

More on object reference counts

♦ Python uses reference-count garbage collection

♦ New objects returned to C with a reference
♦ C must INCREF other objects to retain them
♦ C must DECREF objects it no longer needs
♦ XINCREF/XDECREF ignore NULL pointers

Resources:

♦ New API documentation

● http://www.python.org/doc, Python/C API manual

♦ Older API documentation

● http://www.python.org/doc/ext/ext.html

https://learning-python.com/class/Workbook/unit16.htm 17/23
10/11/2019 Learning Python: Unit 16, Embedding

♦ Other references
● Programming Python, Editions 2+

♦ Abstract object API

● Tools in "abstract.h" header file are well-defined

♦ Extended API in chapter 15 of “Programming Python”

● Run_Function, etc., handle most details

♦ Embedding examples in books, on the ‘net, etc.

● http://rmi.net/~lutz/newex.html

♦ Check the C source code of the API

● Not as hard as you may think!

Common API calls

New Name (1.3+) INCREFs? Python Equivalent
PyImport_AddModule No n/a
PyImport_ImportModule Yes import module
PyImport_ReloadModule Yes reload(module)
PyImport_GetModuleDict No sys.modules
PyModule_GetDict No module.__dict__
PyDict_GetItemString No dict[key]
PyDict_SetItemString n/a dict[key]=val
PyObject_GetAttrString Yes getattr(obj, attr)
PyObject_SetAttrString n/a setattr(obj, attr, val)
PyArg_ParseTuple n/a n/a
Py_BuildValue Yes n/a
PyEval_CallObject Yes apply(func, argtuple)
PyRun_String Yes eval(expr), exec stmt
PyErr_Print n/a traceback.print_exc
PyDict_New Yes {}

https://learning-python.com/class/Workbook/unit16.htm 18/23
10/11/2019 Learning Python: Unit 16, Embedding

Integration error handling

♦ Extending: sending errors to Python

● C extensions raise exceptions by returning NULL

● ‘PyErr_SetString’ sets exception’s name and data

● C may propagate error instead of setting

♦ Embedding: catching Python errors

● API return values: NULL, or integer status codes

● ‘PyErr_Fetch’ fetches exception info

● ‘PyErr_Print’ shows Python stack traceback on stderr

● Extended API functions return -1 or NULL on first error

Fetching exception information in C

file: pyerrors.c
#include <Python.h>
#include <stdio.h>
char save_error_type[1024], save_error_info[1024];

PyerrorHandler(char *msgFromC)
{
/* process Python-related errors */
/* call after Python API raises an exception */

PyObject *errobj, *errdata, *errtraceback, *pystring;

printf("%s\n", msgFromC);

/* get latest python exception info */

PyErr_Fetch(&errobj, &errdata, &errtraceback);

pystring = NULL;
if (errobj != NULL &&
(pystring = PyObject_Str(errobj)) != NULL &&
(PyString_Check(pystring))
)
strcpy(save_error_type, PyString_AsString(pystring));
else
https://learning-python.com/class/Workbook/unit16.htm 19/23
10/11/2019 Learning Python: Unit 16, Embedding

strcpy(save_error_type, "<unknown exception type>");

Py_XDECREF(pystring);

pystring = NULL;
if (errdata != NULL &&
(pystring = PyObject_Str(errdata)) != NULL &&
(PyString_Check(pystring))
)
strcpy(save_error_info, PyString_AsString(pystring));
else
strcpy(save_error_info, "<unknown exception data>");
Py_XDECREF(pystring);

printf("%s\n%s\n", save_error_type, save_error_info);

Py_XDECREF(errobj);
Py_XDECREF(errdata); /* caller owns all 3 */
Py_XDECREF(errtraceback); /* already NULL'd out */
}

Automated integration tools

♦ SWIG
● Generates Python interfaces to external C/C++ libraries
● Uses C/C++ declarations and interface description files
● For C++ class: generates C type + Python wrapper class
● See unit 15, the book Programming Python editions 2+, and
http://www.swig.org/

♦ ILU
● Implements CORBA distributed-object systems
● Uses interface description files
● Platform and language independent

♦ Abstract object API

● C API to create/process Python objects generically
● In Python’s “abstract.h”: exports slicing, concatenation, etc.

♦ Embedded call API

● C API to simplify common embedding tasks
https://learning-python.com/class/Workbook/unit16.htm 20/23
10/11/2019 Learning Python: Unit 16, Embedding

● In “Programming Python”, editions 1..3

♦ Modulator
● Tkinter GUI: generates skeleton C module/type files
● Users fill in the blanks with application-specific logic
● An old idea, which has been revived recently (forgetting history
much?)

♦ Other: ActiveX/COM interfaces

Modulator in action (old)

https://learning-python.com/class/Workbook/unit16.htm 21/23
10/11/2019 Learning Python: Unit 16, Embedding

Lab Session 12
Click here to go to lab exercises

Click here to go to exercise solutions

Click here to go to solution source files

Click here to go to lecture example files

https://learning-python.com/class/Workbook/unit16.htm 22/23
10/11/2019 Learning Python: Unit 16, Embedding

https://learning-python.com/class/Workbook/unit16.htm 23/23