Internals of Python 3.x: Derive Maximum Code Performance and Delve Further into Iterations, Objects, GIL, Memory management, and various Internals

CHAPTER 1

Design of Generic Objects

The components of a generic Python object contain the memory layout, operations, and memory management. This chapter covers the similarities and differences between Python types. The chapter begins with the PyObject and explains how it encapsulates the type, reference count, and bookkeeping pointers. The PyVarObject, which is a PyObject encapsulation with the size of the data structure, is covered along with the different functional attributes of the type such as numerical, sub stringing, and so on.

Structure

In this chapter, we will cover the following topics:

The PyObject

Understanding _PyObject_HEAD_EXTRA

Reference counting

The PyVarObject

The PyTypeObject

Generic type function prototypes

Specific type function prototypes

The Type object substructures

The PyNumberMethods substructure

The PySequenceMethods substructure

The PyMappingMethods substructure

The type object

Name and sizes of types

Allocator, deallocator, and initialization functions

Iterator functions

Methods and attributes

Objective

After studying this chapter, you will be able to understand the basic components, the object and the type, which compromise both the data types and data structures. You will also learn about the TypeObject that contains the operations implemented in the data types.

The PyObject

The PyObject contains the details of all Python objects and understanding its structure is crucial before we delve into the implementation of reference counting and Python types, which are key elements of this generic object:

Include/object.h Line no 104

typedef struct _object {

_PyObject_HEAD_EXTRA

Py_ssize_t ob_refcnt;

struct _typeobject *ob_type;

} PyObject;

The PyObject contains three elements, which are as follows:

_PyObject_HEAD_EXTRA: A macro that expands to track all objects in the Python heap in debug mode.

ob_refcnt: An integer value that holds the reference count of the object.

_typeobject: A pointer to the type of the object such as integer/list/dictionary, which contains the operations that can be performed on an instance of the type.

The upcoming chapters contain both the implementation and structure of each of these elements.

Understanding _PyObject_HEAD_EXTRA

The _PyObject_HEAD_EXTRA adds a forward and reverse pointer to every object used to construct a doubly linked list that tracks live objects in the Python heap at runtime. This flag has to be enabled when building the Python executable using the following command:

../configure CFLAGS='-DPy_DEBUG -DPy_TRACE_REFS' --with-pydebug

The code block explaining the definition of the _PyObject_HEAD_EXTRA macro is as follows:

Include/object.h Line no 67

#ifdef Py_TRACE_REFS -> 1

/* Define pointers to support a doubly-linked list of all live heap objects. */

#define _PyObject_HEAD_EXTRA            \

struct _object *_ob_next;  -> 2          \

struct _object *_ob_prev;  -> 3

#define _PyObject_EXTRA_INIT 0, 0,

#else

#define _PyObject_HEAD_EXTRA

#define _PyObject_EXTRA_INIT

#endif

Code insights are as follows:

The _PyObject_HEAD_EXTRA adds the pointers only when Python is compiled with the Py_TRACE_REFS flag enabled.

The _ob_next pointer points to the next created object in the linked list.

The _ob_prev pointer points to the previously created object in the linked list.

The following code block explains the construction of the live object heap as a doubly-linked list:

Objects/object.c (Line no 81)

static PyObject refchain = {&refchain, &refchain};

void _Py_AddToAllObjects(PyObject *op, int force) {

….

if (force || op->_ob_prev == NULL) {

op->_ob_next = refchain._ob_next;

op->_ob_prev = &refchain;

refchain._ob_next->_ob_prev = op;

refchain._ob_next = op;

}

…..

}

The highlighted part explains the construction of the doubly-linked list with reference to the current pointer to the end of the list that is pointed to by refchain.

This reference chain can be accessed using the getobjects method of the sys module. The sample Python code demonstrates how the objects can be fetched using the method:

>>> import sys

>>> objs = sys.getobjects(1) # 1 returns the first object from the linked list.

>>> more_objs = sys.getobjects(20) # Increase the count to return more objects.

>>> type_objs = sys.getobjects(20, str) # A second argument

The following code block explains the implementation of the getobjects function.

Objects/object.c line no 1929

PyObject * _Py_GetObjects(PyObject *self, PyObject *args)

{

int i, n;

PyObject *t = NULL;

PyObject *res, *op;

if (!PyArg_ParseTuple(args, i|O, &n, &t))

return NULL;

op = refchain._ob_next;

res = PyList_New(0);

if (res == NULL)

return NULL;

for (i = 0; (n == 0 || i < n) && op != &refchain; i++) {

while (op == self || op == args || op == res || op == t ||

(t != NULL && Py_TYPE(op) != (PyTypeObject *) t)) {

op = op->_ob_next;

if (op == &refchain)

return res;

}

if (PyList_Append(res, op) < 0) {

Py_DECREF(res);

return NULL;

}

op = op->_ob_next;

}

return res;

}

Reference counting

Python uses reference counting to track the usage of an object and for removing it from the heap on completion of usage. Every object in Python contains the ob_refcnt variable, which contains the current reference count. The reference count is incremented every time the object is used such as adding to a list and decrementing when the usage is completed. Once the reference count reaches 0, the object is removed from the heap using the custom deallocator of the type.

The following Python program demonstrates how the getrefcount method in the sys module can return the current reference count of the object:

>>> import sys

>>> a = 1

>>> sys.getrefcount(a)

114

The following Python program demonstrates how the getrefcount method in the sys module can track changes to the reference count:

>>> import sys

>>> a = 20000000

>>> sys.getrefcount(a)

2

>>> b = [a, a]

>>> sys.getrefcount(a)

4

>>> b.pop()

>>> sys.getrefcount(a)

3

>>> b.pop()

>>> sys.getrefcount(a)

2

>>> def f(a):

print(sys.getrefcount(a))

>>> a = 2000000

>>> sys.getrefcount(a)

2

>>> f(a)

4

The function returns 4 because the sys.refcount function also increments the reference count by 1.

The following C Python code handles reference counting:

typedef struct _object {

_PyObject_HEAD_EXTRA

Py_ssize_t ob_refcnt;

struct _typeobject *ob_type;

} PyObject;

The ob_refcnt variable in the PyObject structure holds the reference count of every PyObject, which is manipulated using two macros _Py_INCREF and _Py_DECREF.

The code block explaining incrementing of reference counting is as follows:

static inline void _Py_INCREF(PyObject *op)

{

_Py_INC_REFTOTAL;

op->ob_refcnt++;

}

#define Py_INCREF(op) _Py_INCREF(_PyObject_CAST(op))

The macro Py_INCREF increments the reference count of the variable by 1:

static inline void _Py_DECREF(const char *filename, int lineno,

PyObject *op)

{

(void)filename; /* may be unused, shut up -Wunused-parameter */

(void)lineno; /* may be unused, shut up -Wunused-parameter */

_Py_DEC_REFTOTAL;

if (--op->ob_refcnt != 0) {

#ifdef Py_REF_DEBUG

if (op->ob_refcnt < 0) {

_Py_NegativeRefcount(filename, lineno, op);

}

#endif

}

else {

_Py_Dealloc(op);

}

}

#define Py_DECREF(op) _Py_DECREF(__FILE__, __LINE__, _PyObject_CAST(op))

Code insights are as follows:

Decrement the reference count when the requestor completes usage of the object.

Deallocate the object when the reference count becomes 0. The deallocator depends on the type of the object, which will be covered in the subsequent chapters on basic and iterable types.

The PyVarObject

The PyVarObject is the generic container object that holds the data for lists, types, dictionaries, and sets:

Objects/object.h Line no 113

typedef struct {

PyObject ob_base;

Py_ssize_t ob_size; /* Number of items in variable part */

} PyVarObject;

The PyVarObject contains a PyObject for data storage and the size of the