Metaprogramming with Metaclasses in Python

Metaprogramming in Python lets us write code that can modify or generate other code at runtime. One of the key tools for achieving this is metaclasses, which allow us to control the creation and behavior of classes.

What Are Metaclasses?

Metaclasses are classes that define how other classes are created. While regular classes act as blueprints for creating objects, metaclasses serve as blueprints for creating classes themselves. When we create a class in Python, it is, in fact, an instance of a metaclass. By default, every class in Python is an instance of the built-in type metaclass.

This whole meta thing can be summarized as - Metaclass create Classes and Classes creates objects.

The metaclass is responsible for the generation of classes, so we can write our custom metaclasses to modify the way classes are generated by performing extra actions or injecting code. Usually, we do not need custom metaclasses but sometimes it's necessary.

Creating custom Metaclass

To create our custom metaclass, our custom metaclass has to inherit type metaclass and usually override - 

• __new__(): It's a method which is called before __init__(). It creates the object and returns it. We can override this method to control how the objects are created.
• __init__(): This method just initialize the created object passed as a parameter

We can create classes using the type() function directly. It can be called in following ways - 

1. When called with only one argument, it returns the type. We have seen it before in the above examples.
2. When called with three parameters, it creates a class. Following arguments are passed to it - 
   
   1. Class name
   2. Tuple having base classes inherited by class
   3. Class Dictionary: It serves as a local namespace for the class, populated with class methods and variables

Consider this example -  

def test(self):
    print("This is Test class method!")

class Base:
    def myfun(self):
        print("This is inherited method!")

# Creating Test class dynamically using type()
Test = type('Test', (Base, ), dict(x="ankit", my_method=test))

print("Type of Test class: ", type(Test))

obj = Test()
print("Type of obj: ", type(obj))

obj.myfun()

obj.my_method()

print(obj.x)

Type of Test class:  <class 'type'>
Type of obj:  <class '__main__.Test'>
This is inherited method!
This is Test class method!
ankit

Explanation:

• test Method prints "This is Test class method!".
• Base Class has myfun method that prints "This is inherited method!".
• Dynamic Test Class is created with type(), inherits from Base, adds x="ankit" and my_method=test.
• Prints types of Test and obj, calls myfun() and my_method(), then prints x ("ankit").

Now let's create a metaclass without using type() directly. In the following example, we will be creating a metaclass MultiBases which will check if the class being created has inherited from more than one base class. If so, it will raise an error. 

class MultiBases(type):
    # overriding __new__ method
    def __new__(cls, clsname, bases, clsdict):
        # if no of base classes is greater than 1 raise error
        if len(bases)>1:
            raise TypeError("Inherited multiple base classes!!!")
        
        # else execute __new__ method of super class, ie. call __init__ of type class
        return super().__new__(cls, clsname, bases, clsdict)

class Base(metaclass=MultiBases):
    pass

class A(Base):
    pass

class B(Base):
    pass

class C(A, B):
    pass

Output: 

Traceback (most recent call last): 
  File "main.py", line 20, in <module> 
    class C(A, B): 
  File "main.py", line 6, in __new__ 
    raise TypeError("Inherited multiple base classes!!!") 
TypeError: Inherited multiple base classes!!!

Explanation: This code defines a custom metaclass MultiBases that restricts classes from inheriting from more than one base class. In the __new__ method, it checks if the class has more than one base class and raises a TypeError if so. The Base class uses this metaclass, and while A and B (subclasses of Base) don't raise any errors, C (inheriting from both A and B) triggers a TypeError.

Solving problems with metaclass

Some problems in Python can be solved using decorators or metaclasses. While decorators provide a simple solution for many tasks, there are certain situations where only metaclasses can provide a more efficient or scalable solution.

Problem: Debugging Class Methods

The task is to debug class methods, meaning every time a method is executed, it should print its fully qualified name before executing its body.

Solution 1: Using Method Decorators

Here's an implementation using method decorators to debug the methods of a class:

from functools import wraps

def debug(func):
    '''decorator for debugging passed function'''
    
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("Full name of this method:", func.__qualname__)
        return func(*args, **kwargs)
    return wrapper

def debugmethods(cls):
    '''class decorator make use of debug decorator
       to debug class methods '''
    
    # check in class dictionary for any callable(method) if exist, replace it with debugged version
    for key, val in vars(cls).items():
        if callable(val):
            setattr(cls, key, debug(val))
    return cls

# sample class
@debugmethods
class Calc:
    def add(self, x, y):
        return x+y
    def mul(self, x, y):
        return x*y
    def div(self, x, y):
        return x/y
    
mycal = Calc()
print(mycal.add(2, 3))
print(mycal.mul(5, 2))

Full name of this method: Calc.add
5
Full name of this method: Calc.mul
10

Explanation:

• debug Decorator: This decorator prints the fully qualified name of the method (func.__qualname__) and then calls the method.
• debugmethods Class Decorator: This decorator scans the class dictionary for callable methods and applies the debug decorator to them.
• Calc Class: The Calc class uses the @debugmethods decorator, so its methods (add, mul, and div) will print their fully qualified names when called.

Solution 2: Using a Metaclass

Now, let's look at a metaclass-based solution that automatically applies the debugging functionality to all subclasses of a base class.

from functools import wraps

def debug(func):
    '''decorator for debugging passed function'''
    
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("Full name of this method:", func.__qualname__)
        return func(*args, **kwargs)
    return wrapper

def debugmethods(cls):
    '''class decorator make use of debug decorator to debug class methods '''
    
    for key, val in vars(cls).items():
        if callable(val):
            setattr(cls, key, debug(val))
    return cls

class debugMeta(type):
    '''meta class which feed created class object to debugmethod to get debug functionality enabled objects'''
    
    def __new__(cls, clsname, bases, clsdict):
        obj = super().__new__(cls, clsname, bases, clsdict)
        obj = debugmethods(obj)
        return obj
    
# base class with metaclass 'debugMeta' now all the subclass of this will have debugging applied
class Base(metaclass=debugMeta):pass

# inheriting Base
class Calc(Base):
    def add(self, x, y):
        return x+y
    
# inheriting Calc
class Calc_adv(Calc):
    def mul(self, x, y):
        return x*y

# Now Calc_adv object showing debugging behaviour
mycal = Calc_adv()
print(mycal.mul(2, 3))

Full name of this method: Calc_adv.mul
6

Explanation:

• debug Decorator: Same as in the first solution, it prints the fully qualified name of the method and calls it.
• debugmethods Class Decorator: This applies the debug decorator to all methods of a class.
• debugMeta Metaclass: The debugMeta metaclass overrides the __new__ method to automatically apply the debugmethods decorator to the class when it is created. This ensures that all methods in any class using this metaclass will have debugging enabled without needing to explicitly apply the decorator.
• Base Class: The Base class uses the debugMeta metaclass, so all subclasses will automatically have debugging applied to their methods.

When to use Metaclasses

Most of the time we do not use metaclasses, it's usually used for something complicated, but a few cases where we use metaclasses are - 

• As we have seen in the above example, metaclasses propagate down the inheritance hierarchies. It will affect all the subclasses as well. If we have such a situation, then we should use metaclasses.
• If we want to change class automatically, when it is created, we use metaclasses
• For API development, we might use metaclasses

As quoted by Tim Peters 

Metaclasses are deeper magic that 99% of users should never worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why).