Import All Submodules of a Python Namespace Package

What are Namespace Packages?

Namespace packages are a type of package in Python that allows you to split the sub-packages and modules within a single package across multiple, separate distribution packages. Unlike normal packages, the namespace packages don't require _init_.py file.

Automatically importing all submodules within a namespace package serves several purposes, like auto-registration without manually importing, and to load all available plugins in a system. This article discusses the two methods that you can use to import all the sub-modules of a Python namespace package -

• Using pkgutil.iter_modules()
• Using importlib.resources

Using pkgutil.iter_modules()

The pkgutil.iter_modules() is used to find and list the submodules and subpackages within a given package. It returns an iterator that has ModuleInfo objects, each containing information about a found module or package. The code below demonstrates how the pkgutil.iter_modules() is used to iterate through all modules -

import pkgutil
import importlib
import my_namespace_package

def import_submodules(package):
    """Import all submodules of a module, recursively."""
    if isinstance(package, str):
        package = importlib.import_module(package)
    
    results = {}
    
    for _, name, is_pkg in pkgutil.iter_modules(package.__path__, package.__name__ + '.'):
        full_name = name
        results[full_name] = importlib.import_module(full_name)
        
        if is_pkg:
            results.update(import_submodules(full_name))
            
    return results

# Usage
modules = import_submodules('my_namespace_package')

The code above imports all submodules and subpackages of a given package and returns a dictionary where keys are module names and values are the actual imported module objects.

Using importlib.resources

The importlib.resources module allows Python's import system to access resources within a package. A resource is any readable object contained in the package. This is a modern approach and is effective for newer Python versions. The code below illustrates how importlib.resources is used to import all the submodules of a Python namespace package -

import importlib.resources
import importlib.util
import sys
import types
from pathlib import Path

def import_submodules_with_resources(package_name):
    """
    Import all submodules using importlib.resources.files (Python 3.9+)
    
    Args:
        package_name: String name of the package
        
    Returns:
        Dictionary of imported modules
    """
    results = {}
    
    try:
        # Get package as module first
        package = importlib.import_module(package_name)
        
        # Skip if not a package
        if not isinstance(package, types.ModuleType) or not hasattr(package, '__path__'):
            return results
            
        # Use files() to get a traversable object (Python 3.9+)
        package_path = importlib.resources.files(package_name)
        
        # Process all items in the package
        for item_path in package_path.iterdir():
            # Handle Python modules
            if item_path.is_file() and item_path.name.endswith('.py') and not item_path.name.startswith('__'):
                module_name = f"{package_name}.{item_path.name[:-3]}"
                try:
                    module = importlib.import_module(module_name)
                    results[module_name] = module
                except ImportError as e:
                    print(f"Failed to import {module_name}: {e}")
                    
            # Handle potential subpackages
            elif item_path.is_dir() and not item_path.name.startswith('__'):
                subpackage_name = f"{package_name}.{item_path.name}"
                
                # Use find_spec to check if it's a valid package
                # This works for both regular and namespace packages
                spec = importlib.util.find_spec(subpackage_name)
                if spec is not None:
                    try:
                        subpackage = importlib.import_module(subpackage_name)
                        results[subpackage_name] = subpackage
                        
                        # Recursively import submodules
                        sub_results = import_submodules_with_resources(subpackage_name)
                        results.update(sub_results)
                    except ImportError as e:
                        print(f"Failed to import {subpackage_name}: {e}")
    except ModuleNotFoundError as e:
        print(f"Package {package_name} not found: {e}")
    except ImportError as e:
        print(f"Error importing package {package_name}: {e}")
            
    return results

The code above uses importlib.resources.files() to iterate over the contents in a given package. It returns a dictionary that maps fully qualified module names to their corresponding imported module objects.