How to Limit Heap Size in Python?

In Python, the heap size is managed automatically by the interpreter and the Garbage Collector (GC), which makes Python simpler than low-level languages like C or C++. Python doesn't provide direct way to limit Heap Memory. However, there are ways to limit the heap size if you are working on systems with restricted memory, debugging memory usage, or handling large-scale applications.

Using resource Module (POSIX Systems like Linux/macOS)

resource module allows you to set limits on system resources and memory usage. It can be used to limit heap size by restricting the amount of memory a Python process can use.

import resource
import sys

# Set maximum heap size (e.g., 500 MB)
max_heap_size = 500 * 1024 * 1024  # 500 MB in bytes

# Set memory limit
resource.setrlimit(resource.RLIMIT_AS, (max_heap_size, max_heap_size))

try:
    # Simulate a memory-heavy operation
    print("Allocating memory...")
    large_list = [0] * (10**8)  # This will consume significant memory
except MemoryError:
    print("Memory limit reached! Exiting gracefully.")
    sys.exit(1)

print("Memory allocation successful within limit.")

How It Works?

setrlimit() function in the resource module can be used to set the memory limit. Specifically, it sets the RLIMIT_AS resource, which restricts the maximum address space a process can use, including the heap.

The two key parameters for setrlimit() are:

• Soft Limit: The current memory usage limit for the process.
• Hard Limit: The maximum memory usage limit that can be set.

The RLIMIT_AS resource limits the total memory (address space) that the process can use, effectively capping the heap size.

Note:

• resource.setrlimit() Sets the limit for RLIMIT_AS (total memory usage).
• If the memory allocation exceeds the set limit (500 MB in this case), Python will raise a MemoryError.
• If the memory usage exceeds the limit, the process cannot allocate more memory, and Python raises a MemoryError.
• The memory limit is set in bytes (e.g., 500 MB = 500 * 1024 * 1024).

Using ulimit Command (Linux/macOS)

For systems using the shell or terminal, you can use ulimit command to limit the memory (including heap size) a process can use.

Step #1: Open the terminal.

Step #2: Run this command to limit memory uses:

ulimit -v 500000 # Limit to 500 MB (in kilobytes)

Step #3: Run your Python script:

python3 my_script.py

If your Python process exceeds the limit, it will raise a MemoryError or terminate.

Note: This is specific to POSIX shells and won’t work on Windows systems.

Limiting Heap Size in Python Programs via Environment Variables

You can also limit memory usage by configuring environment variables before running Python scripts.

For Linux/Mac

export PYTHONMALLOC=debug # Enables memory debugging
ulimit -v 500000 # Limits memory to 500 MB
python3 your_script.py

For Windows users, there is no built-in ulimit, but you can use tools like Process Explorer or Windows Job Objects to restrict memory usage of Python processes.

abhishek1

Follow

Improve

Article Tags :

• Python
• python-utility

Practice Tags :

• python