Writing Memory Efficient Programs Using Generators in Python
Last Updated :
06 May, 2024
When writing code in Python, wise use of memory is important, especially when dealing with large amounts of data. One way to do this is to use Python generators. Generators are like special functions that help save memory by processing data one at a time, rather than all at once.
The logic behind memory-efficient functions and Python generators is to create functions that generate values on the fly, avoiding the need to store the entire data set in memory. This is especially useful when dealing with large data sets.
Memory Efficiency of Generators
Generators in Python are a powerful tool for creating iterators. Let's deep dive into how generators achieve this efficiency and provide a comparison with traditional loops.
- On-the-Fly Sequence Generation: Python generators efficiently create iterators by generating values only when requested, avoiding the need to store the entire sequence in memory simultaneously.
- Memory Efficiency: Unlike traditional data structures, generators use lazy evaluation, producing elements one at a time as needed. This minimizes memory usage, making them ideal for large datasets or infinite sequences.
- Constant Memory Footprint: Generators maintain a consistent memory footprint regardless of sequence size since they don't store the entire sequence in memory. This feature is beneficial for processing large datasets without encountering memory constraints, offering efficient resource utilization.
Code Memory Efficient Functions with Python Generators
Below, are the example of How to Code Memory Efficient Functions with Python Generators.
Basic Generator Function
Start by creating a function with the yield keyword. This turns it into a generator function. Inside, use a loop to generate values one by one. The yield statement provides the current value. This way, the generator produces values on demand, saving memory.
In the below code example, the memory_efficient_function creates numbers from 0 up to the given max_value. The key is that it doesn't keep all the numbers in memory at once. It produces them one by one, which is helpful when you are working with a large set of data and want to save memory.
Python
def memory_efficient_function(max_value):
current_value = 0
while current_value < max_value:
yield current_value
current_value += 1
# Using the generator function
my_generator = memory_efficient_function(5)
for value in my_generator:
print(value)
Real-Life Example with Log File
Consider a scenario where you need to analyze a large log file without loading it all into memory. Create a generator function, like process_log_file, to read the log file line by line. This way, you process the file gradually without storing the whole thing in memory. In this case, the process_log_file function reads the log file line by line and yields each line as it processes it. This way, we will not be loading the entire log file into memory at once. So this way we can make our code more memory-efficient.
Python
def process_log_file(log_file_path):
with open(log_file_path, 'r') as file:
for line in file:
# Process each line of the log file here
yield line
# Use generator to process the log file
log_file_path = '/Path/To/The/file.txt'
log_generator = process_log_file(log_file_path)
for log_entry in log_generator:
# perform actions on each log entry
print(log_entry)
Output
GeeksforGeeks
above code display the output which is written in your file.txt file.
Filtering Data with Generators
Imagine you have a list of numbers, and you only want to work with the even ones. Instead of creating a new list in memory, you can use a generator to produce only the even numbers when needed. This generator function takes a list of numbers as input and produces only the even numbers one at a time. By doing this, you avoid storing a new list of even numbers in memory, making your code more memory-efficient.
Python
def even_number_generator(numbers):
for num in numbers:
if num % 2 == 0:
yield num
# Example usage
numbers_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Create a generator object
even_gen = even_number_generator(numbers_list)
# Iterate through the generator to get even numbers
for even_num in even_gen:
print(even_num)
Generator vs. For Loop
Let's see the memory efficiency of generators by comparing them with traditional For loop using a simple example below:
In this example, we will generate a sequence of numbers from 0 to 999,999 using both a generator and a for loop. We will then compare the memory usage of both approaches using the sys.getsizeof() function.
Python
# Generator function
def generate_numbers(n):
for i in range(n):
yield i
# For Loop Example
def generate_numbers_list(n):
numbers = []
for i in range(n):
numbers.append(i)
return numbers
# comparing memory usage
import sys
n = 1000000 # generating 1 million numbers
# memory usage for Generator
generator_memory = sys.getsizeof(generate_numbers(n))
# memory usage for For Loop
for_loop_memory = sys.getsizeof(generate_numbers_list(n))
print("memory usage for Generator:", generator_memory, "bytes")
print("memory usage for For Loop:", for_loop_memory, "bytes")
Output :
memory usage for Generator: 104 bytes
memory usage for For Loop: 8448728 bytes
Results:
- Memory Usage for Generator: Less memory consumption due to lazy evaluation.
- Memory Usage for For Loop: Higher memory consumption as it stores the entire sequence in memory.
Conclusion
In Conclusion, Python generators are powerful tools for creating memory-efficient functions. They let you handle large amounts of data without loading everything into memory at once. This is crucial for achieving optimal performance and efficiently processing substantial datasets. By using different methods like basic generator functions, real-life log file examples, understanding space complexity, and exploring advanced techniques, you can enhance your code's memory efficiency.
Similar Reads
Python Tutorial | Learn Python Programming Language Python Tutorial â Python is one of the most popular programming languages. Itâs simple to use, packed with features and supported by a wide range of libraries and frameworks. Its clean syntax makes it beginner-friendly.Python is:A high-level language, used in web development, data science, automatio
10 min read
Python Interview Questions and Answers Python is the most used language in top companies such as Intel, IBM, NASA, Pixar, Netflix, Facebook, JP Morgan Chase, Spotify and many more because of its simplicity and powerful libraries. To crack their Online Assessment and Interview Rounds as a Python developer, we need to master important Pyth
15+ min read
Non-linear Components In electrical circuits, Non-linear Components are electronic devices that need an external power source to operate actively. Non-Linear Components are those that are changed with respect to the voltage and current. Elements that do not follow ohm's law are called Non-linear Components. Non-linear Co
11 min read
Python OOPs Concepts Object Oriented Programming is a fundamental concept in Python, empowering developers to build modular, maintainable, and scalable applications. By understanding the core OOP principles (classes, objects, inheritance, encapsulation, polymorphism, and abstraction), programmers can leverage the full p
11 min read
Python Projects - Beginner to Advanced Python is one of the most popular programming languages due to its simplicity, versatility, and supportive community. Whether youâre a beginner eager to learn the basics or an experienced programmer looking to challenge your skills, there are countless Python projects to help you grow.Hereâs a list
10 min read
Python Exercise with Practice Questions and Solutions Python Exercise for Beginner: Practice makes perfect in everything, and this is especially true when learning Python. If you're a beginner, regularly practicing Python exercises will build your confidence and sharpen your skills. To help you improve, try these Python exercises with solutions to test
9 min read
Python Programs Practice with Python program examples is always a good choice to scale up your logical understanding and programming skills and this article will provide you with the best sets of Python code examples.The below Python section contains a wide collection of Python programming examples. These Python co
11 min read
Spring Boot Tutorial Spring Boot is a Java framework that makes it easier to create and run Java applications. It simplifies the configuration and setup process, allowing developers to focus more on writing code for their applications. This Spring Boot Tutorial is a comprehensive guide that covers both basic and advance
10 min read
Class Diagram | Unified Modeling Language (UML) A UML class diagram is a visual tool that represents the structure of a system by showing its classes, attributes, methods, and the relationships between them. It helps everyone involved in a projectâlike developers and designersâunderstand how the system is organized and how its components interact
12 min read
Enumerate() in Python enumerate() function adds a counter to each item in a list or other iterable. It turns the iterable into something we can loop through, where each item comes with its number (starting from 0 by default). We can also turn it into a list of (number, item) pairs using list().Let's look at a simple exam
3 min read