Read File Using Limited Buffer Size in Python

In Python, when we are working with files that contain a large amount of data then it is often inefficient to read the entire content into memory at once.

In such cases, we can read a fixed amount of data at a time by using the read() method with a specified size. This process is commonly known as buffered or chunk-based reading.

The Chunk-based reading is a program to read a file piece-by-piece rather than loading it completely. This process is particularly helpful when handling large files, as it helps to reduce memory consumption and enhances performance during file processing operations.

Following is the syntax of reading a file with the chunk-based reading or Buffer Reading process in Python -

file.read(size)

Reading a File in Chunks

We can read a file using a limited buffer size. For that, we have to open the file and repeatedly call the read() method inside a loop until all the given data is processed. Following is an example, in which we read a text file using a buffer of 16 characters -

with open(r"D:\Tutorialspoint\Articles\file2.txt", "r") as file:
   while True:
      chunk = file.read(16)  # Read 16 characters at a time
      if not chunk:
         break
      print(chunk)

Following is the output of the above example -

Hello welcome to
 Tutorialspoint.

have a happy le
arning

Reading a Binary File in Fixed Chunks

When working with binary files, we typically read data in fixed-size byte blocks. This is particularly helpful for large multimedia or binary data files. Following is an example of reading a binary file in a fixed buffer size -

with open(r"D:\Tutorials\Examples\image.png", "rb") as binary_file:
   while True:
      chunk = binary_file.read(1024)  # 1 KB
         if not chunk:
            break