ProcessPoolExecutor Class in Python

The ProcessPoolExecutor class in python is part of the concurrent.futures module, is a high level interface for asynchronously executing functions using the processes and threads. The ProcessPoolExecutor allows us to execute the multiple functions in parallel using the multiple processes, which is particularly useful to CPU bound tasks that benefit in the parallelization process.

Multiprocessing and Multithreading

Before going to see about the ProcessPoolExecutor class we have to know about the Multiprocessing and Multithreading. Multiprocessing and Multithreading are the techniques used to achieve the parallelism process and they are different in the way they manage and create concurrent tasks.

Multiprocessing uses multiple processes to execute tasks in parallel and each process has its own memory space to avoid issues with shared memory and concurrency. It is also a means of communication between processes that can be more complex and expensive, as data must be serialized and deserialized between processes. Multiprocessing is particularly used in CPUbound tasks which benefit in performing the parallelization tasks like numerical computations or image processing etc. In python we have a module multiprocessing which allows us to create and manage the processes and the pool class is used to manage the pool of worker processes to execute the functions in parallel.

Multithreading uses multiple threads within a single process to achieve parallelism and each thread shares the same memory space as the main thread, which can simplify communication between threads. This means we need to be careful while accessing shared data to avoid issues with concurrency like race conditions or deadlocks. Multithreading used to perform I/Obound tasks which benefit in parallelization like network programming or file processing etc. In python we have the module threading to create and manage the threads and the thread class used to create the new thread and execute the function in that thread. The lock class is used to synchronize the access of the shared data between the threads.

Creating the ProcessPoolExecutor

The process of creating a ProcessPoolExecutor is similar to the ThreadPoolExecutor only difference is we have to import the class from the concurrent.futures module. We will use the OS module to get the current task PID which we execute within our pool.

Example

from concurrent.futures import ProcessPoolExecutor
import os
def task():
   print("Current executing task PID {}".format(os.getpid()))
if __name__ == '__main__':
   result =[]
   with ProcessPoolExecutor(max_workers=3) as exe:
      result1 = exe.submit(task())

Output

Current executing task PID 6652

Example

from concurrent.futures import ProcessPoolExecutor
import os
values = [10,40,30,4]
def square(x):
   print(f'square of {x}:{x*x}')
   print("Current executing task PID {}".format(os.getpid()))
if __name__ == '__main__':
   result =[]
   with ProcessPoolExecutor(max_workers = 5) as exe:
      for i in values:
         result = exe.submit(square(i))

Output

square of 10:100
Current executing task PID 19276
square of 40:1600
Current executing task PID 19276
square of 30:900
Current executing task PID 19276
square of 4:16
Current executing task PID 19276