0% found this document useful (0 votes)
88 views

Cheat Sheet For Python Multiprocessing

This document provides an overview of key concepts for multiprocessing in Python including processes, locks, events, conditions, queues, pipes and managers. It covers how to create and configure processes, share data between processes, synchronize processes, and communicate between processes.

Uploaded by

hotfoodtaaste
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
88 views

Cheat Sheet For Python Multiprocessing

This document provides an overview of key concepts for multiprocessing in Python including processes, locks, events, conditions, queues, pipes and managers. It covers how to create and configure processes, share data between processes, synchronize processes, and communicate between processes.

Uploaded by

hotfoodtaaste
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

SuperFastPython.

com Cheat Sheet for Python multiprocessing


Why multiprocessing? Locks and Events Condition Variables and Barriers
Use processes for tasks that are CPU-bound, such Locks protect critical section, events are safe flags. Conditions for wait/notify, barriers for syncing.
as compute tasks and math operations.
Mutex lock Condition variable
Create, Config, Use Process Objects lock = Lock() condition = Condition()
lock.acquire() condition.acquire()
Import # ... # ...
from multiprocessing import * lock.release() condition.release()

Create, run target function Mutex lock, context manager Wait on condition to be notified (blocking)
process = Process(target=task) lock = Lock() with condition:
with lock: condition.wait()
Config process name # ...
process = Process(name=’MyProcess’) Wait on condition for expression (blocking)
Reentrant mutex lock, protect critical section with condition:
Config daemon process (background process) lock = RLock() condition.wait_for(check)
process = Process(daemon=True) with lock:
with lock: Notify any single thread waiting on condition
Extend process # ... with condition:
class CustomProcess(Process): condition.notify(n=1)
def run(): Semaphore, set num positions
# ... semaphore = Semaphore(10) Notify all threads waiting on condition
semaphore.acquire() with condition:
Start process (non-blocking) # ... condition.notify_all()
process.start() semaphore.release()
Barrier, set number of parties
Join process, wait to finish (blocking) Semaphore, context manager barrier = Barrier(5)
process.join() semaphore = Semaphore(10)
with semaphore: Arrive and wait at barrier (blocking)
Join process with timeout # ... barrier.wait()
process.join(timeout=5)
Create event, then set event Arrive and wait at barrier with timeout
Check if process is running (not finished) event = Event() barrier.wait(timeout=0.5)
if process.is_alive(): event.set()
# ...
Check if event is set
Terminate process (SIGTERM) if event.is_set():
process.terminate() # ...

Kill process (SIGKILL) Wait for event to be set (blocking)


process.kill() event.wait()

Access process native PID Wait for event with timeout


process.pid if event.wait(timeout=0.5):
# ...
Module Functions Shared ctypes Queues
Utilities for working with processes. Share data between processes safely. Via Queue, SimpleQueue, JoinableQueue

List of all active child processes Create a shared integer value Create queue
children = active_children() var = Value(‘i’, 100) queue = Queue()

Number of logical CPU cores in system Create a shared floating point value Create queue with limited capacity
num = cpu_count() var = Value(‘f’, 2.2) queue = Queue(100)

Process instance for current process Access a shared value Add item to queue (blocking, if limited)
process = current_process() data = var.value queue.put(item)

Process instance for parent process Change a shared value Add item, with timeout (blocking if limited)
process = parent_process() var.value = 200 try:
queue.put(timeout=0.5)
Add multiprocessing support, if frozen (win32) Create a shared integer array except queue.Full as e:
freeze_support() array = Array(‘i’, (1, 2, 3, 4, 5)) # ...

Create context with start method Access all values in an array Retrieve item from queue (blocking)
ctx = get_context(‘spawn’) for item in array: item = queue.get()
# ...
Set the default method for child processes Retrieve item from queue, with timeout
set_start_method(‘spawn’) Change one value in an array try:
array[0] = 22 item = queue.get(timeout=0.5)
Get a list of all supported start methods except queue.Empty as e:
methods = get_all_start_methods() Pipes # ...

Get the current default start method Create unidirectional pipe Check if queue is empty
method = get_start_method() receiver, sender = Pipe() if queue.empty():
# ...
Protect the entry point Create bidirectional pipe (duplex)
if __name__ == '__main__': conn1, conn2 = Pipe(duplex=True) Check if queue is full
# ... if queue.full():
Send object via pipe # ...
Managers sender.send(‘Hello there’)
Get current capacity of queue
Create and use manager Receive object via pipe (blocking) capacity = queue.qsize()
manager = Manager() data = receiver.recv()
manager.start()
# ... Block until there is data to receive from pipe
manager.shutdown() receiver.poll()

Manager, via context manager, shared lock Block until data to receive, with timeout
with Manager() as manager: if receiver.poll(timeout=0.5):
lock = manager.Lock() # ...

You might also like