Programming Models for

Parallel Systems
Programming Models for Parallel
Systems:
Programming models for parallel systems are essential for effectively utilizing
modern multi-core processors, distributed systems, and GPUs. They provide abstractions
that help developers write programs capable of exploiting parallel hardware. Below are
key programming models and frameworks used for parallel systems:

1. Shared Memory Models

2. Message Passing Models
3. Data Parallelism Models
4. Task Parallelism Models
5. Actor Models
6. Functional Programming Models
7. Hybrid Models
8. Domain-Specific Frameworks
1. Shared Memory Models:
In shared memory systems, multiple threads or processes access a
common memory space. All threads can directly read and write to shared
variables. To ensure consistency and avoid conflicts (e.g., two threads
trying to modify the same variable), synchronization mechanisms such
as locks, semaphores, and barriers are used.

• Use Case:
Suitable for multi-core processors where threads run
on cores sharing physical memory (e.g., on a single
machine).
3
• Key Features:
• Data is shared between threads.
• Faster communication since no data transfer between memory spaces is
required.
• Requires careful synchronization to avoid race conditions.

• Example Scenario:
Imagine you are summing up an array of numbers using multiple
threads. Each thread can process a portion of the array, and the results are
combined in shared memory. Synchronization ensures that threads don’t
overwrite each other’s results.
2. Message Passing Models:
In this model, processes communicate by explicitly sending and
receiving messages. Each process has its memory space, and data
must be exchanged explicitly through communication mechanisms
like messages. This is widely used in distributed systems, where
processes may run on different machines.

• Use Case:
Ideal for distributed computing environments, such as
clusters or supercomputers.
5
• Key Features:
• Processes do not share memory; instead, they exchange data through
messages.
• Requires explicit programming to send and receive data.
• Scales well to large systems since there's no shared memory contention.

• Example Scenario:
Consider weather simulation across multiple nodes in a cluster. Each
node models a specific region. At the boundaries, nodes must exchange data
(e.g., temperature, wind speed) with their neighbors using messages to ensure
continuity.
3. Data Parallelism Models:
In data parallelism, the same operation is performed simultaneously on
different parts of a dataset. This model is particularly effective when
working with large datasets, such as in scientific simulations or image
processing.

• Use Case:
Suitable for applications like matrix multiplication, image
processing, and machine learning where the same computation is
applied to large amounts of data.
7
• Key Features:
• Divides data into chunks that can be processed in parallel.
• Relies on hardware like GPUs, which are designed for data-parallel
operations.
• Often involves SIMD (Single Instruction, Multiple Data) or MIMD
(Multiple Instruction, Multiple Data) architectures.

• Example Scenario:
Imagine processing a large image to adjust its brightness. The image
can be divided into smaller tiles, and each tile can be processed
independently on a GPU, adjusting the brightness in parallel.
4. Task Parallelism Models:
Task parallelism divides a program into independent tasks that can run
concurrently. Each task performs a different operation, and the runtime
system schedules them for execution.

• Use Case:
Useful when different tasks can be performed simultaneously,
such as in pipelined workloads or scenarios where tasks can run
independently.

9
•Key Features:
• Emphasizes dividing the work based on tasks, not data.
• Tasks may or may not require communication with each other.
• Often uses a task scheduler to manage task execution.

• Example Scenario:
In a video processing pipeline, one task could read video frames,
another task could compress them, and a third task could write the compressed
frames to disk. Each task runs in parallel and processes different parts of the
video.
5. Actor Models:
The actor model conceptualizes computation as a group of "actors."
Each actor is an independent entity that processes messages and can
send messages to other actors. This model abstracts away details like
threads or locks, making it simpler to manage concurrent behavior.

• Use Case:
Effective for systems requiring high concurrency, such as
real-time systems or chat applications.

11
• Key Features:
• Each actor has its state and processes one message at a time.
• Communication is asynchronous, promoting decoupled design.
• Eliminates the need for explicit locking mechanisms.

• Example Scenario:
In a chat application, each user can be represented as an actor. When a
message is sent, it is delivered to the intended recipient actor, which processes
the message independently.
6. Functional Programming Models:
Functional programming relies on immutable data and pure functions
(functions without side effects). In parallelism, it simplifies reasoning
about concurrent computations because immutable data eliminates
race conditions.

• Use Case:
Widely used in big data processing (e.g., map-reduce
paradigms) and reactive programming.

13
• Key Features:
• Functions operate on immutable data, ensuring thread safety.
• Often uses high-level abstractions like map, filter, and reduce for
parallel computation.
• Promotes a declarative style of programming.

• Example Scenario:
Processing a log file to count the frequency of words. The file can be
split into chunks, and each chunk can be processed independently to compute
word frequencies. The results are then merged.
7. Hybrid Models:
Hybrid models combine different parallel programming paradigms to
leverage the advantages of each. A common combination is shared
memory within a node and message passing across nodes in a
distributed system.

• Use Case:
Ideal for large-scale systems like supercomputers or hybrid
clusters where nodes have multiple cores.

15
• Key Features:
• Mixes paradigms like MPI (message passing) with OpenMP (shared
memory).
• Efficiently utilizes both inter-node and intra-node parallelism.
• Requires managing both types of communication.

• Example Scenario:
In a weather simulation, each compute node simulates a region (using
MPI to communicate with other nodes). Within each node, multiple threads
(using OpenMP) simulate different weather layers like wind, temperature, and
pressure.
8. Domain-Specific Frameworks:
These frameworks are designed to simplify parallelism for specific
types of applications. They abstract away low-level details, letting
developers focus on domain-specific logic.

• Use Case:
Used in areas like machine learning (TensorFlow, PyTorch),
big data (Spark), or scientific computing (Dask).

17
• Key Features:
• Built-in support for parallelism tailored to specific domains.
• High-level abstractions reduce development complexity.
• Optimized for performance in their respective domains.

• Example Scenario:
In machine learning, training a deep neural network involves
processing large batches of data. Frameworks like TensorFlow split the data
across multiple GPUs, performing computations in parallel to accelerate
training.
 SUMMARY TABLE

Model Memory Sharing Synchronization Best Used For

Shared Memory Shared Memory Shared Memory Shared Memory

Message Passing Message Passing Message Passing Message Passing

Implicit
Shared or Large data sets,
Data Parallelism (hardware/softwar
distributed GPU computing
e)
Independent or
Shared or Scheduler
Task Parallelism pipeline-style
distributed manages tasks
tasks
High-
Asynchronous
Actor Model Independent actors concurrency, real-
message passing
time applications
Big data
Implicit (pure
Functional Programming Immutable data processing,
functions)
reactive systems19
Any Question?

20
Thanks

21