Scribd
Upload
6 views

Parallel and Distributed lec 8

The document covers key concepts in parallel and distributed computing, including various parallelization algorithms such as loop, pipeline, and speculative parallelism. It discusses synchronization mechanisms like mutexes, semaphores, and locks to prevent race conditions and deadlocks, along with best practices for synchronization. Additionally, it addresses data and work partitioning strategies to optimize task execution across multiple processors.

Uploaded by

reactuser76
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views

Parallel and Distributed lec 8

The document covers key concepts in parallel and distributed computing, including various parallelization algorithms such as loop, pipeline, and speculative parallelism. It discusses synchronization mechanisms like mutexes, semaphores, and locks to prevent race conditions and deadlocks, along with best practices for synchronization. Additionally, it addresses data and work partitioning strategies to optimize task execution across multiple processors.

Uploaded by

reactuser76
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 24

Parallel and Distributed

Computing
COMP3139
Contents
• Parallelism algorithms
• Synchronization
• Synchronization mechanisms
• Challenges
• Best practices
• Data and work partitioning
PARALLELIZATION ALGORITHMS
LOOP PARALLELISM

• Loop parallelism is one of the simplest and most widely used


strategies in parallel computing.
• It involves distributing the iterations of a loop across multiple
processors, where each iteration can be executed independently.
• This strategy works best when there is no data dependency between
iterations, meaning each iteration can run without needing to wait for
the result of others.

3
PARALLELIZATION ALGORITHMS
LOOP PARALLELISM

• Example: Suppose you want to


compute the square of numbers in
an array.
• Each computation is independent,
making it a perfect candidate for
loop parallelism. Using OpenMP,
this can be easily parallelized

4
PARALLELIZATION ALGORITHMS
PIPELINE PARALLELISM

• It is a strategy where a task is broken down into a sequence of


stages, with each stage performing part of the overall task.
• Each stage works concurrently, processing data as soon as it is
available from the previous stage, like an assembly line.
• This approach is useful in scenarios where data flows through
different processing phases, such as video streaming or image
processing.

5
PIPELINE PARALLELISM
• Example code
PARALLELIZATION ALGORITHMS
SPECULATIVE PARALLELISM

• It involves executing tasks that might or might not be necessary.


• The idea is to compute potential outcomes in advance, even when it’s
uncertain which computation path will be chosen.
• This is especially useful in algorithms that involve branching, such as
decision trees, where future computation depends on a choice that
hasn’t been made yet.

7
SPECUL ATIVE PARALLELISM
• EXAMPLE CODE
SYNCHRONIZATION

• Synchronization is a key concept in


parallel computing that ensures correct
operation when multiple threads or
processes are working concurrently and
accessing shared resources.

• The goal is to prevent race conditions,


inconsistencies, or deadlocks that can
occur when parallel threads try to read and
write shared data at the same time.
9
SYNCHRONIZATION MECHANISMS

1. Mutex (Mutual Exclusion): A lock that ensures only one thread at a time
can access a shared resource or critical section of code.

• Mutexes are used when modifying shared variables or data structures to


prevent race conditions.

• Example:

10
SYNCHRONIZATION MECHANISMS

2. Semaphores: A counter-based
synchronization mechanism that
controls access to a pool of resources.
• Useful when multiple threads need to
access limited resources (e.g.,
multiple readers/writers).
• Example: A semaphore with a count
of 3 would allow three threads to
access the resource simultaneously
before blocking others.
11
SYNCHRONIZATION MECHANISMS

3. Locks: General synchronization


primitives that ensure only one thread
can execute a block of code at a time.

• Holding a lock is how one


thread tells other threads: “I'm
changing this thing, don't
touch it right now.”

12
• SYNCHRONIZATION MECHANISMS

Types:
Spinlocks: Threads continuously
check if a lock is available, useful in
situations where waiting times are
very short.
spinlock is a synchronization
mechanism used in operating
systems to protect shared resources
from single access by multiple
13
threads or processes.
• SYNCHRONIZATION MECHANISMS

Types:
• Read-Write Locks: Allows multiple
readers but only one writer at a
time.
• Read-write locks allow simultaneous
read access by many threads while
restricting write access to only one
thread at a time.

14
CHALLENGES ADDRESSED BY
SYNCHRONIZATION

• Race Conditions: Occur when multiple threads try to update


shared data without proper synchronization,

• Leading to unpredictable results.

• Example: Incrementing a shared counter in two threads


without a mutex might lead to lost updates.

15
CHALLENGES ADDRESSED BY
SYNCHRONIZATION

• Deadlocks: Happen when two or


more threads are waiting
indefinitely for resources locked by
each other.
• Example: Thread A locks resource
1 and waits for resource 2, while
Thread B locks resource 2 and
waits for resource 1.

16
BEST PRACTICES FOR
SYNCHRONIZATION

• Minimize the use of locks to avoid performance bottlenecks.

• Ensure locks are acquired and released in the same order by


all threads to prevent deadlocks.

• Use high-level synchronization constructs (e.g., thread-safe


queues) where possible.

17
DATA AND WORK PARTITIONING

• Partitioning is one of the fundamental


steps in parallel computing,
• where tasks or data are divided into
smaller units that can be processed
concurrently by different processors or
cores.
• Data partitioning criteria and the
partitioning strategy decide how the
dataset is divided.
18
DATA AND WORK PARTITIONING

1. Data Partitioning: Dividing the input data into chunks that can be
processed independently.

• In applications like matrix multiplication or image processing, the data


can be partitioned into smaller blocks or sections that are processed in
parallel.

i. Block Partitioning: The data is divided into equal-sized blocks, and


each block is assigned to a processor.

• This is simple but may not balance the load efficiently if some data
chunks are more compute-intensive than others.
19
DATA AND WORK PARTITIONING

ii. Cyclic Partitioning: Tasks are


distributed in a round-robin fashion. This
method is more efficient when data chunks
have varying computational loads.
Example: In a matrix multiplication
problem, partitioning each row of the
matrix across processors would allow each
processor to compute the partial results
concurrently.

20
DATA AND WORK PARTITIONING

2. Work Partitioning:

• Dividing computational tasks into smaller tasks that can


be executed in parallel.

• Work partitioning is typically used in recursive algorithms


(like quicksort) where the problem can be split into
subproblems that can be solved in parallel.

21
DATA AND WORK PARTITIONING

i. Task Decomposition: The program’s functionality is divided into


tasks, where each task performs a separate part of the overall
computation.

ii. Functional Decomposition: Each processor performs a different


function or operation on the same or different data.

• Example: In a parallel quicksort algorithm, after choosing a pivot, the


partitioning of the array into left and right sub-arrays can be
performed by separate threads.

22
CHALLENGES IN PARTITIONING

• Ensuring that tasks or data are independent of each other


to avoid race conditions.

• Balancing the workload across processors to avoid idling


or overloading any single processor.

23
THANK YOU

You might also like