Scribd
Upload
2 views

07 Parallel Algorithms in Parallel and Distributed Computing

Parallel algorithms enhance computational efficiency by dividing tasks into subtasks executed simultaneously, applicable in various fields such as AI and big data. They can be categorized into types like data parallel, task parallel, and hybrid algorithms, and are designed based on models such as PRAM and message passing. Key considerations in their design include performance metrics like speedup and scalability, as well as challenges like synchronization overhead and load imbalance.

Uploaded by

amnashah001122
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
2 views

07 Parallel Algorithms in Parallel and Distributed Computing

Parallel algorithms enhance computational efficiency by dividing tasks into subtasks executed simultaneously, applicable in various fields such as AI and big data. They can be categorized into types like data parallel, task parallel, and hybrid algorithms, and are designed based on models such as PRAM and message passing. Key considerations in their design include performance metrics like speedup and scalability, as well as challenges like synchronization overhead and load imbalance.

Uploaded by

amnashah001122
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 13

Parallel Algorithms in Parallel and

Distributed Computing
Concepts, Models, Strategies, and
Performance Metrics
Introduction to Parallel Algorithms
• Parallel algorithms divide a task into subtasks
executed simultaneously.
• They improve computational speed and
efficiency.
• Used in scientific computing, AI, big data, and
simulations.
• Designed for multi-core processors, clusters,
GPUs, and distributed systems.
Types of Parallel Algorithms
• 1. Data Parallel Algorithms - Operate on large
datasets in parallel.
• 2. Task Parallel Algorithms - Different tasks run
concurrently.
• 3. Divide and Conquer Algorithms - Break problems
into smaller subproblems solved in parallel.
• 4. Pipelined Algorithms - Break computation into
stages processed in sequence.
• 5. Hybrid Algorithms - Combine multiple parallel
techniques for optimization.
Models of Parallel Computation
• 1. PRAM (Parallel Random Access Machine) -
Theoretical model for parallel processing.
• 2. Message Passing Model - Processes communicate
via message passing (MPI, RPC).
• 3. Shared Memory Model - Multiple
threads/processes access a common memory.
• 4. Dataflow Model - Execution depends on
availability of input data.
• 5. Hybrid Model - Mix of shared memory and
message passing approaches.
Parallel Algorithm Design Strategies
• 1. Partitioning - Divide problem into independent
parts.
• 2. Communication - Ensure efficient data sharing
between processors.
• 3. Synchronization - Manage dependencies and
execution order.
• 4. Load Balancing - Distribute workload evenly
across processors.
• 5. Scalability Optimization - Ensure performance
remains efficient as processors increase.
Performance Metrics of Parallel Algorithms

• 1. Speedup = The formula for Speedup (S) is:


S = T(serial) / T(parallel)
Where T(serial) is the execution time of a task using a single processor and
T(parallel) is the execution time of the same task using multiple processors.

• 2. Efficiency (E)=
• 3. Scalability = Ability to maintain efficiency as
problem size/processors increase.
• 4. Communication Overhead = Extra time
spent in data exchange.
• 5. Amdahl’s Law & Gustafson’s Law
Graph show expectation and reality
Example: Parallel Sorting Algorithms
• 1. Parallel Merge Sort = Divides array into
subarrays, sorts them in parallel, and merges
results.
• 2. Parallel Quick Sort = Uses multiple threads to
sort subarrays independently.
• 3. Bitonic Sort = Efficient parallel sorting
algorithm used in GPU programming.
• 4. Odd-Even Transposition Sort = Parallel
sorting for distributed systems.
Example: Parallel Matrix Multiplication
• Row-wise Parallelism: Each processor
computes different rows.
• Column-wise Parallelism: Each processor
computes different columns.
• Block-based Decomposition: Matrix divided
into smaller blocks for efficient parallel
execution.
• GPU Acceleration: Uses CUDA or OpenCL for
high-speed execution.
Challenges in Parallel Algorithm Design
• Synchronization Overhead = Extra time spent
coordinating tasks.
• Load Imbalance = Uneven distribution of tasks
leading to inefficiency.
• Data Dependency Issues = Some computations
depend on previous results.
• Memory Bottlenecks = Limited memory bandwidth
can slow performance.
• Scalability Limitations = Not all problems scale well
with more processors.
Applications of Parallel Algorithms
• Scientific Simulations like Weather forecasting,
climate modeling.
• Big Data Analytics like Hadoop, Spark for large-scale
data processing.
• Artificial Intelligence like Training deep learning
models in parallel.
• Cryptography & Security like Parallel cryptographic
algorithms for fast encryption.
• Bioinformatics like DNA sequencing and protein
structure analysis.
Conclusion
• Parallel algorithms improve performance by
dividing tasks among multiple processors.
• Different computation models like PRAM,
Message Passing, and Shared Memory influence
design.
• Efficient algorithm design must balance speedup,
communication overhead, and scalability.
• Applications range from AI and Big Data to
scientific computing and security.

You might also like