Parallel & Distributed

Computing
By
Mrs.SD.Shareefunnisa
Course Description & Objectives
• This course addresses the challenges involved in developing parallel
and distributed applications. It explores various implementation
strategies, paradigms, architectures, and parallel algorithms.
Additionally, it highlights current trends in parallel and distributed
computing, such as Open MP, POSIX Threads, and Apache Hadoop
(DFS).
Course Outcomes
• Apply the fundamentals of parallel and parallel computing including
architectures and paradigms.
• Analyse the various design principles of parallel algorithms.
• Learn the intricacies of distributed programming.
• Develop and execute basic parallel and distributed applications using
basic programming models and tools.
 MODULE-1
UNIT-1 12L+8T+0P=20 Hours

INTRODUCTION TO PARALLEL COMPUTING

The idea of Parallelism, Power and potential of parallelism, examining sequential and
parallel programs, Scope and issues of parallel and distributed computing, Goals of
parallelism, Parallelism and concurrency using multiple instructions streams. Parallel
Architecture: Pipeline architecture, Array processor, Multi-processor architecture, Systolic
architecture, Dataflow architecture, Architectural classification schemes, Memory access
classifi- cation, Memory Issues: Shared vs. distributed, Symmetric multiprocessing (SMP),
SIMD, Vector processing, GPU co-processing, Flynn’s Taxonomy, Instruction Level
support for parallel programming, Multiprocessor caches and Cache Coherence, Non-
Uniform Memory Access (NUMA).
UNIT-2 12L+8T+0P=20 Hours

PARALLEL ALGORITHM DESIGN PRINCIPLES AND

PROGRAMMING
Need for communication and coordination/synchronization, Scheduling and
contention, Independence and partitioning, Task- Based Decomposition, Data
Parallel Decomposition, Characteristics of task and interaction, Load balancing,
Data Management, parallel algorithm models, Sources of overhead in parallel
programs, Performance metrics for parallel algorithm implementations, Parallel
algorithmic patterns like divide and conquer, Map and Reduce, Specific
algorithms like parallel Merge Sort, Parallel graph Algorithms.
Parallelism
• Parallelism refers to the process of performing multiple tasks or computations
simultaneously by dividing them into smaller, independent parts that can be
executed concurrently.
• This approach is commonly used to increase computational speed and efficiency,
particularly in systems with multiple processors, cores, or threads.
Types of Parallelism

• Data Parallelism: Dividing data across processors to apply the same operation
concurrently
Ex: Matrix multiplication
• Task Parallelism: Assigning different tasks or processes to different processors
Ex: executing different functions simultaneously.
• Bit-Level Parallelism: Performing operations on smaller data sizes simultaneously.
Ex: Add two 8-bit numbers using a processor
• Instruction-Level Parallelism: Executing multiple instructions simultaneously
within a single CPU.
Ex: Loop Unrolling
 Parallel computing
• It is a type of computation in which multiple tasks or calculations are carried out

simultaneously to solve a problem more efficiently. It involves breaking a large problem

into smaller subproblems that can be solved concurrently, leveraging multiple processors,
cores, or computers working together.

Key Characteristics:

• Concurrency: Tasks are executed simultaneously.

• Resource Utilization: Utilizes multiple CPUs, GPUs, or distributed systems.

• Speedup: Aims to reduce execution time by dividing workloads.

Components of Parallel Computing:

• Parallel Hardware: Multicore CPUs, GPUs, clusters, or distributed systems.

• Parallel Software: Programming models like MPI, OpenMP, CUDA, or multithreading libraries.

• Communication: Coordinating tasks and sharing data efficiently between processors.

Comparison Of Sequential Processing And Parallel Processing:
 Advantages of parallel computing
• Increased Speed: Tasks are divided and executed simultaneously, significantly reducing computation time.

• Efficient Resource Utilization: Makes use of multiple processors, cores, or machines to achieve higher efficiency

and throughput.

• Scalability: Can handle larger datasets or more complex problems by adding more processing units.

•Cost Efficiency: Distributed systems can use many low-cost machines instead of relying on a single, expensive

high-performance system.

•Real-Time Processing: Ideal for time-sensitive applications like autonomous vehicles, medical diagnostics, or

financial trading.
•Energy Efficiency: Performing computations in parallel can reduce energy consumption compared to
sequential processing over an extended period.
•Fault Tolerance: Some parallel systems can continue processing even if one processor or node fails
(redundancy in distributed systems).
•Solving Larger Problems: Enables tackling problems that are too large or complex for a single
processor to handle.
•Improved Performance in AI/ML: Speeds up training of machine learning models and processing of
large datasets in applications like natural language processing or image recognition.
•Diverse Application Areas: Widely used in fields like scientific simulations, big data analytics,
weather forecasting, video rendering, and cryptography.
 Parallel Processing Examples

1. Weather Forecasting: Simultaneous analysis of atmospheric conditions over different regions.

2. Video Rendering: Encoding or rendering video frames using GPU cores simultaneously.
3. Machine Learning Training: Training a neural network by processing batches of data in parallel on
multiple GPUs.
4. Search Engines: Indexing and searching web pages across multiple servers.
5. Financial Simulations: Running Monte Carlo simulations in parallel for faster decision-making.
6. Big Data Analytics: Processing large datasets using frameworks like Apache Hadoop or Spark.
7. Medical Imaging: Performing simultaneous computations for 3D reconstruction of CT or MRI scans.
8. Real-Time Gaming: Computing physics, rendering, and AI logic in parallel on different cores or GPUs.
9. Cryptography: Breaking down encryption or hashing algorithms into parallel tasks for faster
computation.
10.Astronomical Simulations: Studying star formations or galactic collisions by processing huge datasets
concurrently.
Sequential
Processing

Parallel
Processing
 Why Use Parallel Computing?
• Save time and money:
In theory, throwing more resources at a task will shorten its time to completion, with potential cost
savings. Parallel clusters can be built from cheap, commodity components.
• Solve larger problems:
Many problems are so large and complex that it is impractical or impossible to solve them on a single
computer, especially given limited computer memory.
• Provide concurrency:
A single compute resource can only do one thing at a time. Multiple computing resources can be doing
many things simultaneously.
• Use of non-local resources:
Using computer resources on a wide area network, or even the Internet, when local computer resources
are scarce.
• Parallel computing allows one to solve problems that don't fit on a single CPU and solve problems that
can't be solved in a reasonable time.
Other Reasons Might Include:
1.Taking advantage of non-local resources:
Using available compute resources on a wide area network or even the Internet when local compute
resources are scarce.
2.Cost savings:
Using multiple "cheap" computing resources instead of paying for time on a supercomputer.
3.Overcoming memory constraints:
Single computers have very finite memory resources. For large problems, using the memories of
multiple computers may overcome this obstacle.
 1.3 Motivating Parallelism
Importance of Parallelism:
•Parallelism has been essential for speeding up computing by handling multiple tasks simultaneously.
•It helps manage large amounts of data and provides faster access to storage in commercial applications.
Scalability and Cost Efficiency:
•Parallel computing systems offer better performance and are cost-effective.
•However, developing parallel hardware and software traditionally requires time and effort.
Challenges with Uniprocessors:
•While uniprocessors are improving, they may not keep up with the growing demand for faster and more
powerful computing.
•This highlights the need for parallel computing to overcome the limitations of single-processor systems.
•Advancements in Parallel Computing:
•Standardized programming tools, libraries, and hardware have made parallel computing more accessible
and efficient.

1. Computational Power Argument: From Transistors to FLOPS

• In 1965, Gordon Moore said:

"The complexity for minimum component costs has increased at a rate of roughly a factor of two per
year. Certainly over the short term this rate can be expected to continue if not to increase. Over the long
term..."

• From 1975, he revised the doubling period to 18 months, which came to be known as Moore's Law.
With more devices in a chip, the pressing issue is achieving increasing operations per second. A logical
solution is to rely on parallelism, both implicit and explicit.
Most serial processors rely extensively on implicit parallelism.
• Moore's Law:
• In 1965, Gordon Moore observed that the number of transistors on a chip doubles approximately every
two years, leading to faster and cheaper computing.
• In 1975, this doubling rate was adjusted to every 18 months, a trend known as Moore's Law.
• Challenge:
• As chips get more transistors, the key challenge is achieving more operations per second.
• Solution:
• Parallelism is the answer:
• Implicit Parallelism: Automatically managed by the processor.
• Explicit Parallelism: Designed directly into software and algorithms.
• Most processors today rely heavily on implicit parallelism for better performance.
2. The Memory / Disk Speed Argument
•Problem: Processors are getting faster (40% improvement per year), but memory (DRAM) is not keeping up (only 10%
improvement per year). This causes delays in performance.
•Solution: Parallel computing helps by:
• Increasing memory bandwidth (faster data flow between memory and processor).
• Using larger caches (temporary storage for quicker access).
• Optimizing how data is accessed (locality of reference) to reduce delays.
•Key Use: Parallel computing excels at moving data quickly between memory and storage, which is critical for modern
applications.
3. The Data Communication Argument
•Problem: Large-scale applications, like databases and data mining, generate enormous data volumes that cannot be
easily moved for analysis.
•Solution: Parallel computing enables analysis directly over the network, reducing the need to transfer data.
•Example: SETI@home, which uses the Internet as a global computing platform to search for extraterrestrial intelligence.
Difference between Parallel Computing and Distributed Computing:
 Parallel and Distributed Computing Applications
•Scientific Simulations: Weather forecasting, climate modeling, and astrophysics
simulations leverage high-performance computing for faster results.
•Big Data Processing: Platforms like Apache Spark and Hadoop enable distributed
data processing for large-scale analytics.
•Machine Learning and AI: Training large neural networks and deep learning
models using GPU clusters and distributed frameworks (e.g., TensorFlow, PyTorch).
•Genomic Research: Genome sequencing and analysis rely on distributed systems
for processing massive biological datasets.
• Financial Modeling: Risk analysis, stock market predictions, and algorithmic trading utilize
parallel algorithms for real-time computations.
• Image and Video Processing: Parallel techniques process high-resolution images and videos
for tasks like object detection and rendering.
• Cryptography and Blockchain: Distributed systems provide the backbone for secure data
transmission and decentralized ledgers.
• Search Engines: Companies like Google use parallel and distributed computing for indexing
and retrieving vast amounts of web data.
• Healthcare Systems: Distributed computing aids medical imaging, telemedicine, and drug
discovery simulations.
• Gaming and Virtual Reality: Multiplayer gaming and VR use distributed computing to
synchronize data across networks in real time.