Assignment 2
Assignment 2
Signals
National University
of Sciences &
ASSIGNEMNT 2
Course: BEIS-2
Table of Contents
1. INTRODUCTION..................................................................................................................................... 3
5. CONCLUSION......................................................................................................................................... 6
Page 2 of 6
Parallel Distributed Computing
The shared memory model allows multiple processors within a single computing node to access a
common memory space. Since all processors share the same memory, communication between
them occurs through shared variables rather than explicit data transfer. This reduces communication
overhead and enables faster access to data. Shared memory architectures are commonly found in
multi-core processors and Graphics Processing Units (GPUs), where multiple threads operate on the
same dataset.
Figure 1.1
A popular programming model for shared memory parallelism is OpenMP (Open Multi-Processing),
which enables developers to efficiently distribute workloads among multiple threads within a single
computing node. While this model is highly efficient for intra-node parallelism, it does not scale well
across multiple nodes in a distributed system.
Page 3 of 6
Parallel Distributed Computing
In contrast, the distributed memory model assigns each processing node its own private memory,
meaning that data must be explicitly exchanged between nodes using message-passing techniques.
This model is well-suited for large-scale HPC clusters and supercomputers, where computations are
distributed across multiple machines. However, communication between nodes introduces
additional complexity, including network latency and synchronization challenges.
Figure 1.2
The Message Passing Interface (MPI) is a widely used standard for implementing distributed
memory parallelism, allowing processes to communicate efficiently across a network of nodes.
Although distributed memory architectures provide excellent scalability, the need for frequent data
exchanges can impact performance, especially for applications requiring extensive inter-node
communication.
Maximizes the use of multi-core processors within a node while leveraging multiple
nodes for large-scale computations.
Shared memory within a node enables faster data access compared to sending
messages between nodes.
Reduces the need for frequent inter-node communication, which is a major
performance bottleneck in distributed computing.
Page 4 of 6
Parallel Distributed Computing
o Each node processes a portion of the problem and communicates with other nodes
as needed.
o Each MPI process runs multiple OpenMP threads, allowing computations within a
node to be parallelized using shared memory.
In a weather forecasting application, scientists use computational models that divide the Earth’s
atmosphere into a large grid:
Page 5 of 6
Parallel Distributed Computing
The global domain is divided into subdomains, with each subdomain assigned to a separate
computing node.
OpenMP runs parallel computations within each node, using multiple threads to perform
calculations on grid points simultaneously.
This approach ensures that computations within a node are optimized using shared memory, while
inter-node communication is minimized using MPI, leading to efficient and scalable performance.
5. Conclusion
Hybrid memory models combining shared and distributed memory offer significant advantages for
large-scale HPC applications. By leveraging MPI for inter-node communication and OpenMP for intra-
node parallelism, hybrid models reduce communication overhead, improve resource utilization, and
enhance scalability. This approach is essential for solving complex scientific and engineering
problems in modern high-performance computing environments.
Page 6 of 6