Project Report

PRAM and Algorithm Designing

Submitted to:
Mr. Malik Ghulam Hussain

Submitted by:
Muhammad Junaid Iqbal (BSCSA-21-34)
Muzammil Majeed (BSCSA-21-10)
BS Computer Science (2021-25)

Department of Computer Science

Bahauddin Zakariya University Multan,
Pakistan
Introduction to Parallel and Distributed Computing

Parallel and Distributed Computing are essential fields in modern computing that focus on executing
computational tasks in parallel or across multiple systems. These approaches aim to address the growing
demand for high computational power and efficient resource management in a variety of domains,
including scientific research, machine learning, data analysis, and large-scale simulations.

-Parallel Computing refers to the simultaneous execution of multiple processes or tasks using multiple
processors or cores. The goal is to divide a problem into smaller tasks and execute them concurrently to
achieve faster results.

-Distributed Computing involves multiple autonomous systems working together on a shared task or
problem. These systems can be geographically dispersed and communicate through a network to achieve
a common objective.

The significance of Parallel and Distributed Computing lies in its ability to scale, providing performance
improvements by leveraging multiple processors and systems.

2. What is PRAM (Parallel Random Access Machine)?

The Parallel Random Access Machine (PRAM) is an idealized theoretical model used to design and
analyze parallel algorithms. It is a conceptual model that simplifies the complexities of parallel
computing systems and provides a foundation for studying the potential performance of parallel
algorithms in an idealized environment.

The key components of the PRAM model are:

-Processors (P): A set of processors that can operate in parallel.

- Shared Memory: A global memory accessible by all processors for reading and writing data.

- Memory Access: Processors can read from and write to any memory location in the shared memory
space.

The PRAM model helps in analyzing the time complexity of parallel algorithms and determining how
efficiently tasks can be parallelized. The simplicity of this model allows algorithm designers to focus on
the parallel aspects of the problem rather than the intricacies of hardware or system constraints.

3. PRAM Model Variants

There are several variants of the PRAM model, each with different assumptions about how processors
interact with shared memory. These variants help in modeling different levels of concurrency and
communication in parallel systems.

- EREW (Exclusive Read Exclusive Write): In this model, no two processors are allowed to simultaneously
read from or write to the same memory location. This ensures that there are no conflicts in memory
access.

- CREW (Concurrent Read Exclusive Write): Multiple processors can read from the same memory
location simultaneously, but only one processor is allowed to write to a location at any given time.

-CRCW (Concurrent Read Concurrent Write): Multiple processors can both read and write to the same
memory location simultaneously. This variant is more relaxed but often leads to more complex
synchronization issues.

Each of these variants has its own implications for the design and efficiency of parallel algorithms, and
the choice of model can affect the performance and complexity analysis of an algorithm.

4. Parallel Algorithm Design on PRAM

The design of parallel algorithms involves breaking down a computational problem into smaller,
independent tasks that can be solved simultaneously by multiple processors. The PRAM model simplifies
this process by assuming idealized conditions for parallel execution, such as instant communication and
no hardware limitations.

The general steps in designing parallel algorithms are:

1. Problem Decomposition: Break the overall problem into smaller sub-problems that can be solved
independently.

2. Task Distribution: Assign each sub-problem to a separate processor.

3. Parallel Execution: Processors work concurrently on their assigned tasks.

4. Result Combination: After all sub-problems are solved, the results are combined to form the final
solution.

Example of Parallel Algorithm Design:

Matrix Multiplication

In matrix multiplication, each element of the result matrix can be computed independently by
processors. The PRAM model allows for a parallel approach, where multiple processors handle different
elements of the matrix simultaneously, leading to a significant reduction in computation time compared
to a sequential approach.

Example: Parallel Sorting Algorithms on PRAM

Parallel sorting algorithms can be designed effectively using the PRAM model. Sorting involves arranging
elements of a data set in a specific order (e.g., ascending or descending). On PRAM, parallel sorting
techniques such as Merge Sort and Bitonic Sort can be used to achieve significant speedups.

- Merge Sort: Divide-and-conquer approach where the array is recursively divided into smaller sub-
arrays, sorted, and then merged together. This can be parallelized by sorting the sub-arrays concurrently.

- Bitonic Sort: This is a parallel sorting algorithm that works by repeatedly merging two sorted sequences
in a specific "bitonic" pattern. It is highly suited for PRAM models and can be executed in parallel using
multiple processors.

The time complexity of these algorithms depends on the number of processors available and the
memory access model. The efficiency of the algorithm is evaluated based on the PRAM variant used
(e.g., EREW, CREW, CRCW).

6. Design Challenges in PRAM

While PRAM provides an idealized model for parallel algorithm design, there are several challenges when
working with it:

- Memory Access Conflicts: In models where, multiple processors can access the same memory location
(e.g., CREW or CRCW), ensuring proper synchronization is crucial to avoid conflicts and data corruption.

- Load Balancing: It is important to distribute tasks evenly among processors to minimize idle time and
achieve maximum performance. Uneven load distribution can result in inefficient use of resources.

- Communication Overhead: In parallel computing systems, communication between processors can

become a bottleneck. PRAM assumes instant communication, but in real systems, message-passing or
shared memory communication can introduce significant delays.

7. PRAM vs Real-World Parallel Computing

While PRAM serves as an idealized model, real-world parallel computing systems are more complex. In
practical systems, memory is often non-uniform (NUMA), and processors may be connected in different
topologies, such as in a grid or network. Additionally, hardware limitations, network latency, and
communication bottlenecks affect performance.

In real systems, parallel models such as BSP (Bulk Synchronous Parallel) or LogP provide a more realistic
framework for designing and analyzing parallel algorithms. These models account for communication
costs, latency, and memory hierarchies, which are crucial for optimizing performance in distributed
systems.

8. Case Studies and Applications

PRAM-based parallel algorithms have been applied in various domains:

- Graph Algorithms: Parallel algorithms for graph traversal (e.g., BFS, DFS) and shortest path problems.

- Scientific Computing: Simulations and numerical methods such as finite element analysis or fluid
dynamics.

- Machine Learning: Parallel algorithms for training models, especially deep learning models, leveraging
GPU parallelism.

- Distributed Systems: Coordination and synchronization of processes in distributed memory systems,

such as cloud computing environments.

9. Conclusion

In summary, the PRAM model provides a simplified framework for designing and analyzing parallel
algorithms. It allows researchers and developers to focus on algorithmic design without being concerned
with hardware specifics. However, real-world systems are far more complex, requiring additional
considerations such as communication overhead, memory hierarchies, and fault tolerance.

Despite its idealized nature, the PRAM model serves as a useful tool for understanding the scalability of
parallel algorithms and serves as a stepping stone toward more realistic parallel computing models used
in modern systems.

Future directions in the field of parallel and distributed computing include the development of new
parallel models, optimization techniques for emerging hardware platforms (e.g., GPUs, multi-core
processors), and algorithmic advances in cloud and distributed systems.

10. References

- Cormen, T.H., Leiserson, C.E., Rivest, R.L., & Stein, C. (2009). Introduction to Algorithms (3rd ed.). MIT
Press.

- Pardalos, P.M., & Resende, M.G.C. (2014). Handbook of Parallel Computing: Models, Algorithms, and
Applications. CRC Press.

- Sarma, M., & Ramaswamy, M. (2021). Parallel Computing Algorithms: Analysis and Applications.
Springer.

- Almasi, G.S., & Gottlieb, A. (1989). Highly Parallel Computing. Benjamin/Cummings.