UNIVERSITY OF EDUCATION

Name:
Manahil Maqbool(bsf2201584)
Nawal Sultan(bsf2201563)

Class:
BSCS-5(Evening)

Subject:
Parallel Distributed Computing
Assignment-2

Submitted to:
Dr. Khalid Hamid

Topic:
COMMON PARALLELIZATION STRATEGIES
 COMMON PARALLELIZATION STRATEGIES

ARTICLE-1 (A domain decomposition strategy for hybrid parallelization of moving particle semi-
implicit (MPS) Method for computer cluster)
Lately, hybrid parallelization strategies have attracted much attention since they aim at possible performance
improvement of computational methods, as applied, for example, in fluid dynamics and particle-based simulation.
In order to handle a large number of particles on computer resources, an MPS method, initially developed for a
simulation of incompressible flows, presents special difficulties. The domain decomposition method decomposes
the computational domain into subdomains that can be computed in parallel. An improved load balancing and
distribution of memory across clusters using MPI with OpenMP. Additional improvements include hierarchical
and non-geometric strategies for domain decomposition, which exhibit better scalability and faster execution
times. These strategies enable the simulation of models with millions of particles, and this is almost indispensable
when dealing with complex engineering problems at different scales, such as fluid-structure interactions or large-
scale tsunami simulations. [1].

ARTICLE-2 (A Hierarchical Partitioning Strategy for an Efficient Parallelization of the Multilevel

Fast Multipole Algorithm)
Emphasizing the increasing trend toward optimization of load balancing and reduction of communication overhead
in distributed memory architectures, a review of parallelization techniques for large-scale computations in
computational electromagnetics is discussed. Among various strategies, one notably proposed by [2] had a clear
view of hierarchical partitioning with the best implementation at the multilevel fast multipole algorithm (MLFMA).
Thus, it spreads clusters and samples across processors, avoiding the limitations of previous approaches, which
divide only clusters or only samples. Improving both the balance on workload distribution and the inter-processor
communication, the hierarchical strategy has shown till now much improvement, especially when solving problems
with millions of unknowns. Earlier methods, such as the simple or hybrid parallelization schemes, have adverse load
balancing at higher levels of MLFMA or suffer from high communication requirements, which appear very
inefficient when scaled to a large number of processors [2].

The hierarchical method improves efficiency not only on traditional systems but is especially well suited to modern,
multi-core architectures making it very relevant for problems in modern computations [3] [4]. This advance has
made it possible to solve the scattering problems involving hundreds of millions of unknowns, thus enlarging the
scope of practical application in fields such as antenna design and radar cross-section analysis.

ARTICLE-3 (A Review of the Parallelization Strategies for Iterative Algorithm)

Parallelization of iterative algorithms is the necessity so that its computational efficiency can be enhanced
significantly by considering the upscaling multi-core processors and distributed systems. This nature requires
iterative algorithms to converge to the solution only by repetition of computation, which of course incurs a huge cost
in computation for large data [5]. Discuss some strategies to fight the bottlenecks of these algorithms, such as the
use of concurrently computable logical units, multi-initial state parallel search, data parallelism, and task
parallelism. These strategies are intended to distribute the computational tasks for acceleration of the overall
computation across several threads or processors without lowering the accuracy of the algorithm itself. The authors
also discuss the need to address the convergence issues inherent in asynchronous iterative algorithms, which should
invoke more sophisticated techniques like detection of convergence for reliability of computation.

Article-4 (A survey on Parallel computing and its applications in Data-parallel problems using gpu
architecture)
This research, which shows parallel computing and its application to data-parallel problems using GPU architecture,
highlights the critical role of parallelism in HPC. Given the fact that the field of computer science was evolving
toward multi-core and many-core architectures, an inclination toward parallel computing, especially from the
introduction of GPUs, or Graphics Processing Units, ensued. GPUS offer tremendous parallel processing power for
a relatively affordable price, which means super-computing can be done at a desktop scale. ) [6] discuss several
computational problems like n-body simulations, collision detection, and cellular automata, which benefit
significantly from the parallel algorithms that leverage GPUS. The difficulties in programming GPUs are due to
engineering limitations imposed by the architecture of the GPU itself. Once those barriers are crossed, however, the
payoffs of code performance are quite impressive; for instance, there are two orders-of-magnitude speedups over
traditional CPU-based solutions. As a complement to the above explanation, the authors also note that efficient
algorithms on the GPU are not trivial and need careful consideration of the memory and synchronization
mechanisms of the GPU.

Article-5 (Automatically Finding Model Parallel Strategies with Heterogeneity Awareness)

Authors of the paper, "Automatically Finding Model Parallel Strategies with Heterogeneity Awareness" by [7],
argued that the complexity in the training process of huge machine learning models is increasing. The prevailing
model parallelism techniques tend to be centered on expert-in-the-loop approaches and typically do not fit
heterogeneous cluster settings well, which comprises various devices with heterogeneity differences in computation
power and communication bandwidth. The authors present AMP, a system that automatically derives optimal
parallel strategies based on heterogeneity in both the model and cluster. AMP uses dynamic programming along
with a novel cost model to evaluate various strategies. The presented system outperforms previous methods by up to
1.77× more throughput on heterogeneous models and clusters compared to state-of-the-art systems like Megatron-
LM.

Article-6 (Co-optimizing Network Topology and Parallelization Strategy for Distributed Training
Jobs)
In this work, we design a new system that improves the training time of distributed deep neural networks (DNNs).
Traditional datacenter architectures, like Fat-tree, fall short for emerging DNN workloads as they don't optimize the
network topology aside from leveraging the parallelization strategies of the training jobs. TOPOOPT overcomes the
former limitation by simultaneously co-optimizing both dimensions: computation and communication using
alternating optimization. The system constructs efficient network topologies for DNN training jobs, exploiting
mutability of All Reduce traffic and an algorithm inspired by group theory called Totient Perms. This offers
impressive performance gains, showing up to a 3.4× times reduction in training time over a standard Fat-tree
network. TOPOOPT promises good scalability and cost-effectiveness for large-scale DNN training on top of
heterogeneous workloads [8].

Article-7 (Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML.

Proceedings of the VLDB Endowment)
This paper demonstrates that hybrid parallelization strategies have become more important in managing the extreme
scale and complexity of ML workloads recently in the machine learning domain. SystemML, as designed based on
top of MapReduce, offers a declarative framework to efficiently automate the optimization of large-scale machine
learning algorithms with a balance between data and task parallelism. [9] even proposes a unified strategy by tying
together complementary strategies such as ParFOR, usually applied in HPC. The above complementary strategies
take advantage of multi-core processors and distributed systems by adeptly partitioning activities and data according
to the workload [9]. Furthermore, the framework ensures the optimal selection of execution plans; it actually does
that automatically.

On the other hand, this framework analyses memory constraints, workload characteristics, and data access patterns
to achieve both scalability and efficiency. These strategies have been demonstrated in experiments to better
traditional approaches because of their dynamic adaptation to different workloads, be they small or large, and this
translates to considerable improvements in performance [9]. This hybrid approach effectively addresses the dual
challenge of constraints on memory and also requires co-execution in large-scale machine learning systems of the
compute-intensive and data-intensive tasks.

Article-8 (Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training)

It teaches us about advancements in multi-GPU parallelization strategies, which are focused on optimizing the
training of deep learning. Particularly interesting is how the strategy of DP has been to date usually-the exact same
model is trained on different subsets of the data across distinct devices. However, DP has higher overhead due to
increased communication as the number of devices increases [10] [11]. As a consequence, researchers started
probing hybrid approaches that combine DP with MP, where the computational graph is split over more than one
device but across different devices to parallelize within a single model.

Studies also indicate that hybrid parallelism provides superior scaling and efficiency at large device counts by
overcoming the limitations of DP alone [12].

For instance, hybrid strategies outperform DP-only strategies for providing massive speedups in Inception-V3,
GNMT, and BigLSTM models, reducing end-to-end training times up to 26.5% [13]. The emphasis of such
approaches, therefore, lies on the balancing of data and task distribution so as to maximize performance on both
multi-core processors and distributed systems.

Article-9

This study shows that developing parallel strategies for generalized planning receives much attention because
solving multiple instances of planning problems often faces serious computational challenges. Best-First
Generalized Planning is perhaps one of the most prominent approaches that rely on heuristic search to produce
general plans, making BFGP a promising candidate for parallelism as well. [14] Contemplated two parallelization
strategies for BFGP. The strategy aimed at improving scalability in addition to reducing the execution time when
dealing with large-scale planning domains. Initially, nodes were expanded sequentially to a given threshold where
the search will be conducted parallelly in multiple threads without sharing nodes among the threads. This strategy
indeed offered drastic improvements in terms of performance with some runs attaining a 98x speedup. The second
strategy distributed promising nodes over threads to balance the load and achieve better workload distribution as
well to reduce communication overheads. The results were mixed in this regard for the domain; as "Visitall"
performed much better, whereas in others it brought up execution times [14]. It is concluded that parallelization
may significantly enhance the efficiency of GP algorithms, although the choice of the strategy depends on the
particular problem domain to be addressed.

Article-10

(Parallelism Strategies for Distributed Training)

Distributed training is highly essential for large-scale deep learning models in scenarios involving big data or huge
architectures. Several parallelism strategies have been developed that work effectively over the limitations of single-
node computations, while common practices for alleviating large batch sizes include splitting the data across
multiple GPUs, thereby allowing for faster and more efficient computations. By contrast, model parallelism is
commonplace when the number of parameters in the model exceeds the capacity of memory that a single machine
can hold, and so some portion of it can be treated by each GPU. Other advanced techniques, such as pipeline
parallelism, minimize underutilization by running different parts of the model on different devices. The other recent
technique, such as the ZeRO-optimizer, optimizes memory usage by distributing the states of the models across
multiple processes. Each approach addresses a particular challenge of distributed training and can be combined for
specific cases.
Article-11

Article-11 (Parallelization of Swarm Intelligence Algorithms)

Swarm intelligence algorithms have become popular for solving complex optimization problems since they can
inherit this nature-inspired collective behavior of, say, ant colonies or a flock of birds. However, the computational
cost of these algorithms grows strongly with the complexity of problems to be solved. Therefore, parallel versions of
SI algorithms have been realized using modern multi-core processors and GPUs which enhance performance. The
popular algorithms developed in PSO, ACO and ABC had parallel implementations that ensured reduced
computation time while increasing the accuracy of outcomes with large problem instances being processed. Framing
the algorithms into OpenMP, MPI, and CUDA are some of the common frameworks used for parallelization.
Moreover, hybrid approaches like nested parallelism can fully exploit the capabilities of modern hardware with the
combination of a large number of levels of parallelism. [15]

Abstract
This paper presents general strategies on parallelization to improve computing with high performance. The objective
is to discuss several approaches to parallelization, in this context, data, task, and pipeline parallelism, benefits of
these approaches, which support optimization of the processing speed and resource usage. This work compares
frameworks using comparative study frameworks to illustrate their effectiveness in various types of computational
tasks through simulation-based experiments.

The research design is based on a systematic literature review followed by practical simulation experiments to
estimate performance gains and pinpoint bottlenecks in parallelized applications. Key results show that for
independent tasks, task parallelism provides considerable speedups, while data parallelism dominates in applications
with large datasets, demonstrating a robust improvement in the time to execution and resource usage.

However, the results so far seem to indicate that there is potential for getting more performance using an integrative
approach with parallelization strategies being applied in parallel.

Implications for Future Work

Adaptive parallelization techniques would be an interesting new direction along with their use in such areas as
machine learning and big data analytics. The results from the present work are valuable for practitioners who have
to include efficient parallel processing solutions in their systems.

Keywords: Parallelization Strategies, Computational Efficiency, Data Parallelism, Task Parallelism,

Performance Optimization, High-Performance Computing.
 Introduction

overview

Computational demands continually grow almost in every field of research and practice. It becomes critically
necessary to provide efficient processing techniques for such demanding computations. A way of trying to achieve
this is through the parallelization strategy of partitioning a computational task into manageable sub-tasks which can
be carried out at the same time, hence improving performance and reducing the execution time. This capability to
handle big data as well as complex computations, which can be found increasingly, especially in scientific research,
data analytics, or artificial intelligence, overcomes much difficulty via the utilization of a number of processing
units; these include distributed systems and cloud computing along with multicore processors.

Several common parallelization strategies are used, with all of them having a number of strengths in different fields.
Data parallelism is intended to spread pieces of data across several processors and is ideal for operations that
implement a predetermined function on large amounts of data. A task-parallel description of executing multiple
tasks or functions in parallel is suited when the tasks do not depend on each other. Pipeline parallelism is suited for
various steps of a process operating simultaneously, optimizing the flow of data throughout a series of operations.

Parallelization is one of the strategies employed for improving computational efficiency as well as addressing
challenges posed by modern computing workloads. Therefore, in this paper, we take a look at these strategies and
analyze their merits and implementation methods, along with contexts of most successful usage.

Litrature

Parallelization has been an area of intensive study for many years in the context of high performance computing. An
important work by [16] focuses on parallel architectures and algorithms, emphasizing the principles behind
successful parallelization strategies. The work demonstrates how this particular flavor of parallelism-data, task, or
pipeline-is highly dependent on the computational workload in the generation of good performance.

Recently, the works by [17] titled "MapReduce: Simplified Data Processing on Large Clusters" have dominated
discourse on data parallelism. This work demonstrates effective management of large scale processing for big data.
This work was established with that principle demonstrated by experience that resulted in better performance using
it. It shows how parallelization can result in scale-out and fault-tolerant tasks on computers.

On the other hand, task parallelism has been studied by [18], in which they proposed methods for optimizing the
execution of tasks in parallel. They found that optimal utilization of resources with little idleness time on processors
depends solely on effective load balancing and scheduling of tasks. Such a study importance considers the
independence of the tasks when parallel strategies are to be carried out.

Another work that always comes to mind when discussing parallelism is [19], especially his law, which serves to
quantify the potential speedup of a task if only some part of it can be parallelized. We have illustrated here how
Amdahl's insights have an effect on the design of parallel algorithms by bringing to light the limitations imposed by
the sequential elements of the task.

In summary, the literature on parallelization strategies produces a rich tapestry of methodologies and findings that
feed into contemporary practice. Studies are altogether concerted toward that goal of understanding how best to put
parallelization into place in the face of modern computing.

Identify Gaps in the Literature

This means there is insufficient published literature on the effectiveness of hybrid parallelization techniques that
combine data and task parallelism. It leaves an incidence gap in the scope of understanding how these strategies can
be merged to produce optimal performance.
Despite the fact that a large number of research work has been done on the same topic, none of these studies has
shown a keen interest in investigating the real-time adaptability of parallelization techniques in dynamic computing
environments. This deficiency restricts the use of parallelization in situations where characteristics of workloads
change frequently.

As shown by [16], the efficiency of metrics more than speedup, such as energy efficiency and fault tolerance,
remains unexplored. Such a framework that can fully assess these critical issues will provide better insight into the
trade-offs between different parallelization strategies.

Contrary to many expectations, data on the efficiency of existing parallelization methods argues that they do not
fully realize their potential in most modern multi-core and distributed architectures. Many of them appear to rely on
static approaches that do not adapt to the new complexity of computations.

Solution

Hybrid Parallelization Strategies: Design and Experiment Hybrid Models of Data and Task Parallelism. This
research will design and experiment with hybrid models combining data and task parallelism and show how
effectiveness is combined to improve performance across differing computational tasks.

Adaptive Parallelization Framework: Design a dynamic adjustment of parallelization strategies as influenced by

real-time workload analysis; indeed, employ machine learning algorithms to optimize the distribution of tasks
involved in parallel processing.

Extensive evaluation framework: For obtaining a comprehensive assessment of the parallelization strategy, we are
designing our multi-component evaluation framework based on metrics such as speedup, energy consumption,
scalability, and fault tolerance.

Empirical Validation: We would want to present empirical studies on real applications like cloud computing and
machine learning to be able to validate the sanity of our strategy and draw practical insights.

Justification for Suggested Solutions

1. The growing need to parallelize new sophisticated computational tasks renders a novel method of parallelization.
The present parallelization schemes need to be overcome with the limitations and the hybrid strategy proposed will
definitely enhance the performance as well as the utilization of resources.

2. Since no study till date has undertaken experiments to find out if the adaptive parallelization frameworks are
adaptable in real-time environments, this work will surely open further scopes for research on dynamic
optimizations concerning resource usages.

3. The outcome will enable practitioners to apply pertinent parallelization techniques in cloud computing and
machine learning and, thus achieve substantial processing efficiency improvements.

4. It can be seen that the contribution of this work is pretty clear since the gained outcomes shall be used as
guidelines for real-world applications in deploying the parallelization technique; in essence, the results yield
essential metrics toward informed decision making.
METHODOLOGY

Common parallelization Strategies split large, complex tasks into smaller, units that can be processed concurrently,
which makes it faster execution and hence better performance in systems making use of it. It divides big workload
into small independent units to enable concurrent processing by the system. The system keeps track of load
distribution for resource-efficient allocation that can balance workloads to avoid bottlenecks. Important strategies
include task parallelism, where different tasks are executed in parallel, and data parallelism, whereby the same
operation is enforced across distributed chunks of data. The system can be dynamically adjusting resources to
optimize its performance depending on characteristics and also maintain consistent efficiency with varying tasks.

Challenges faced during processing

In parallel processing, many difficulties can arise and affect performance. The most difficult part is to deal with data
dependency as different tasks may have to wait for some other output before continuing themselves. Balancing
workloads among processors also presents a challenging problem due to the fact that unbalancing work results in
processors and, as a result, affects the overall effectiveness. Another is the overhead in communication because one
may need to interchange information or sometimes synchronize between tasks, slowing down the processing. Also,
synchronization will now be necessary, should tasks share resources, and may cause delay and complexity. Fault
tolerance must also be obtained since system failures may even bring processing to a halt; hence recovery
mechanisms for tasks with minimal impact are needed. There is also the problem of scale: the management of
dependencies, communication, and resources becomes more complex as systems increase in scale. Debugging and
testing becomes more difficult in parallel environments because bugs may manifest in a non-deterministic way.
Finally, because there will be competing demands on the same resources, delays and reductions in performance will
result.
 Start

Task Initialization

Parallelization
Strategy |Selection

Resource
Allocation

Concurrent Task
Execution

Data Dependency
Check

Is
Dependent

Ye No
sss
s

Wait/Reschedule Task until Synchronization

dependency is resolved

Scalability Check

End
Task Initialization: Start by defining tasks and preparing for execution.

Task Dependency Check: Determine if tasks have dependencies; if yes, order or group them.

Load Balancing: Distribute tasks evenly across processors and adjust if imbalance is detected.

Task Execution with Synchronization: Run tasks concurrently, applying controls to prevent conflicts.

Monitor Communication Overhead: Check for delays in task communication; optimize if necessary.

Fault Detection and Recovery: Isolate and recover failed tasks while continuing execution.

Resource Contention Check: Assess if tasks compete for resources and adjust allocations.

Scalability Assessment: Evaluate system performance and optimize for better scaling if needed.

Task Completion: Ensure all tasks are completed efficiently and processors are balanced.

Results:
In addition, simulation experiments showed that task parallelism is able to result in up to 60% decrease in execution
time for independent workloads, whereas data parallelism could improve processing speed by up to 50% in
processing large datasets. The combined use of hybrid parallel strategies showed the potential to reach
improvements in overall system performance by up to 70%, especially in heterogeneous environments. The other
related challenges found in the study are those related to communication overhead, load balancing, and resource
contention that might substantially affect scalability and efficiency. Alleviating these challenges using adaptive and
integrated parallelization techniques, in turn, may bring significant changes in high performance computing
applications pertaining to fields like cloud computing and machine learning.

Conclusions:
Analysis of various parallelization strategies indeed shows that "one size fits not all," as specific methods provide
certain advantages under particular characteristics of computational workloads. Task parallelism is very effective for
independent tasks, providing large speedups due to zero resources wasted and near minimal processor idle time.
Data parallelism is most helpful when dealing with large datasets, significantly speeding execution and optimizing
the resource assignment. The results do show the potential of these strategies within a hybrid parallelization model
towards better efficiency for complex and dynamic workloads. Pipeline parallelism has equal benefits in processes
amenable to division into sequential stages, thereby increasing overall throughput. However, the search study
highlights the gaps in areas related to adaptability and efficiency and thus calls for further research into the adaptive
parallelization framework that can dynamically adjust to real-time workload variations.
References

[1] D. C. L. F. E. &. N. K. Fernandes, " A domain decomposition strategy for hybrid parallelization of
moving particle semi-implicit (MPS) method for computer cluster. Cluster Computing.," 2015.

[2] Ö. &. G. L. Ergül, "A Hierarchical Partitioning Strategy for Efficient Parallelization of the Multilevel
Fast Multipole Algorithm. IEEE Transactions on Antennas and Propagation, 57(6), 1740-1750. DOI:
10.1109/TAP.2009.2019913.," 2009.

[3] J. L. C.-C. &. C. W. C. Song, "Multilevel fast multipole algorithm for electromagnetic scattering by
large complex objects. IEEE Transactions on Antennas and Propagation, 45(10), 1488-1493.," 1997.

[4] G. Sylvand, "Performance of a parallel implementation of the FMM for electromagnetics

applications. International Journal for Numerical Methods in Fluids, 43, 865-879.," 2003.

[5] Z. e. al., "A Review of the Parallelization Strategies for Iterative Algorithms," 2023.

[6] C. A. H.-K. N. &. M. L. References: Navarro, "A Survey on Parallel Computing and its Applications in
Data-Parallel Problems Using GPU Architectures. Communications in Computational Physics.,"
2013.

[7] D. W. H. X. E. &. Z. H. Li, " Automatically Finding Model Parallel Strategies with Heterogeneity
Awareness. NeurIPS," 2022.

[8] W. K. M. Z. Z. G. M. J. Z. M. D. Z. Y. &. K. Wang, " A. Co-optimizing Network Topology and

Parallelization Strategy for Distributed Training Jobs. 20th USENIX Symposium on Networked
Systems Design and Implementation (NSDI)," 2023.

[9] M. T. S. R. B. S. P. T. Y. B. D. R. &. V. S. Boehm, "Hybrid Parallelization Strategies for Large-Scale

Machine Learning in SystemML. Proceedings of the VLDB Endowment," 2014.

[10] A. S. I. &. H. G. E. Krizhevsky, "Imagenet classification with deep convolutional neural networks.
Communications of the ACM, 60(6), 84-90.," 2017.

[11] J. e. a. Dean, "Large scale distributed deep networks. Advances in Neural Information Processing
Systems 25.," 2012.

[12] 2. Wu et al. and 2. Szegedy et al..

[13] 2. Pal et al..

[14] A. &. S.-A. J. Fernández-Alburquerque, "Parallel Strategies for Best-First Generalized Planning.,"
2024.

[15] B. A. D. M. K. H. &. L. N. F. B. D. Menezes, "Parallelization of Swarm Intelligence Algorithms.

International Journal of Parallel Programming, 50, 486–514.," 2022.

[16] (Amdahl, 1967)

[17] (Culler, D. E. (1999). Parallel Computer Architecture: A Hardware/Software Approach. Morgan Kaufmann. )

[18] (Dean, J., & Ghemawat, S. . (2004). "MapReduce: Simplified Data Processing on Large Clusters." Proceedings
of the 6th Symposium on Operating System Design and Implementation.)

[19] (Gibbons, P. B. (1997). "Optimal load balancing in a parallel computation model. In Journal of Parallel and
Distributed Computing. (pp. 43(1), 54-70).)

[20]