Parallel & Distributed Computing: By: M. Imran Siddiqui
Parallel & Distributed Computing: By: M. Imran Siddiqui
Computing
Parallel Computing
Parallel computing refers to the process of executing several
processors an application or computation simultaneously.
• Instruction-Level Parallelism
• Task Parallelism
Types…
Bit – Level Parallelism
Every task is dependent on processor word size. In terms of
performing a task on large-sized data, it reduces the number of
instructions the processor must execute. There is a need to split the
operation into series of instructions.
For example, there is an 8-bit processor, and you want to do an
operation on 16-bit numbers. First, it must operate the 8 lower-order
bits and then the 8 higher-order bits. Therefore, two instructions are
needed to execute the operation. The operation can be performed
with one instruction by a 16-bit processor.
• Parallel Processing in Cryptography:
– Example: Bit-level parallelism is crucial in cryptographic algorithms like bitwise XOR operations used in
encryption and decryption. In parallel computing, multiple processors can perform XOR operations on
different bits of the data simultaneously, improving the overall processing speed.
• Parallel Bitwise Operations in Image Processing:
– Example: In image processing tasks, bitwise operations (e.g., AND, OR, XOR) are commonly used for tasks like
image masking or merging. Parallelizing these operations allows different processors to handle distinct
portions of the image, enhancing the speed of the overall processing.
• Distributed Binary Search:
– Example: In a distributed computing environment, a binary search algorithm can be parallelized at the bit
level. Each node in the distributed system can handle a specific range of bits, searching for the target value
simultaneously. This approach improves the efficiency of the search process.
• Parallel Compression and Decompression:
– Example: Bit-level parallelism is employed in parallel compression and decompression algorithms, such as
those used in distributed file systems. Multiple processors can work on compressing or decompressing
different sections of a file concurrently, enhancing the overall throughput.
Types…
Instruction – Level Parallelism
In a single CPU clock cycle, the processor decides in instruction-
level parallelism how many instructions are implemented at the
same time.
For each clock cycle phase, a processor in instruction-level
parallelism can have the ability to address that is less than one
instruction. The software approach in instruction-level
parallelism functions on static parallelism, where the computer
decides which instructions to execute simultaneously.
Types…
Instruction-Level Parallelism refers to the execution of multiple instructions simultaneously within a single processor. It
involves breaking down a program into smaller instruction-level tasks that can be executed concurrently to improve
overall performance. ILP is primarily concerned with how multiple instructions from a single program can be scheduled
and executed simultaneously. There are various techniques and approaches to exploit ILP
• Superscalar Processors:
– Definition: Superscalar processors can execute multiple instructions in parallel by having multiple execution units.
– Example: In a superscalar processor, multiple arithmetic or logic operations can be performed simultaneously, exploiting ILP to
improve instruction throughput. For instance, while one execution unit is performing a multiplication, another unit can be executing
an addition operation.
• Vector Processing:
– Definition: Vector processors operate on arrays or vectors of data in a single instruction, enabling parallel processing of multiple
data elements.
– Example: In scientific simulations within parallel computing, vector processing is often used. For instance, in weather simulations,
vector processors can simultaneously perform calculations on arrays representing different geographical points.
• SIMD (Single Instruction, Multiple Data):
– Definition: SIMD architecture allows a single instruction to be applied to multiple data elements simultaneously.
– Example: In distributed computing environments, SIMD can be employed for tasks like image processing. Each node in the
distributed system can apply the same operation to different parts of a large image in parallel, speeding up the overall processing
time.
Types…
Task Parallelism
Task parallelism is the form of parallelism in which the tasks
are decomposed into subtasks. Then, each subtask is allocated
for execution. And, the execution of subtasks is performed
concurrently by processors.
Types…
• Task: Image Processing
• Sequential Approach:
– In a traditional, sequential image processing algorithm, each pixel is processed one after the other.
• Task-Level Parallelism:
– Divide the image into distinct regions or segments.
– Assign each segment to a separate processor or core.
– Process each segment concurrently.
• Benefits:
• Faster image processing as multiple segments are processed simultaneously.
• Utilizes the capabilities of multi-core processors.
• More resources are used to complete the task that led to decrease the
time and cut possible costs. Also, cheap components are used to
construct parallel clusters.
• Comparing with Serial Computing, parallel computing can solve larger
problems in a short time.
• For simulating, modeling, and understanding complex, real-world
phenomena, parallel computing is much appropriate while comparing
with serial computing.
Advantages…
Advantages of Parallel Computing
• When the local resources are finite, it can offer benefit you over non-local
resources.
• There are multiple problems that are very large and may impractical or
impossible to solve them on a single computer; the concept of parallel
computing helps to remove these kinds of issues.
• One of the best advantages of parallel computing is that it allows you to
do several things in a time by using multiple computing resources.
• Furthermore, parallel computing is suited for hardware as serial
computing wastes the potential computing power.
Advantages…
• Faster Processing Speed:
– Example: Parallel computing is widely used in scientific simulations. For instance, in weather forecasting models, the workload can be divided
among multiple processors to simulate different regions of the atmosphere concurrently, significantly reducing the time required for complex
calculations.
• Improved Throughput:
– Example: In a parallel database system, multiple processors can handle queries and transactions simultaneously. This leads to improved
throughput, allowing more users to access and retrieve data concurrently, enhancing overall database performance.
• Scalability:
– Example: Parallel computing systems can easily scale to handle larger workloads. In the context of web servers, as the number of users accessing
a website increases, additional servers can be added to the network to distribute the load and maintain responsiveness.
• Resource Utilization:
– Example: Parallelism is crucial in multimedia applications. In video rendering, parallel processing can be employed to simultaneously handle
different frames or segments of a video, making efficient use of computational resources and speeding up the rendering process.
• High Performance for Parallel Algorithms:
– Example: Sorting large datasets is a common operation in many applications. Parallel computing can significantly accelerate sorting algorithms
by distributing the sorting task among multiple processors, allowing for faster and more efficient data processing.
• Redundancy and Fault Tolerance:
– Example: In distributed systems, redundancy can be achieved by replicating data or tasks across multiple nodes. If one node fails, the system can
still function using the redundant information from other nodes. This enhances fault tolerance and system reliability.
Advantages…
• Diverse Applications:
– Example: Parallel computing is employed in machine learning, particularly in training
deep neural networks. Parallel processing on GPUs accelerates the training process by
simultaneously updating the weights of different parts of the neural network.
• Cost-Effectiveness:
– Example: Cloud computing platforms offer parallel processing capabilities, allowing
users to scale their computing resources based on demand. This flexibility provides cost-
effective solutions for tasks that require varying levels of computational power.
• Scientific and Engineering Simulations:
– Example: Finite Element Analysis (FEA) in engineering simulations involves solving
complex equations to model structural behavior. Parallel computing accelerates FEA
simulations by distributing the computational workload among multiple processors.
Disadvantages of
Parallel Computing
Disadvantages…
Disadvantages of Parallel Computing
• It addresses Parallel architecture that can be difficult to achieve.
• In the case of clusters, better cooling technologies are needed in
parallel computing.
• It requires the managed algorithms, which could be handled in
the parallel mechanism.
• The multi-core architectures consume high power consumption.
Disadvantages…
Disadvantages of Parallel Computing
• The parallel computing system needs low coupling and high
cohesion, which is difficult to create.
• The code for a parallelism-based program can be done by
the most technically skilled and expert programmers.