Scribd
Upload
361 views

CS416 - Parallel and Distributed Computing: Lecture # 01

This document provides an overview of the CS416 Parallel and Distributed Computing course being offered in Spring 2021 at FAST-NUCES, Faisalabad Campus. It discusses the motivations for parallelism including limitations of Moore's Law and the memory/disk speed argument. It then distinguishes between parallel and distributed computing and covers practical applications in scientific, commercial, and other domains. The document concludes with discussing the organization and contents of the course which will cover fundamentals of parallel platforms, parallel programming, parallel algorithms, and references.

Uploaded by

abdullah ashraf
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
361 views

CS416 - Parallel and Distributed Computing: Lecture # 01

This document provides an overview of the CS416 Parallel and Distributed Computing course being offered in Spring 2021 at FAST-NUCES, Faisalabad Campus. It discusses the motivations for parallelism including limitations of Moore's Law and the memory/disk speed argument. It then distinguishes between parallel and distributed computing and covers practical applications in scientific, commercial, and other domains. The document concludes with discussing the organization and contents of the course which will cover fundamentals of parallel platforms, parallel programming, parallel algorithms, and references.

Uploaded by

abdullah ashraf
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

CS416 – Parallel

and Distributed
Computing
Lecture # 01

Spring 2021
FAST – NUCES, Faisalabad Campus
Agenda
2

 Motivating Parallelism

 Computing vs Systems

 Parallel vs Distributed Computing

 Practical Applications of P&D Computing

 Organization and Contents of the Course

CS416 - Spring 2021


3
Motivating Parallelism

CS416 - Spring 2021


Motivating Parallelism
4
 Developing parallel hardware and software has traditionally
been time and effort intensive.

 If one is to view this in the context of rapidly improving


uniprocessor speeds, one is tempted to question the need for
parallel computing.

 Latest trends in hardware design indicate that uni-processors may


not be able to sustain the rate of realizable performance increments
in the future .

 This is the result of a number of fundamental physical and


computational limitations.

 The emergence of standardized parallel programming environments,


libraries, and hardware have significantly reduced time to develop
(parallel) solution.

CS416 - Spring 2021


Motivating Parallelism
5

Moore’s Law
 Proposed by Gorden E. Moore in 1965 and revised in
1975.

 It states that [Simplified Version]


“Processing speeds, or overall processing power for
computers will double every18 months.”

 A more technically correct interpretation


“The number of transistors on an affordable CPU
doubles after every two year [18 months].”

CS416 - Spring 2021


Motivating Parallelism
6

Moore’s Law
 More computational power implicitly means more
transistors.
 Then why need second interpretation?
 Let’s have a look on empirical data from 1970 to 2009
 In 1970’s (i.e., from 1970 to 1979), processor speeds ranged from
740 KHz to 8 Mhz. Difference shows that both the interpretations
are correct.
 From 2000 to 2009, Speeds ranged from 1.3 GHz to 2.8 GHz.
 Relative speed difference is too low but, number of integrated
transistors ranged from 37.5 million to 904 million.
 So, second interpretation is more accurate.

CS416 - Spring 2021


Motivating Parallelism
7

Moore’s Law
 Why doubling the transistors does not doubles the
speed?
 Increase in number of transistor per processor is due to multi-core
CPU’s.
 It means, to follow Moore’s law, companies had to:
 Introduce ULSI( ultra large-scale integrations)
 And multi-core processing era.
 Will Moore’s law hold forever?
 Adding multiple cores on single chip causes heat issues.
 Furthermore, increasing the number of cores, may not be able to
increase speeds [Due to inter-process interactions].
 Moreover, transistors would eventually reach the limits of
miniaturization at atomic levels
CS416 - Spring 2021
Motivating Parallelism
8

Moore’s Law
 So, we must look for efficient parallel [multi-core]
software solutions to fulfill our future computational
needs.

 As stated earlier, number of cores on a single chip also


have some restrictions.

 Solution[s]?
 Need to find more scalable distributed and hybrid solutions

CS416 - Spring 2021


Motivating Parallelism
9

The Memory/Disk Speed Argument


 While clock rates of high-end processors have increased at
roughly 40% per year over the past decade, DRAM access
times have only improved at the rate of roughly 10% per year
over this interval.
 This mismatch in speeds causes significant performance
bottlenecks.
 Distributed platforms provide increased bandwidth to
memory of the system.
 Parallel platforms also provide higher caches [aggregated].
 Some of the fastest growing applications of parallel computing
utilize not their raw computational speed, rather their ability to
pump data to memory and disk faster.
CS416 - Spring 2021
Motivating Parallelism
10

The Data Communication Argument


 As the network evolves, the vision of the Internet as one
large computing platform has emerged.

 In many applications like databases and data mining


problems, the volume of data is such that it (sometimes)
cannot be moved.

 Any analyses on this data must be performed over the


network using parallel computing techniques

CS416 - Spring 2021


Computing vs Systems
11

Distributed Systems
 A collection of autonomous computers, connected
through a network and distribution middleware.
 This enables computers to coordinate their activities, and to share
the resources of the system.
 The system is usually perceived as a single integrated computing
facility.
 Mostly concerned with the hardware-based accelerations
Distributed Computing
 A specific use of distributed systems to split a large and
complex processing into subparts, and then execute the
parts in parallel to increase the productivity.
 Computing mainly concerned with software-based accelerations
(i.e., designing and implementing parallel algorithms)

CS416 - Spring 2021


Parallel vs Distributed Computing
12

Parallel (shared-memory) Computing


 The term is usually used for developing concurrent
solutions on the following two types of the systems:
1. Multi-core Architecture
2. Many-core architectures (i.e., GPU’s)

Distributed Computing
 This type of computing is mainly concerned with
developing algorithms for the distributed cluster systems.
 Here distributed means a geographical distance
between the computers without any shared-Memory.

CS416 - Spring 2021


Practical Applications of P&D Computing
13

Scientific Applications
 Bioinformatics
 DNA sequence alignment (fine-grained, Coarse-grained, and
hybrid solutions)
 Applications in astrophysics have explored the evolution
of galaxies, thermonuclear processes, and applications in
the analysis of extremely large datasets from telescope.
 Advances in computational physics and chemistry have
explored new materials, understanding of chemical
pathways, and more efficient processes
 e.g., Large Hydron Collider (LHC) at European Organization for
Nuclear Research (CERN) generates petabytes of data for a
single collision.
CS416 - Spring 2021
Practical Applications of P&D Computing
14

Scientific Applications

 Weather modeling for simulating the track of natural


hazards like the extreme cyclones (storms).
 Flood prediction

CS416 - Spring 2021


Practical Applications of P&D Computing
15

Commercial Applications
 Some of the largest parallel computers power the wall
street.

 Data mining-analysis for optimizing business and


marketing decisions.

 Large scale servers (mail and web servers) are often


implemented using parallel platforms.

 Applications such as information retrieval and search are


typically powered by large clusters.

CS416 - Spring 2021


Practical Applications of P&D Computing
16

Other Applications
 Network intrusion detection: A large amount of data
needs to be analyzed and processed

 Cryptography (the art of writing or solving codes)


employs parallel infrastructures and algorithms to solve
complex codes.

 Graphics processing

 Embedded systems increasingly rely on distributed


control algorithms.
CS416 - Spring 2021
Limitations of Parallel Computing:
17

 Exploring the proper parallelism from a problem is a


hectic process.

 It requires designing the proper communication and


synchronization mechanisms between the processes
and sub-tasks.
 The program’s subtasks must have low coupling and
high cohesion. But it’s difficult to create such
programs.

 It needs relatively more technical skills to code a


parallel program.
CS416 - Spring 2021
Organization and Contents of the Course
18

 Fundamentals: This part of the class covers basic


parallel platforms, principles of algorithm design,
group communication primitives, and analytical
modeling techniques.

 Parallel Programming: This part of the class deals


with programming using message passing libraries,
threads and the kernels.

 Parallel Algorithms: This part of the class covers basic


algorithms for matrix computations, graphs, sorting,
and optimization using dynamic programming.

CS416 - Spring 2021


References
19
1. Kumar, V., Grama, A., Gupta, A., & Karypis, G. (1994). Introduction to
parallel computing (Vol. 110). Redwood City, CA: Benjamin/Cummings.

2. Quinn, M. J. Parallel Programming in C with MPI and OpenMP,(2003).

CS416 - Spring 2021


Questions
20

CS416 - Spring 2021

You might also like