Parallel & Distributed

Computing: Spring-2020

lec#1

Dr Muhammad Imran

1.1
 Course Information
• Prereq: Computer Architecture, Programming in C/C++
• Text/Ref:
- Parallel Computers: Architecture & Programming by Rajaraman
- Introduction to Parallel Computing (2nd ed) by Grama
- Programming Massively Parallel Processors (2nd Ed) by Kirk

• Grading Policy
– Quizzes (4-6) : 4%
– Assignments(4-6) : 6%
– Semester Project : 10
– Midterm Exam : 30 %
– Final Exam : 50 %

• Instructor: Dr Muhammad Imran

– Email: [email protected]
• Student Resource for assignments/quizes/exams info:
– LMS/CR/GR
1.2
 Course Introduction
With advances in computer architecture, high performance
multiprocessor computers have become readily available
& affordable. As a result supercomputing is accessible to
a large segment of industry that was once restricted to
military research & large corporations.
The course is comprised of architecture & programming
of Parallel computing systems; design concepts,
principles, paradigms, models, performance evaluation &
applications.
1.3
 Course Contents
• Parallel Architecture Models:
– Implicit: Pipelining, Superscalar & VLIW.
– Explicit: SIMD,MIMD, multicore SMP, vector/array
processor, MPP & Clusters
• Parallel Memory Models:
– SM, DM, DSM, UMA / NUMA
– Cache Coherence protocols

• Interconnection Networks (IN):

– Static vs Dynamic network topologies
– LANs vs SANs (Infiniband)

1.4
• Heterogeneous (CPU/GPU) Parallel Computing
using CudaC
• Multicore SMP programming using OpenMP
• Cluster multicomputer programming using MPI
• Group/Parallel Communications & MPI
• Combining Parallel Computing (Cluster/GPU)
using MPI / cudaC together

1.5
 Course Contents…cont’d

• Embedding / Mapping application to IN

– e.g Ring/ Mesh into Hypercube & vice versa

• Parallel Data Partitioning Techniques

– 1D/ 2D/ 3D; row-wise/ column-wise;

• Parallel Algorithms: Performance measures &

Programming models: SPMD/MPMD & PRAM
• Applications:
– Cloud, Robotics, Neural Networks, Cyber security

1.6
 Introduction
• What is Parallel Computing
– Parallel computing is an evolution of serial computing
• Parallel computing is a type of computing architecture
in which several processors execute or process an
application or computation simultaneously.
• Parallel computing is also known as parallel
processing.

1.7
 Why Use Parallel Computing?
• Save time and/or money:
• Throwing more resources at a task shortens its time to
completion
• Solving problems in shorter time results in saving big
Money in many practical situations.
• Parallel computers can be built from cheap, commodity
components.
– A computer which consists of a number of inter-connected
computers which cooperatively execute a single program to solve a
problem is called a parallel computer.

1.8
 Why Use Parallel Computing?
• Provide concurrent Working environment:
– A single compute resource can only do one thing
at a time. Multiple computing resources can do
many things simultaneously.
– For example, Access Grid www.accessgrid.org provides
global collaboration network where people around the
world can meet & conduct work "virtually"

1.9
Why Use Parallel Computing?
• Integrating Remote Resources usage:
– Using compute resources on a wide area network, or
even the Internet when local compute resources are
scarce.
– For example:
• SETI@home satiathome.berkeley.edu uses over 330,000
computers for a compute power over 528 TeraFLOPS
• Folding@home folding.stanforg.edu uses over 340,000
computers for a compute power of 4.2 PetaFLOPS

1.10
 Who and What ?
• Top500.org provides statistics on parallel computing users in the charts below

1.11
 Why Use Parallel Computing?
• Solve larger problems:
– Many problems are so large and/or complex that it is
impractical or impossible to solve them on a single
computer.
– For example:
• "Grand Challenge“ en.wikipedia.org/wiki/Grand_Challenge
• Ones that cannot be solved in a reasonable amount of time with
today’s computers. Obviously, an execution time of 10 years is always
unreasonable
• Such problems require PetaFLOPS & PetaBytes of
computing resources
• Some of these are:
– Numerical simulation to predict the behaviour of physical systems.
– High performance graphics—particularly visualization, and animation.
– Big data analytics for strategic decision making.
– Synthesis of molecules for designing medicines.
1.12
 Why Use Parallel Computing?
• In whether forecasting, a mathematical model of the behavior
of the earth’s atmosphere is developed in which the most
important variables are
– the wind speed, air temperature, humidity and atmospheric pressure.
• The objective of numerical weather modeling is to predict the
status of the atmosphere at a particular region at a specified
future time based on the current and past observations of the
values of atmospheric variables.

1.13
 Why Use Parallel Computing?
• In one model, the regions are demarcated by using 180 latitudes and 360
longitudes equally spaced around the globe.
• In the vertical direction 12 layers are used to describe the atmosphere.
• For each region one point is taken as representing the region and this is
called a grid point.
• At each grid point the values of the 5 variables (namely air velocity,
temperature, pressure, humidity, and time) are computed.
• The simultaneous algebraic equations are normally solved using an iterative
method.

1.14
 Why Use Parallel Computing?
• In an iterative method several iterations (100 to 1000) are needed
for each grid point before
• the results converge. The calculation of each trial value normally
requires around 100 to 500 floating point arithmetic operations
• Thus, the total number of floating point operations required for
each simulation is approximately given by:
• Number of floating point operations per simulation = Number of
grid points × Number of values per grid point × Number of trials ×
Number of operations per trial.
• In this example we have:
• Number of grid points = 180 × 360 × 12 = 777600
• Number of values per grid point = 5, Number of trials = 500
• Number of operations per trial = 400
• Thus, the total number of floating point operations required per
simulation = 777600 × 5 × 500 × 400 = 7.77600 × 1011.
1.15
 Why Use Parallel Computing?
• In an iterative method several iterations (100 to 1000) are needed
for each grid point before
• the results converge. The calculation of each trial value normally
requires around 100 to 500 floating point arithmetic operations
• Thus, the total number of floating point operations required for
each simulation is approximately given by:
• Number of floating point operations per simulation = Number of
grid points × Number of values per grid point × Number of trials ×
Number of operations per trial.
• In this example we have:
• Number of grid points = 180 × 360 × 12 = 777600
• Number of values per grid point = 5, Number of trials = 500
• Number of operations per trial = 400
• Thus, the total number of floating point operations required per
simulation = 777600 × 5 × 500 × 400 = 7.77600 × 1011.
1.16
 Visualization and Animation
• In visualization and animation, the results of computation are to be
realistically rendered on a high resolution terminal.
• Assume that
– G is the number of area elements where the picture is to be rendered
– R is the number of picture elements (called pixels) to be processed in each area
element.
– N is the no. of instructions which are required to process a pixel
– P is the no. of times the computation should be repeated in a second for animation.
• Thus, computational complexity in this case is: G × R × P × N
• Typically, for animation
– The computation should be repeated at least 60 times a second i.e P= 60
– G = 105, R = 107
• Thus, a computer must process 105 x 107 x 60 x N = 6 x N x 1013
instructions/sec for animation.
• If N=1 and each instruction takes 100 ns then
• Processing for Animation will be complotted in 11.57days

1.17
 Encrypted Code Breaking
• In brute-force attack, simply try every key
• most basic attack, proportional to key size
• assume either know / recognise plaintext
Key Size Number of Time required at Time required at
(bits) Alternative 1 decryption/µs 106 decryptions/µs
Keys
32 232 = 4.3  109 231 µs = 35.8 2.15 milliseconds
minutes
56 (DES) 256 = 7.2  1016 255 µs = 1142 10.01 hours
years
128 (AES) 2128 = 3.4  1038 2127 µs = 5.4  1024 5.4  1018 years
years

168 (3-DES) 2168 = 3.7  1050 2167 µs = 5.9  1036 5.9  1030 years
years

26 26! = 4  1026 2  1026 µs = 6.4  6.4  106 years

characters 1012 years
permutation
1.18
 The Future:
• During the past 20 years, the trends
indicated by ever faster networks,
distributed systems, & multi-processor
computer architectures
(even at the desktop level) clearly show that
parallelism is the future of computing

1.19