Microprocessor & Computer

Architecture (μpCA)
Unit 4: Introduction to Parallel Computing

UE22CS251B

Session : 4.1
Microprocessor & Computer Architecture (μpCA)
Sequential Computing

• Traditionally, software has been written for serial computation:

• To be run on a single computer having a single Central Processing Unit (CPU);
• A problem is broken into a discrete series of instructions.
• Instructions are executed one after another.
• Only one instruction may execute at any moment in time.
Sequential Computing Payroll Example
Microprocessor & Computer Architecture (μpCA)
Sequential Computing

• Traditionally, software has been written for serial computation:

• To be run on a single computer having a single Central Processing Unit (CPU);
• A problem is broken into a discrete series of instructions.
• Instructions are executed one after another.
• Only one instruction may execute at any moment in time.
Sequential Computing Payroll Example
Microprocessor & Computer Architecture (μpCA)
Parallel Computing

• In the simplest sense, parallel computing is the simultaneous use of

multiple compute resources to solve a computational problem.
• To be run using multiple CPUs
• A problem is broken into discrete parts that can be solved concurrently
• Each part is further broken down to a series of instructions
• Instructions from each part execute simultaneously on different CPUs
Parallel Computing Payroll Example
Microprocessor & Computer Architecture (μpCA)
Parallel Computing

• In the simplest sense, parallel computing is the simultaneous use of

multiple compute resources to solve a computational problem.
• To be run using multiple CPUs
• A problem is broken into discrete parts that can be solved concurrently
• Each part is further broken down to a series of instructions
• Instructions from each part execute simultaneously on different CPUs
Parallel Computing Payroll Example
Microprocessor & Computer Architecture (μpCA)
Sequential Compiler Vs Parallel Compiler

Vs
Microprocessor & Computer Architecture (μpCA)
A Computational Problem on a Parallel System

• The computational problem should be able to:

• Be broken apart into discrete pieces of work that can be solved simultaneously
• Execute multiple program instructions at any moment in time
• Be solved in less time with multiple compute resources than with a single compute
resource.
• The compute resources are typically:
• A single computer with multiple processors/cores
• An arbitrary number of such computers connected by a network
Microprocessor & Computer Architecture (μpCA)
Parallel Computers

• Virtually all stand-alone computers today are parallel from a hardware perspective
• Multiple functional units
 L1 cache, L2 cache, branch, prefetch, decode,
 floating-point, graphics processing (GPU), integer, etc.
• Multiple execution units/cores
• Multiple hardware threads

IBM BG/Q Compute Chip with 18 cores (PU) and 16 L2 Cache units(L2)
 Microprocessor & Computer Architecture (μpCA)
Parallel Computers
• Networks connect multiple stand-alone computers (nodes)
to make larger parallel computer clusters.

Example of typical LLNL parallel computer cluster

• Each compute node is a multi-processor parallel

computer in itself
• Multiple compute nodes are networked together
with an Infiniband network
• Special purpose nodes, also multi-processor, are
used for other purposes
Microprocessor & Computer Architecture (μpCA)
Why use Parallel Computers ?
The Real World is Massively Parallel :
• In the natural world, many complex, interrelated events are happening at the
same time, yet within a temporal sequence.
• Compared to serial computing, parallel computing is much better suited for
modeling, simulating and understanding complex, real world phenomena.

• Main Reasons for using Parallel Programming is to

• SAVE TIME AND/OR MONEY
• SOLVE LARGER / MORE COMPLEX PROBLEMS

Working in parallel shortens completion time Parallel computing can solve increasingly complex problems
Microprocessor & Computer Architecture (μpCA)
Why use Parallel Computers ?

• Main Reasons for using Parallel Programming

is to
• PROVIDE CONCURRENCY Collaborative networks

• TAKE ADVANTAGE OF NON-LOCAL RESOURCES

• MAKE BETTER USE OF UNDERLYING PARALLEL
HARDWARE

Parallel computing cores

TAKE ADVANTAGE : NON-LOCAL RESOURCES
 Microprocessor & Computer Architecture (μpCA)
Why use Parallel Computers ?
• Main Reasons for using Parallel Programming is to
• Science and Engineering
• Industrial and Commercial
• Global Applications

simulating a range of complex

commercial applications Top 500 Companies
physical phenomena
Microprocessor & Computer Architecture (μpCA)
Applications of Parallel Computing

• Real world Phenomena

can be simulated with
parallel computing
Microprocessor & Computer Architecture (μpCA)
Applications of Parallel Computing
“Grand Challenge Problems"
• “Big Data", databases, data mining
• Artificial Intelligence (AI) & Machine Learning ( ML)
• Web search engines, web based business services
• Medical imaging and diagnosis
• Pharmaceutical design
• Financial and economic modeling
• Management of national and multi-national corporations
• Advanced graphics and virtual reality, particularly in the entertainment industry
• Networked video and multi-media technologies
 Microprocessor & Computer Architecture (μpCA)
High Performance Computing
The use of the most efficient algorithms on computers
“capable of the highest performance” to solve the most demanding problems.

1 Higher speed (solve problems faster)

Important when there are “hard” or “soft” deadlines;
e.g., 24-hour weather forecast
Higher throughput (solve more problems)
2 Important when we have many similar tasks to perform;
e.g., Transaction processing

3 Higher computational power (solve larger problems)

e.g., Weather forecast for a week rather than 24 hours,
or with a finer mesh for greater accuracy
Microprocessor & Computer Architecture (μpCA)
Quantifying the Problem

Large problems
• 10,000 x 10,000 x 10,000 grid
• 10^12 grid points
• 4x10^12 double variables
• 32x10^12 bytes
• 32 Tera-Bytes.
Microprocessor & Computer Architecture (μpCA)
Quantifying the Capability to Solve Problem

FLOPS, or FLOP/S: FLoating-point Operations Per Second

Name Unit Value

103
kiloFLOPS kFLOPS
106
megaFLOPS MFLOPS
109
gigaFLOPS GFLOPS
1012
teraFLOPS TFLOPS
1015
petaFLOPS PFLOPS
1018
exaFLOPS EFLOPS
1021
zettaFLOPS ZFLOPS
1024
yottaFLOPS YFLOPS
 Microprocessor & Computer Architecture (μpCA)
A Growth-Factor of a Billion in Performance in a Career Super Scalar/Vector/Parallel
1 PFlop/s
(1015) IBM
Parallel BG/L

ASCI White
ASCI Red
Pacific
1 TFlop/s
(1012)
TMC CM-5 Cray T3D
2X Transistors/Chip Vector
Every 1.5 Years TMC CM-2
Cray 2
1 GFlop/s Cray X-MP
(109) Super Scalar
Cray 1
1941 1 (Floating Point operations / second, Flop/s)
CDC 7600 IBM 360/195 1945 100
1 MFlop/sScalar 1949 1,000 (1 KiloFlop/s, KFlop/s)
1951 10,000
(106) CDC 6600 1961 100,000
1964 1,000,000 (1 MegaFlop/s, MFlop/s)
IBM 7090 1968 10,000,000
1975 100,000,000
1987 1,000,000,000 (1 GigaFlop/s, GFlop/s)
1992 10,000,000,000
1993 100,000,000,000
1 KFlop/s 1997 1,000,000,000,000 (1 TeraFlop/s, TFlop/s)
(103) UNIVAC 1 2000 10,000,000,000,000
EDSAC 1 2005 131,000,000,000,000 (131 Tflop/s)

1950 1960 1970 1980 1990 2000 2010

Microprocessor & Computer Architecture (μpCA)
In progress : Top500.org

•During the past 20+ years, the trends indicated by ever faster networks,
distributed systems, and multi-processor computer architectures (even at the
desktop level) clearly show that parallelism is the future of computing.

•In this same time period, there has been a greater than 500,000x increase in
supercomputer performance, with no end currently in sight.

•The race is already on for Exascale Computing!

• Exaflop = 1018 calculations per second
Microprocessor & Computer Architecture (μpCA)
This is what you are expected to Program
2

4 5
Microprocessor & Computer Architecture (μpCA)
Where Do We live?
Rpeak
Rmax (TFlop/s Power
Rank System Cores (TFlop/s) ) (kW)
165 Supercomputer SERC - Cray XC40, Xeon E5-2680v3 12C 31,104 901.5 1,244.2 608
Education and Research 2.5GHz, Aries interconnect
Centre (SERC), Indian Cray Inc.
Institute of Science
India

261 Indian Institute of iDataPlex DX360M4, Xeon E5-2670 8C 38,016 719.2 790.7 790
Tropical Meteorology 2.600GHz, Infiniband FDR
India IBM

356 Indian Lattice Gauge TIFR - Cray XC30, Intel Xeon E5- 11,424 558.8 730.7 320
Theory Initiative (ILGTI), 2680v2 10C 2.8GHz, Aries
Tata Institute of interconnect , NVIDIA K20x
Fundamental Research Cray Inc.
(TIFR)
India

392 Indian Institute of HP Apollo 6000 Xl230/250 , Xeon E5- 22,572 524.4 1,170.1 498
Technology Delhi 2680v3 12C 2.5GHz, Infiniband FDR,
India NVIDIA Tesla K40m
HPE
Microprocessor & Computer Architecture (μpCA)
Top 5 Super Computers of India

SahasraT (Cray XC40) Aaditya (IBM/Lenovo System)

TIFR Colour Boson IIT Delhi HPC

NVIDIA’s GPU Tesla platform
CDACs Param Yuva-2

https://www.digit.in/general/top-5-supercomputers-in-india-29784.html
Microprocessor & Computer Architecture (μpCA)
Growth & Change in Trend
THANK YOU

Team MPCA
Department of Computer Science and Engineering