An Introduction to Parallel Programming
Lecture 1
Why Parallel Computing?
1
INTRODUCTION
WEEK 01
Course Objectives
Learn how to program parallel processors and
systems
Learn how to think in parallel and write correct parallel
programs
Achieve performance and scalability through
understanding of architecture and software mapping
Significant hands-on programming experience
Develop real applications on real hardware
Discuss the current parallel computing context
What are the drivers that make this course timely
Contemporary programming models and architectures, and
where is the field going
3
Why is this Course Important?
Multi-core and many-core era is here to stay
Why? Technology Trends
Many programmers will be developing parallel software
But still not everyone is trained in parallel programming
Learn how to put all these vast machine resources to the best
use!
Useful for
Joining the industry
Graduate school
Our focus
Teach core concepts
Use common programming models
Discuss broader spectrum of parallel computing
4
Roadmap
Why we need ever-increasing performance.
Why we’re building parallel systems.
Why we need to write parallel programs.
What we’ll be doing.
Concurrent, parallel, distributed!
5
Parallel and Distributed
Computing
Parallel computing (processing):
the use of two or more processors (computers), usually
within a single system, working simultaneously to solve a
single problem.
Distributed computing (processing):
any computing that involves multiple computers
remote from each other that each have a role in a
computation problem or information processing.
Parallel programming:
the human process of developing programs that express what
computations should be executed in parallel.
6
Parallel Computing
To be run using multiple CPUs
◦A problem is broken into discrete parts that can be solved
concurrently
◦Each part is further broken down to a series of
instructions
Instructions from each part execute simultaneously on
different CPUs
Page 7
Parallel Computing Example
Page 8
Compute Resources
The compute resources can include:
◦A single computer with multiple processors/cores
◦An arbitrary number of computers connected by
a network
◦A combination of both
Page 10
Why we need ever-increasing
performance
Computational power is increasing, but so are our
computation problems and needs.
Problems we never dreamed of have been
solved because of past increases, such as
decoding the human genome.
More complex problems are still waiting to be
solved.
10
Climate modeling
National Oceanic and Atmospheric Administration
(NOAA) has more than 20PB of data and processes
80TB/day
11
Climate modeling
One
Another processor
processor computes
computes this part
this part in
parallel
Processors in adjacent blocks in the grid communicate their result.
12
Data analysis
CERN’s Large Hadron Collider (LHC) produces about 15PB per year
High-energy physics workflows involve a range of both data-intensive and
compute-intensive activities.
The collision data from the detectors on the LHC needs to be filtered to select a
few thousand interesting collisions from as many as one billion that may take
place each second.
The WLCG produces a massive sample of billions of simulated beam crossings,
trying to predict the response of the detector and compare it to known physics
processes and potential new physics signals.
13
Drug discovery
Computational drug discovery and design (CDDD) based on HPC is a
combination of pharmaceutical chemistry, computational chemistry, and
biology using supercomputers, and has become a critical technology in
drug research and development.
14
Why Parallel Computing?
The Real World is Massively Parallel:
◦Parallel computing attempts to emulate the natural world
◦Many complex, interrelated events happening at the same time, yet within a
sequence.
Page 12
Why Parallel Computing?
Page 13
Why Parallel Computing?
To solve larger, more complex Problems:
numerical simulations of complex systems and
"Grand Challenge Problems" such as:
◦weather and climate forecasting
◦chemical and nuclear reactions
◦geological, seismic activity
◦mechanical devices (spacecraft )
◦electronic circuits
◦manufacturing processes
Why Parallel Computing?
Example applications include:
◦parallel databases, data mining
◦web search engines, web based business services
◦computer-aided diagnosis in medicine
◦management of national and multi-national corporations
◦advanced graphics and virtual reality, particularly in the
entertainment industry
Page 16
Why Parallel Computing?
◦ To save time
◦ To solve larger problems
◦ To provide concurrency
Page 17
Why Parallel Computing?
Parallel computing is an attempt to maximize the
infinite but seemingly limited commodity called
time!
Page 18
Who and What?
Top500.org provides statistics on parallel computing
users.
The Future?
During the past 20 years, the trends indicated by ever faster
networks, distributed systems, and multi-processor
architectures clearly show that parallelism is the future of
computing.
In this same time period, there has been a greater than
500,000x increase in supercomputer performance, with no
end currently in sight.
The race is already on for Exascale Computing!
Exaflop = 1018 calculations per second
Page 21
Towards parallel hardware
23
Why we’re building parallel
systems
Up to now, performance increases have been
attributable to increasing density of transistors.
But there are inherent
problems.
24
A little physics lesson
Smaller transistors = faster processors.
Faster processors = increased power
consumption.
Increased power consumption =
increased heat.
Increased heat = unreliable processors.
25
Evolution of processors in the
last 50 years
Evolution of processors in the last 50 years
26
How small is 5nm?
https://www.tsmc.com/english/dedicatedFoundry/technology/logic/l_5nm
27
An intelligent solution
Instead of designing and building faster
microprocessors, put multiple processors on a
single integrated circuit.
Move away from single-core systems to
multicore processors.
Introducing parallelism!!!
28
Thank You
