Scribd
Upload
13 views

01 - Parallel Programming

The document discusses parallel computing and different approaches for achieving parallelism including using multiple interconnected computers. It describes classifications of parallel computers and software tools for parallel programming on clusters.

Uploaded by

Dd d
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
13 views

01 - Parallel Programming

The document discusses parallel computing and different approaches for achieving parallelism including using multiple interconnected computers. It describes classifications of parallel computers and software tools for parallel programming on clusters.

Uploaded by

Dd d
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 18

Parallel Programming

Parallel Computers

Parallel Programming - Lecture 1


The Demand for Computational Speed

 Continual demand for greater computational


speed from a computer system than is currently
possible

 Areas requiring great computational speed include


numerical modeling and simulation of
scientific and engineering problems.

 Computations must be completed within a


"reasonable" time period.

Parallel Programming - Lecture 1 2


Grand Challenge Problems

 One that cannot be solved in a reasonable


amount of time with today’s computers.
Obviously, an execution time of 10 years is always
unreasonable.

 Examples
 Modeling large DNA structures.

 Global weather forecasting.

 Modeling motion of astronomical bodies.

Parallel Programming - Lecture 1 3


Global Weather Forecasting Example

 Atmosphere modeled by dividing it into


3-dimensional cells.

 Calculations of each cell repeated many times to


model passage of time.

Parallel Programming - Lecture 1 4


Global Weather Forecasting Example

 Suppose whole global atmosphere divided into


cells of size 1 mile  1 mile  1 mile to a height of
10 miles (10 cells high) - about 5  108 cells.
 Suppose each calculation requires 200 floating
point operations. In one time step, 1011 floating
point operations necessary.

Parallel Programming - Lecture 1 5


Global Weather Forecasting Example

 To forecast the weather over 7 days using


1-minute intervals, a computer operating at
1Gflops (109 floating point operations/s) takes 106
seconds or over 10 days.
 To perform calculation in 5 minutes requires
computer operating at 3.4 Tflops (3.4  1012
floating point operations/sec).

Parallel Programming - Lecture 1 6


Modeling Motion of Astronomical Bodies

 Each body attracted to each other body by


gravitational forces. Movement of each body
predicted by calculating total force on each body.

 With N bodies, N - 1 forces to calculate for each


body, or approx. N2 calculations. (N log2 N for an
efficient approx. algorithm.)

 After determining new positions of bodies,


calculations repeated.

Parallel Programming - Lecture 1 7


Modeling Motion of Astronomical Bodies

 A galaxy might have, say, 1011 stars.

 Even if each calculation done in 1 ms, it takes


109 years for one iteration using N2 algorithm and
almost a year for one iteration using
an efficient N log2 N approximate algorithm.

Parallel Programming - Lecture 1 8


Parallel Computing

 Using more than one computer, or a computer


with more than one processor, to solve a
problem.

Motives
 Usually faster computation - very simple idea -
that n computers operating simultaneously can
achieve the result n times faster.
 Other motives include: fault tolerance, larger
amount of memory available, ...
Parallel Programming - Lecture 1 9
Flynn’s Classifications

Flynn (1966) created a classification for computers


based upon instruction streams and data streams:

 Single instruction stream - single data stream


(SISD) computer

Single processor computer - single stream of


instructions generated from program. Instructions
operate upon a single stream of data items.

Parallel Programming - Lecture 1 10


Multiple Instruction Stream - Multiple
Data Stream (MIMD) Computer
General-purpose multiprocessor system - each
processor has a separate program and one
instruction stream is generated from each program
for each processor. Each instruction operates upon
different data.

Both the shared memory and the message-passing


multiprocessors so far described are in the MIMD
classification.

Parallel Programming - Lecture 1 11


Single Instruction Stream - Multiple
Data Stream (SIMD) Computer

 A specially designed computer - a single


instruction stream from a single program, but
multiple data streams exist. Instructions from
program broadcast to more than one processor.
Each processor executes same instruction in
synchronism, but using different data.

 Developed because a number of important


applications that mostly operate upon arrays of
data.

Parallel Programming - Lecture 1 12


Multiple Program Multiple Data (MPMD)
Structure

Within the MIMD classification, each processor will


have its own program to execute:

Program Program

Instructions Instructions

Processor Processor

Data Data

Parallel Programming - Lecture 1 13


Single Program Multiple Data (SPMD)
Structure

Single source program written and each processor


executes its personal copy of this program, although
independently and not in synchronism.

Source program can be constructed so that parts of


the program are executed by certain computers and
not others depending upon the identity of the
computer.

Parallel Programming - Lecture 1 14


Interconnected Computers as a
Computing Platform
 A network of computers became a very attractive
alternative to expensive supercomputers and parallel
computer systems for high-performance computing in
early 1990’s.

 Several early projects:


 Berkeley NOW (network of workstations) project.

 NASA Beowulf project.

Parallel Programming - Lecture 1 15


Beowulf Clusters*

 A group of interconnected computers achieving


high performance with low cost.

 Typically using interconnects - high speed


Ethernet, and Linux OS.

* Beowulf comes from name given by NASA


Goddard Space Flight Center cluster project.

Parallel Programming - Lecture 1 16


Key advantages
 Very high performance workstations and PCs
readily available at low cost.

 The latest processors can easily be incorporated


into the system as they become available.

 Existing software can be used or modified.

Parallel Programming - Lecture 1 17


Software Tools for Clusters

 Based upon Message Passing Parallel


Programming:
 Parallel Virtual Machine (PVM) - developed in

late 1980’s. Became very popular.


 Message-Passing Interface (MPI) - standard

defined in 1990s.
 Both provide a set of user-level libraries for

message passing. Use with regular


programming languages (C, C++, ...).

Parallel Programming - Lecture 1 18

You might also like