Scribd
Upload
62 views17 pages

Parallel Programming

This document provides an introduction to parallel programming, including reasons for parallel execution such as shortening execution time and permitting larger problems. It describes Amdahl's law for determining speedup from additional processors. It discusses types of parallel machines like symmetric multiprocessors (SMPs), distributed computing, and clusters. It also covers types of parallelism including process and data parallelism. The document outlines parallel programming paradigms of shared memory using techniques like OpenMP and message passing using MPI. It discusses key parallel programming concepts such as synchronization and load balancing.

Uploaded by

Yang Yi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
62 views17 pages

Parallel Programming

This document provides an introduction to parallel programming, including reasons for parallel execution such as shortening execution time and permitting larger problems. It describes Amdahl's law for determining speedup from additional processors. It discusses types of parallel machines like symmetric multiprocessors (SMPs), distributed computing, and clusters. It also covers types of parallelism including process and data parallelism. The document outlines parallel programming paradigms of shared memory using techniques like OpenMP and message passing using MPI. It discusses key parallel programming concepts such as synchronization and load balancing.

Uploaded by

Yang Yi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 17

Parallel Programming

Introduction
Parallel programming is using multiple cpus
concurrently.
Reasons for parallel execution:
1. shorten execution time
2. to permit a larger problem (memory resources)
The days of waiting for the next-generation chip to
improve your serial-code throughput are gone.

Amdahls Law
Describes the time speedup one can expect as a
function of the number of processors used and the
fraction of parallel code:
speedup = 1/(1-p+p/N)
N - number of procs
p - fraction of
parallel code

Types of Parallel Machines


Symmetric Multiprocessors (SMP) - multiple
cpus sharing memory resource, bus connection kaibab, desktop Macs

Types of Parallel Machines


Distributed computing - individual computing
elements each with their own memory, and
net work connection - Cray T3E

Types of Parallel Machines


Clusters - combine the above t wo models. SMP
nodes can be connected by net work - slikrock,
saddleback

Node

Node

Node

Node

Types of Parallelism
Process Parallelism (MPMD) - a code may
contain different segments that can be
computed concurrently. Example: ocean, land,
and ice parts of climate model, or convection
and radiation parameterizations in an
atmosphere.
Low overhead, but often limits on how many
procs can be used.

Types of Parallelism
Data Parallelism (SPMD) - the same code works
on different datastreams. For example, dividing
a global domain into subdomains - each
processor executes all the code for an individual
subdomain.
Data and process parallelism may be employed
together.

Parallel Programming
Paradigms: Shared Memory
Shared memory techniques launch threads
during execution.
Automatic Parallelizers - just turn on the
compiler switch - it finds the do loops that can
be done in parallel.
Compiler Directives - Open MP is the current
standard. User inserts c omments in code that
compiler recognizes as parallelization
instructions. Modest changes to code.
Only works with shared memory.

Parallel Programming
Paradigms: Message Passing
Can work with both distributed and shared
memory.
MPI is the standard, several packages exist:
MPICH2, lam-mpi, open-mpi.
Library calls explicitly control the parallel
behavior - extensive user rewrite of code. Code
is explicitly instructed to send and receive
messages from the other processes.
Ross will discuss in much more detail next few
weeks.
Message passing and shared memory
techniques can be used in a hybrid-mode.

Parallel Programming
Concepts
Synchronization - making sure all code gets to
a certain point before proceeding.
Load balancing - trying to keep processes from
being idle while others are computing.
Granularity - how much work is in each parallel
section.

Open MP - a Brief Intro

Tutorial: http://www.osc.edu/hpc/training/openmp/big/fsld.
002.html
OpenMP: http://www.openmp.org/

Open MP - first steps


Identify parallel do-loops. Each do loop carries
overhead so it can be helpful to have a larger
outer do-loop for parallelism.
Identify functionally parallel regions (Think
F90 case construct as an analog).
Identify shared and private data
Identify r ace conditions where shared data can
change program output unexpectedly.

Open MP - parallel do loop

Open MP - reduction

Open MP - sections

Open MP - data dependency

Open MP - run time

You might also like