CS6461 Computer Architecture
Fall 2016
Morris Lancaster - Lecturer
Adapted from Professor Stephen Kaislers Notes
Lecture 10
High Performance Computing:
Multiprocessors
Introduction
So far, we have studied uniprocessors one processor,
possibly pipelined, and one memory - and vector processors,
still one computer with special functional units.
Performance can be improved through the use of multiple
processors. Multiprocessors allow multiprogramming, e.g., true
concurrent or parallel programming.
Idea: create powerful computers by connecting many
smaller ones
good news: works for timesharing (better than supercomputer)
bad news: its really hard to write good concurrent programs;
many commercial failures
Ref: Introduction to Parallel Processing: Algorithms and
Processing, by Behrooz Parhami
10/7/2017 CS61 Computer Architecture 10-2
Paradigms
Parallel Computing
Simultaneous use of multiple processors - all components of a
single architecture - to solve a task. Typically processors
identical, single user (even if machine is multi-user).
Distributed Computing
Use of a network of processors, each capable of being viewed
as a computer in its own right, to solve a problem. Processors
may be heterogeneous, multi-user, and usually individual tasks
are assigned to individual processors.
Concurrent Computing
Both of the above
10/7/2017 CS61 Computer Architecture 10-3
Types of Parallelism
10/7/2017 CS61 Computer Architecture 10-4
For A Given Problem
10/7/2017 CS61 Computer Architecture 10-5
Speedup
If we can do some computations in parallel, then we
can attain a speedup over sequential execution:
Speedup = Tsequential/Tparallel
Amdahl (Gene) defined the speedup for a parallel
processor as:
S = 1/(f + (1-f)/p)
where: p = # processors; f = fraction of
unparallelizable code
So, if f = 10%, the speedup can be no greater than
10!
With p = 10, S = 1/(0.1 + 0.9/10) ~= 5.3
With p = infinity, S = 1/(0.1 + 0.9/infinity) = 10
10/7/2017 CS61 Computer Architecture 10-6
Efficiency
Efficiency is the ratio of speedup to p: 5.3/10 ~= 53%
Ignores the possibility of new algorithms, with much smaller f
Ignores possibility that more of a program is run from higher
speed memory, such as registers, cache, main memory
often, problem is scaled with the number of processors
f is a function of size of the program which may be decreasing
serial code may take constant time, independent of size
(George Michael 80/20 vs 20/80 rule)
10/7/2017 CS61 Computer Architecture 10-7
Multiprocessor Example
Compare performance of a single Motorola 68020 with ten
68020s coupled as a multiprocessor
Consider the task of adding 100 numbers in memory
The ADD.W <ea>,Dn op takes 4 clock cycles. Thus, a single
68020 would take 400 clock cycles to add 100 numbers one at
a time
Multiprocessor works as follows:
a. All 10 uPs would add first 10 numbers each = 40 cycles
b. 5 uPs would add 10 partial sums = 4 cycles
c. 2 uPs would 4 of 5 partial sums = 4 cycles
d. one uP would add three partial sums of two = 8 cycles
Total: 56 cycles!
Performance Improvement: 7.14 (not 10 due to overhead)
10/7/2017 CS61 Computer Architecture 10-8
Limitations
Software Inertia: Billions of dollars worth of
FORTRAN and C/C++ and COBOL software exists.
Who will rewrite them?
Into what programming language?
Many programmers have experience with multicore
computers, but to the extent they program they
have no direct experience.
Who will retrain them ?
What about languages which are becoming
obsolete: Tcl, other scripting languages, etc.
10/7/2017 CS61 Computer Architecture 10-9
The Path to PetaFLOPS
10/7/2017 CS61 Computer Architecture 10-10
Michael Flynns Hardware Taxonomy
I: Instruction Stream D: Data Stream
SI: Single Instruction Stream (a)
All processors execute the same instruction in the same cycle
Instruction may be conditional
in Multiprocessors, control processor issues the instruction
MI: Multiple Instruction Stream (c)
Different processors may be simultaneously executing different instructions
SD: Single Data Stream (d)
All processors operate on the same data item (e.g., copies of it) at the
same time
MD: Multiple Data Stream (b)
Different processors may be simultaneously operating on different data
items
Multiplying a coefficient vector by a data vector (e.g., in filtering)
y[i] := c[i] x[i], 0 i < n
10/7/2017 CS61 Computer Architecture 10-11
Taxonomy Visual
10/7/2017 CS61 Computer Architecture 10-12
SISD
This model has been the subject of previous lectures.
10/7/2017 CS61 Computer Architecture 10-13
SIMD
Cray-1
This was the topic of lecture 9.
10/7/2017 CS61 Computer Architecture 10-14
SIMD
Execute one operation on multiple data streams
concurrency in time vector processing
concurrency in space array processing
CM-1 (SIMD) CM-5 (MIMD)
Thinking Machines, Inc. Connection Machines
10/7/2017 CS61 Computer Architecture 10-15
Thinking Machines CM-1
10/7/2017 CS61 Computer Architecture 10-16
MISD
10/7/2017 CS61 Computer Architecture 10-17
CMU/GE WARP
1987: 100 MFLOPS for
$300,000; about 30
times cheaper than a
Cray-1 (also 100
MFLOPS) @ $10M
Limited programming
models, however.
Example: WARP Systolic Array
Designed by H.T. Kung at CMU (now at Harvard)
Built by General Electric for me for the DARPA
Strategic Computing Program (ca 1986-1987)
10/7/2017 CS61 Computer Architecture 10-18
MIMD
10/7/2017 CS61 Computer Architecture 10-19
Processor Coupling
Tightly Coupled System
Tasks and/or processors communicate in a highly
synchronized fashion
Communicates through a common shared memory
Shared memory system
Loosely Coupled System
Tasks or processors do not communicate in a
synchronized fashion
Communicates by message passing packets
Overhead for data exchange is high
Distributed memory system
10/7/2017 CS61 Computer Architecture 10-20
Granularity of Parallelism
Coarse-grain
A task is broken into a handful of pieces, each of which is executed by
a powerful processor
Processors may be heterogeneous
Computation/communication ratio is very high
Example: BBN Butterfly
Medium-grain
Tens to few thousands of processors typically running the same code
Computation/communication ratio is often hundreds or more
Intel Paragon XP, Touchstone Series
Fine-grain
Thousands to perhaps millions of small pieces, executed by very small,
simple processors or through pipelines
Processors typically have instructions broadcasted to them
Compute/communicate ratio often near unity
Example: Thinking Machines CM-1, CM-2, CM-200
10/7/2017 CS61 Computer Architecture 10-21
Intel Paragon XP/S 140 Supercomputer
10/7/2017 CS61 Computer Architecture 10-22
Memory Architectures
Shared (Global) Memory
A Global Memory Space accessible by all processors
Processors may also have some local memory
Distributed (Local, Message-Passing) Memory
All memory units are associated with processors
To retrieve information from another processor's memory a
message must be sent there
Uniform Memory : All processors take the same time
to reach all memory locations
Non-uniform (NUMA) Memory: Processors have
varying access patterns to shared memory
10/7/2017 CS61 Computer Architecture 10-23
Shared Memory Multiprocessors
Characteristics
All processors have equally direct access to one large memory address
space
Example systems
Bus and cache-based systems: Sequent Balance, Encore Multimax
Multistage IN-based systems: Ultracomputer, Butterfly, RP3, HEP
Crossbar switch-based systems: C.mmp, Alliant FX/8
Limitations
Memory access latency; Hot spot problem
10/7/2017 CS61 Computer Architecture 10-24
Centralized Shared vs. Distributed Memory
Large caches single memory serves Pro: reduces latency of local memory
small number of processes (up to 16 or accesses
so) Con: communicating data between
More processors introduces more processors becomes more complex
contention
10/7/2017 CS61 Computer Architecture 10-25
Message Passing Multiprocessors
Characteristics
Interconnected computers
Each processor has its own memory, and communicates via message-
passing
Messages are variable-length data containers
Example systems
Tree structure: Teradata, DADO
Mesh-connected: Rediflow, Series 2010, J-Machine
Hypercube: Cosmic Cube, iPSC, NCUBE, FPS T Series, Mark III
10/7/2017 CS61 Computer Architecture 10-26
Message Passing Multiprocessors
One-to-one communication: one source, one destination
Collective communication
One-to-many: multicast, broadcast (one-to-all), scatter
Many-to-one: combine (fan-in), global combine, gather
Many-to-many: all-to-all broadcast (gossiping), scatter-gather
10/7/2017 CS61 Computer Architecture 10-27
Message Routing
10/7/2017 CS61 Computer Architecture 10-28
Non-Uniform Memory Access
All memories can be addressed by all processors, but access to a processors own
local memory is faster than access to another processors remote memory.
Looks like a distributed machine, but the interconnection network is usually custom-designed
switches and/or buses.
CC-NUMA: Cache Coherent NUMA
Exemplified by Kendall Square Research KSR1
10/7/2017 CS61 Computer Architecture 10-29
Operating System Support
Individual OS
Each CPU has its own OS
Statically allocate physical memory to each CPU
Each CPU runs its own independents OS
Share peripherals
Each CPU handles its processes system calls
Used in early multiprocessor systems
Simple to implement
Avoids concurrency issues by not sharing
10/7/2017 CS61 Computer Architecture 10-30
Operating System Support
Individual OS Issues:
Each processor has its own scheduling queue.
Each processor has its own memory partition.
Consistency is an issue with independent disk buffer caches and
potentially shared files.
10/7/2017 CS61 Computer Architecture 10-31
Operating System Support
Master-Slave Multiprocessors
OS mostly runs on a single fixed CPU.
User-level applications run on the other CPUs.
All system calls are passed to the Master CPU for
processing
Very little synchronization required
Single to implement
Single centralized scheduler to keep all processors busy
Memory can be allocated as needed to all CPUs.
Issues: Master CPU becomes the bottleneck.
10/7/2017 CS61 Computer Architecture 10-32
Operating System Support
Master-Slave Multiprocessors Issues:
Master CPU becomes the bottleneck.
10/7/2017 CS61 Computer Architecture 10-33
Operating System Support
OS kernel runs on all processors, while load and resources are
balanced between all processors.
One alternative: A single mutex (mutual exclusion object) that make
the entire kernel a large critical section; Only one CPU can be in the
kernel at a time; Only slight better than master-slave
Better alternative: Identify independent parts of the kernel and make
each of them their own critical section, which allows parallelism in the
kernel
Issues: A difficult task; Code is mostly similar to uniprocessor code;
hard part is identifying independent parts that dont interfere with each
other
10/7/2017 CS61 Computer Architecture 10-34
Interconnection Topologies
Shared Bus:
M3 wishes to communicate with S5
[1] M3 sends signals (address) on the bus that causes S5 to
respond
[2] M3 sends data to S5 or S5 sends data to M3 (determined by
the command line)
Master Device: Device that initiates and controls the
communication
Slave Device: Responding device
Multiple-master buses: Bus conflict requires bus arbitration
10/7/2017 CS61 Computer Architecture 10-35
Interconnection Topologies
Shared Bus:
All processors (and memory) are connected to a
common bus or busses
Memory access is fairly uniform, but not very scalable
A collection of signal lines that carry module-to-module
communication
Data highways connecting several digital system
elements
Can handle only one data transmission at a time
Can be easily expanded by connecting additional
processors to the shared bus, along with the
necessary bus arbitration circuitry
10/7/2017 CS61 Computer Architecture 10-36
Interconnection Topologies
Mesh Architecture:
Diameter = 2(m - 1)
In general, an n-
dimensional mesh has
diameter = d ( p1/n - 1)
Diameter can be halved by
having wrap-around
connections (=> Torus)
Ring is a 1-dimensional
mesh with wrap-around
connections
10/7/2017 CS61 Computer Architecture 10-37
Mesh-Type Interconnects
10/7/2017 CS61 Computer Architecture 10-38
Tree-Type Interconnects
10/7/2017 CS61 Computer Architecture 10-39
Example: Mesh Matrix Multiplication
10/7/2017 CS61 Computer Architecture 10-40
Interconnection Topologies
Multiport Memory:
Has multiple sets of address, data, and
control pins to allow simultaneous data
transfers to occur Memory Modules
CPU and DMA controller can transfer data
concurrently MM 1 MM 2 MM 3 MM 4
A system with more than one CPU could
handle simultaneous requests from two
different processors CPU 1
Does not scale well because of explosion of
number of busses CPU 2
Memory Module Control Logic
CPU 3
Each memory module has control logic
Resolve memory module conflicts via fixed
priority among CPUs CPU 4
Requests to read from and write to the
same memory location simultaneously
Advantages
Multiple paths -> high transfer rate
Disadvantages
Multiple copies of Memory control logic
Large number of connections
10/7/2017 CS61 Computer Architecture 10-41
Interconnection Topologies
Crossbar Switch:
Processors p and Memory banks b are
connected to routing switches like in telephone Memory modules
system MM1 MM2 MM3 MM4
Switches might have queues (combining logic),
which improve functionality but increase latency
Switch settings may be determined by message CPU1
headers or preset by controller
Connections can be packet-switched or circuit-
switched (remain connected as long as it is
needed) CPU2
Nonblocking Switch: the connection of a
processing node to a memory bank does not
block the connection of any other processing CPU3
nodes to other memory banks.
Older versions used circuit switching where a
dedicated path was created for the duration of CPU4
the communication
More recently, packet switching has been used
across the interstitial nodes
Requires p*b switches
Examples of machines that employ crossbars
include the Sun Ultra HPC 10000 and the Fujitsu
VPP500
10/7/2017 CS61 Computer Architecture 10-42
Butterfly Network
An example of blocking in omega network: one of the messages
(010 to 111 or 110 to 100) is blocked at link AB.
10/7/2017 CS61 Computer Architecture 10-43
Butterfly Network
10/7/2017 CS61 Computer Architecture 10-44
Butterfly Routing
10/7/2017 CS61 Computer Architecture 10-45
Interconnection Topologies
Omega Network (Butterfly Network)
Consists of log p stages, p is the number of inputs (processing
nodes) and also the number of outputs (memory banks)
Each stage consists of an interconnection pattern that connects
p inputs and p outputs:
Perfect shuffle(left rotation): 2i, 0 i p / 2 1
j
2i 1 p, p / 2 i p 1
Each switch has two connection modes:
Pass-thought connection: the inputs are sent straight through to the
outputs
Cross-over connection: the inputs to the switching node are crossed
over and then sent out.
Has p*(log p)/2 switching nodes: if p = 8, nodes = 12
Much better than crossbar using p*p=64 switches
The cost of such a network grows as (p log p)
10/7/2017 CS61 Computer Architecture 10-46
Interconnection Topologies
Omega Network (Butterfly Network)
Omega network has self-routing property
The path for a cell to take to reach its destination can be
determined directly from its routing tag (i.e., destination port id)
Stage k of the network looks at bit k of the tag
If bit k is 0, then send cell out upper port
If bit k is 1, then send cell out lower port
Works for every possible input port (really!)
Route from any input x to output y by selecting links determined by
successive d-ary digits of ys label.
This process is reversible; we can route from output y back to x by
following the links determined by successive digits of xs label.
This self-routing property allows for simple hardware-based routing
of cells.
yk-1xk-1 . . . x2yk-2
xk-1 . . . x1 yk-1 yk-1 yk-2xk-1 . . . x3yk-3 yk-1 yk-2 . . . xk-1y1
x1 x2
x=xk-1 . . . x0 x0 y1 xk-1
y0 y=yk-1 . . . y0
yk-1 yk-2 yk-3 xk-2
10/7/2017 CS61 Computer Architecture 10-47
Interconnection Topologies
Hypercube:
Processors are directly connected to only certain other
processors and data must traverse multiple hops to get to
additional processors
Usually distributed memory
Hardware may handle only single hops, or multiple hops
Software may mask hardware limitations
Latency is related to graph diameter, among many other
factors
Usually NUMA, nonblocking, scalable, upgradeable
Examples: Ring, Mesh, Torus, Hypercube, Binary Tree
p = 2n, n >= 0
Processors are conceptually on the corners of a n-
dimensional hypercube, and each is directly connected to
the n neighboring nodes
10/7/2017 CS61 Computer Architecture 10-48
Interconnection Topologies
10/7/2017 CS61 Computer Architecture 10-49
Interconnection Topologies
10/7/2017 CS61 Computer Architecture 10-50
Red Storm Cray Research (2003)
Red Storm Overview
Red Storm Network
Red Storm Processor Board
Growth Over Thirty Years
10/7/2017 CS61 Computer Architecture 10-55
What Have We learned Over 30 Years?
Building general-purpose parallel machines is a very
difficult task.
Proof by contradiction:
Many companies have gone bankrupt or left the parallel machine
market
Even harder is developing general parallel programming
schemes
Still an art rather than a science
10/7/2017 CS61 Computer Architecture 10-56
Additional Material
SIMD => Data Level Parallelism
