CS6461 Computer Architecture
Fall 2016
Adapted from Professor Stephen H. Kaislers Slides
Lecture 9 Vector Operations
(Partially based on notes from David Patterson, UC Berkeley)
Anyone can build a fast CPU. The trick is
to build a fast computer.
- Seymour Cray -
Improving Performance
Many scientific programs compute using collections of
like numbers either integer or floating point - e.g.,
vectors
Performance can be improved if we structure hardware
to efficiently deal with such collections
Vector processors have high-level operations that work
on linear arrays of numbers, e.g., vectors
Vector instructions access memory with a known pattern
No data caches required
Single vector instruction implies a lot of work
CSCI 6461 Computer Architecture 2
Conventional Computer
Initialize I = 0
20 Read B(I)
Read C(I)
Store A(I) = B(I) + C(I)
Increment I = I + 1
If I <= 100 Go to 20
B(1) will be fetched from memory.
C(1) will be fetched from memory.
A scalar add instruction will operate
on B(1) and C(1).
A(1) will be stored back to memory
Step (1) to (4) will be repeated 100
times.
CSCI 6461 Computer Architecture 3
General Purpose Computer
General purpose computer: A(i) = B(i) * C(i) ; i =1, ... ,N
Cycle: 1 2 3 4 5 6 ... N*5
Operation
Separate B(1) B(2)
mant. / exp. C(1) C(2)
...
Multiply B(1)
mantissa C(1)
...
Add B(1)
exponents C(1)
...
Normal.
result
A(1) ...
Put
sign
A(1) ... A(N)
CSCI 6461 Computer Architecture 4
Vector Computer
A(1:100) = B(1:100) + C(1:100)
Fetch vectors of values B(I) and C(I) into memory
Use vector integer add instruction to operate on B(I), C(I) pairs
Stream of A(I) values will be stored back to memory, one value
every clock cycle
CSCI 6461 Computer Architecture 5
Vector Computer
Vector pipeline (5 sub units / segments): A = B * C
Cycle: 1 2 3 4 5 6 ... N+4
Operation
Separate B(1) B(2) B(3) B(4) B(5) B(6)
...
Mant. / Exp. C(1) C(2) C(3) C(4) C(5) C(6)
Multiply B(1) B(2) B(3) B(4) B(5)
...
mantissa C(1) C(2) C(3) C(4) C(5)
Add B(1) B(2) B(3) B(4)
...
exponents C(1) C(2) C(3) C(4)
Normal. B(3)
A(1) A(2) ...
result C(3)
Put
A(1) A(2) ... A(N)
sign
CSCI 6461 Computer Architecture 6
Basic Ideas
Vector registers: Each vector register is a fixed-
length bank holding a single vector.
Usually comprised of normal general-purpose registers and
floating-point registers.
They can provide data as input to the vector functional
units, as well as compute addresses.
Vector functional units: Fully pipelined and can start
a new operation on every clock cycle.
Vector load-store unit: loads or stores a vector to or
from memory.
Vector Length Control: A vector has a natural length
determined by the length of the vector registers.
CSCI 6461 Computer Architecture 7
Two Types of Vector Processors
Vector-Register Processors:
All vector operations (except load and store) occur in the
vector registers.
Vector counterpart of a load-store architecture
All major vector computers (Cray machines, NEC SX/2 ~
SX/5, Fujitsu VP200, etc.)
Memory-Memory Processors:
All vector operations are memory to memory.
CDC vector computers: CDC 203, CDC 205, TI ASC
All are obsolete!
CSCI 6461 Computer Architecture 8
Properties of Vector Processors
Vector instructions access memory with known pattern
Highly interleaved memory
Amortize memory latency over multiple elements
No (data) caches required! (Do use instruction cache)
Single vector instruction implies lots of work ( loop)
=> fewer instruction fetches
Vector processor
Memory
Mask-
Unit registers MASK
I/O LOAD ADD
ControlUnit (CU) STORE Vector-
registers MULT
ScalarUnit (SU)
DIV
(RISC Processor)
Vector pipelines
CSCI 6461 Computer Architecture 9
Basic Vector-Register Processor Architecture
Main Memory
FP add/subtract
FP multiply
Vector load-store
FP divide
Integer
Vector
registers Logical
8 64-element vector registers
Scalar 5 Functional Units; each unit is
registers fully pipelined,
can start a new operation on
every clock cycle
Load/store unit - fully pipelined
Scalar registers
CSCI 6461 Computer Architecture 10
Whats in a Vector Processor
A scalar processor
Scalar register file
Scalar functional units (arithmetic, load/store, etc)
A vector register file (a 2D register array)
Each register is an array of elements, e.g. 32 registers with 32 64-bit
elements per register
MVL = maximum vector length = max # of elements per register
A set of pipelined vector functional units: Integer, FP, load/store, etc
Sometimes vector and scalar units are combined (share ALUs)
Three types of addressing
Unit stride
Contiguous block of information in memory
Fastest: always possible to optimize this
Non-unit (constant) stride
Harder to optimize memory system for all possible strides
Prime number of data banks makes it easier to support different strides at full
bandwidth
Indexed (gather-scatter)
Vector equivalent of register indirect
Good for sparse arrays of data
Increases number of programs that vectorize
CSCI 6461 Computer Architecture 11
How a Vector Pipeline Works
Consider the steps involved in a floating-point addition on a
vector machine with IEEE Arithmetic hardware
The exponents of the two floating-point numbers to be added are
compared to find the number with the smallest magnitude.
The significands of the number with the smaller magnitude is
shifted so that the exponents of the two numbers agree.
The significands are added.
The result of the addition is normalized.
Checks are made to see if any floating-point exceptions occurred
during the addition, such as overflow.
Rounding occurs.
CSCI 6461 Computer Architecture 12
Cray-1 Vector Computer
CSCI 6461 Computer Architecture 13
Cray Processors
From Bottom Left:
Cray-1,
Cray-XMP,
Cray-2,
Cray-T916
Cray Research built
aestheticallly
pleasing
supercomputers.
For over two
decades they were
the fastest machines
on earth.
CSCI 6461 Computer Architecture 14
Vector Instructions
Instruction Operands Operation Comment
VADD.VV V1,V2,V3 V1=V2+V3 vector + vector
VADD.SV V1,R0,V2 V1=R0+V2 scalar + vector
VMUL.VV V1,V2,V3 V1=V2*V3 vector x vector
VMUL.SV V1,R0,V2 V1=R0*V2 scalar x vector
VLD V1,R1 V1=M[R1...R1+63] load, stride=1
VLDS V1,R1,R2 V1=M[R1R1+63*R2] load, stride=R2
VLDX V1,R1,V2 V1=M[R1+V2i,i=0..63] indexed("gather")
VST V1,R1 M[R1...R1+63]=V1 store, stride=1
VSTS V1,R1,R2 V1=M[R1...R1+63*R2] store, stride=R2
VSTX V1,R1,V2 V1=M[R1+V2i,i=0..63] indexed(scatter")
CSCI 6461 Computer Architecture 15
SAXPY: A Common Equation
32 element SAXPY: scalar SAXPY: S = aX + Y
LD F0, a
ADDI R4, Rx,#256 X,Y are vectors (of same length);
Loop: a is a scalar
LD F2, 0(Rx) One of the most common vector
MUL.D F2, F0, F2
operations found in all arithmetic
LD F4, 0(Ry)
ADD.D F4, F2, F4 systems.
SD F4, 0(Ry) All transformations in linear algebra
ADDI Rx, Rx, 8 can be expressed in this basic triad.
ADDI Ry, Ry, 8
SUB R20,R4,Rx
BNZ R20,loop
Now, 32 element SAXPY: vector
LD F0,a #load a
VLD V1,Rx #load X[0:31]
VMULD.SV V2,F0,V1 #vector mult
VLD V3,Ry #load Y[0:31]
VADDD.VV V4,V2,V3 #vector add
VST Ry,V4 #store Y[0:31]
CSCI 6461 Computer Architecture 16
Terminology
Vector Start-up Time: A measure of the latency in starting up
the vector pipeline.
The number of clock cycles required prior to the generation of the
first result.
The start-up time adds a considerable overhead for small
value of N.
The effect of start-up time is negligible for large value of N.
To maintain an initiation rate of one word fetched/store per
clock, the memory must be able to meet this rate.
Usually done by interleaving memory in banks.
CSCI 6461 Computer Architecture 17
Issues
What to do when the application vector length is not exactly
maximum vector length (MVL)?
Vector-length (VL) register controls the length of any vector
operation, including a vector load or store
Set it before performing any vector operation
VADD.VV with VL=10 is equivalent to
for (i=0; i<10; i++)
V1[i] = V2[i]+V3[i]
VL can be anything from 0 to MVL
CSCI 6461 Computer Architecture 18
Issues
Problem: Vector registers have finite length
Solution: Break loops into pieces that fit in registers,
Stripmining
Vector Length modulo VL /= 0!!
So, do short piece first, then do rest with length VL
EX: Suppose VL = 64. We have a vector that is 264, which
is mod 8.
So, process a vector length 8, then four vectors of length
64.
Problem: All computations have some scalar
components, e.g., non-vectorizable
Solution: Separate scale from vector computations
(by hand; but maybe automatically)
CSCI 6461 Computer Architecture 19
Ex: Vector Code
Note: Fast processing rates do not always translate directly into
Fast processing of loops.
CSCI 6461 Computer Architecture 20
Assessing Performance
Pipe(line)length p: Number of stages in pipeline = N
segments
One result per cycle (if pipe is full)
Speed-up:
Serial computation: N*p cycles
Vector computation: N + p - 1 cycles
Speed-up: S = (N * p) / (N + p - 1)
N >> p S ~ p
Problems:
N~ p
No recursive references: A(i) = A(i-1) + C(i)
CSCI 6461 Computer Architecture 21
Characteristics of Vectorizable Code - I
Vectorization can only be done within a DO/FOR
loop; it must be the innermost loop.
It is crucial to ensure that there are sufficient
iterations in the DO loop to offset the start-up time
overhead.
Put as much work as possible into a vectorizable
statement to provide more opportunities for
concurrent operations.
There is a limit to vectorization because a compiler
may not vectorize the code if it is too complicated.
Exercise: How do you vectorize a WHILE loop??
CSCI 6461 Computer Architecture 22
Characteristics of Vectorizable Code - II
The existence of certain operations in the DO loop may
prevent the compiler from converting the entire, or part of
the DO loop for vector processing:
vectorization inhibitors include subroutine calls, recursion,
references to external functions, and any input/output statements
(which are actually system calls)
These types of vector inhibitors can be removed by:
expanding the function
in-lining subroutines at the point of reference.
CSCI 6461 Computer Architecture 23
Vector Code Example
Vector Processing Example:
/* Multiply a[m][k] * b[k][n] to get c[m][n] */
for (i = 1; i < m; i++)
{
for (j =1; j < n; j++)
{
sum = 0;
for (t =1; t <k; t++)
{
sum = sum + a[i][t] * b[t][j]; //// This is a dependency!!!
}
c[i][j] = sum;
}
}
CSCI 6461 Computer Architecture 24
Optimized Vector Code
/* Multiply a[m][k] * b[k][n] to get c[m][n] */
for (i = 1; i < m; i++)
{
for (j = 1; j < n; j += 32) /* Step j by 32 at a time. */
{
sum[0:31] = 0; /* Initialize a vector register to zeros. */
for (t = 1; t < k; t++)
{
a_scalar = a[i][t];
b_vector[0:31] = b[t][j:j+31];
/* Do a vector-scalar multiply. */
prod[0:31] = b_vector[0:31] * a_scalar; It's actually better to
/* Vector-vector add into results. */ interchange the i and
sum[0:31] += prod[0:31];
j loops, so that you
}
/* Unit-stride store of vector of results. */ only change
c[i][j:j+31] = sum[0:31]; vector length once
} during the whole
} matrix multiply
CSCI 6461 Computer Architecture 25
Vector Stride
Suppose adjacent elements of the vector are not sequential in
memory
do 10 i = 1,100
do 10 j = 1,100
A(i,j) = 0.0
do 10 k = 1,100
10 A(i,j) = A(i,j)+B(i,k)*C(k,j)
Either B or C accesses not adjacent (800 bytes between)
stride: distance separating elements that are to be merged into
a single vector (caches do unit stride)
=> LVWS (load vector with stride) instruction
Strides => can cause bank conflicts
(e.g., stride = 32 and 16 banks)
CSCI 6461 Computer Architecture 26
Vector Chaining
Suppose:
MULV V1,V2,V3
ADDV V4,V1,V5
chaining: vector register (V1) is not as a single entity
but as a group of individual registers, then pipeline
forwarding can work on individual elements of a
vector
Flexible chaining: allow vector to chain to any other
active vector operation => more read/write ports, e.g.
pass the result from one vector operation to another
vector operation
As long as enough HW, increases convoy size
CSCI 6461 Computer Architecture 27
Vector Register Bypassing
CSCI 6461 Computer Architecture 28
Vector Conditional Execution
CSCI 6461 Computer Architecture 29
Two Approaches
CSCI 6461 Computer Architecture 30
Vectors w/ Sparse Matrices
Suppose:
do 100 i = 1,n
100 A(K(i)) = A(K(i)) + C(M(i))
gather (LVI) operation takes an index vector and fetches data
from each address in the index vector
This produces a dense vector in the vector registers
After these elements are operated on in dense form, the sparse
vector can be stored in expanded form by a scatter store
(SVI), using the same index vector
Can't be figured out by a compiler since it can't know elements
distinct, no dependencies
Use CVI to create index 0, 1xm, 2xm, ..., 63xm
CSCI 6461 Computer Architecture 31
Gather Example
CSCI 6461 Computer Architecture 32
Vector Issues
Pitfall: Concentrating on peak performance and ignoring
start-up overhead:
NV (length faster than scalar) > 100!
Pitfall: Increasing vector performance, without
comparable increases in scalar performance (Amdahl's
Law)
problems of Cray competitor (ETA)
Pitfall: Good processor vector performance without
providing good memory bandwidth
MMX?
CSCI 6461 Computer Architecture 33
Some Previous Vector Processors
CSCI 6461 Computer Architecture 34
Vector Memory-Memory vs Register Machines
Vector memory-memory instructions hold all vector operands
in main memory
The first vector machines, CDC Star-100 (73) and TI ASC
(71), were memory-memory machines
Cray-1 (76) was first vector register machine
CSCI 6461 Computer Architecture 35
Vector Memory-Memory vs Register Machines
Vector memory-memory architectures (VMMA) require greater
main memory bandwidth, why?
All operands must be read in and out of memory
VMMAs make if difficult to overlap execution of multiple vector
operations, why?
Must check dependencies on memory addresses
VMMAs incur greater startup latency
Scalar code was faster on CDC Star-100 for vectors < 100
elements
For Cray-1, vector/scalar breakeven point was around 2
elements
Apart from CDC follow-ons (Cyber-205, ETA-10) all major
vector machines since Cray-1 have had vector register
architectures
CSCI 6461 Computer Architecture 36
CSCI 6461 Computer Architecture
The Cell Processor
Observed clock speed: > 4 GHz
37
Peak performance (single precision): > 256 GFlops
Peak performance (double precision): >26 GFlops
Local storage size per SPU: 256KB
Total number of transistors: 234M
The Cell Processor
Sony Playstation 3
Partnership between Sony,
Toshiba, IBM
Power PC-based main core (PPE)
Multiple SPEs
On die memory controller
Inter-core transport bus
High speed IO
Clocked at 3-4ghz
256GFLOPS Single Precision @
4ghz
Offload a large amount of work
onto compiler / software.
CSCI 6461 Computer Architecture 38
Cell Processor Die Layout
CSCI 6461 Computer Architecture 39
Power Processing Element (PPE)
PowerPC instruction set with AltiVec VMX instructions
Slow, but power-efficient
Used for general purpose computing and controlling
SPEs
Simultaneous Multithreading
Separate 32 KB L1 Caches for instructions and data
Unified 512 KB L2 Cache
Two issue in-order instruction fetch
Conspicuous lack of instruction window
PPEs and SPEs use different instruction sets.
CSCI 6461 Computer Architecture 40
Synergistic Processing Element (SPE)
SPEs are vector processors:
Not efficient for general-purpose
computation.
Meant to be used in parallel
(7 on PS3 implementation)
Instructions based on VMX
In-order execution w/ dual issue
Modified for 128 registers
Instructions assumed to be 4x 32 bits
128 registers (each 128 bits wide)
Vector logic
8 single precision operations per cycle
Significant performance hit for double
precision
CSCI 6461 Computer Architecture 41
SPE Local Storage
On chip local storage (256KB)
NOT a cache
Completely private to each SPE
Directly addressable by software
Software controlled DMA to and from main memory
Request queue handles 16 simultaneous requests
Up to 16 KB transfer each
Priority: DMA, L/S, Fetch
Fetch / execute parallelism
CSCI 6461 Computer Architecture 42
SPE Control Logic/Pipeline
Little ILP, and thus little control
logic faster execution
No hardware branch prediction
Software branch prediction
Loop unrolling
18 cycle penalty
Simple commit unit
no reorder buffer or other
complexities
Same execution unit for FP/int
Instruction Scheduling a HUGE
problem
Done primarily in software
IBM predicted 80-90% usage
ideally
CSCI 6461 Computer Architecture 43
Modern Vector Supercomputer
65nm CMOS technology
Vector unit (3.2 GHz)
8 foreground VRegs + 64 background
VRegs (256x64-bit elements/VReg)
64-bit functional units: 2 multiply, 2 add, 1
divide/sqrt, 1 logical, 1 mask unit
8 lanes (32+ FLOPS/cycle, 100+
GFLOPS peak per CPU)
1 load or store unit (8 x 8-byte
accesses/cycle)
Scalar unit (1.6 GHz)
4-way superscalar with out-of-order and
speculative execution
64KB I-cache and 64KB data cache
Memory system provides 256GB/s DRAM bandwidth per CPU
Up to 16 CPUs and up to 1TB DRAM form shared-memory node
total of 4TB/s bandwidth to shared DRAM memory
Up to 512 nodes connected via 128GB/s network links (message passing
between nodes)
CSCI 6461 Computer Architecture 44
Vector Advantages
Easy to get high performance: N operations
are independent
use same functional unit
access disjoint registers
access registers in same order as previous instructions
access contiguous memory words or known pattern
can exploit large memory bandwidth
hide memory latency (and any other latency)
Scalable: (get higher performance by adding HW resources)
Compact: Describe N operations with 1 short instruction
Predictable: performance vs. statistical performance (cache)
Multimedia ready: N * 64b, 2N * 32b, 4N * 16b, 8N * 8b
Mature, developed compiler technology
CSCI 6461 Computer Architecture 45
Vector Disadvantages
Vector Disadvantage: Out of Fashion?
Hard to say. Many irregular loop structures seem to still
be hard to vectorize automatically.
Not as fast with scalar instructions
Complexity of the multi-ported Vector Register File
Difficulties implementing precise exceptions
High price of on-chip vector memory systems
Increased code complexity
CSCI 6461 Computer Architecture 46
The
Last
(Vector)
Samurais
CSCI 6461 Computer Architecture 47
