Digital Design & Computer Arch.

Lecture 19: SIMD Processors

Prof. Onur Mutlu

ETH Zürich
Spring 2020
7 May 2020
We Are Almost Done With This…
◼ Single-cycle Microarchitectures

◼ Multi-cycle Microarchitectures

◼ Pipelining

◼ Issues in Pipelining: Control & Data Dependence Handling,

State Maintenance and Recovery, …

◼ Out-of-Order Execution

◼ Other Execution Paradigms

2
Approaches to (Instruction-Level) Concurrency
◼ Pipelining
◼ Out-of-order execution
◼ Dataflow (at the ISA level)
◼ Superscalar Execution
◼ VLIW
◼ Systolic Arrays
◼ Decoupled Access Execute
◼ Fine-Grained Multithreading
◼ SIMD Processing (Vector and array processors, GPUs)

3
Readings for this Week
◼ Required
◼ Lindholm et al., "NVIDIA Tesla: A Unified Graphics and
Computing Architecture," IEEE Micro 2008.

◼ Recommended
❑ Peleg and Weiser, “MMX Technology Extension to the Intel
Architecture,” IEEE Micro 1996.

4
Announcement
◼ Late submission of lab reports in Moodle
❑ Open until June 20, 2020, 11:59pm (cutoff date -- hard
deadline)
❑ You can submit any past lab report, which you have not
submitted before its deadline
❑ It is NOT allowed to re-submit anything (lab reports, extra
assignments, etc.) that you had already submitted via other
Moodle assignments
❑ We will grade your reports, but late submission has a
penalization of 1 point, that is, the highest possible score per
lab report will be 2 points

5
Exploiting Data Parallelism:
SIMD Processors and GPUs
 SIMD Processing:
Exploiting Regular (Data) Parallelism
Flynn’s Taxonomy of Computers
◼ Mike Flynn, “Very High-Speed Computing Systems,” Proc.
of IEEE, 1966

◼ SISD: Single instruction operates on single data element

◼ SIMD: Single instruction operates on multiple data elements
❑ Array processor
❑ Vector processor
◼ MISD: Multiple instructions operate on single data element
❑ Closest form: systolic array processor, streaming processor
◼ MIMD: Multiple instructions operate on multiple data
elements (multiple instruction streams)
❑ Multiprocessor
❑ Multithreaded processor
8
Data Parallelism
◼ Concurrency arises from performing the same operation on
different pieces of data
❑ Single instruction multiple data (SIMD)
❑ E.g., dot product of two vectors

◼ Contrast with data flow

❑ Concurrency arises from executing different operations in parallel (in
a data driven manner)

◼ Contrast with thread (“control”) parallelism

❑ Concurrency arises from executing different threads of control in
parallel

◼ SIMD exploits operation-level parallelism on different data

❑ Same operation concurrently applied to different pieces of data
❑ A form of ILP where instruction happens to be the same across data
9
SIMD Processing
◼ Single instruction operates on multiple data elements
❑ In time or in space
◼ Multiple processing elements

◼ Time-space duality

❑ Array processor: Instruction operates on multiple data

elements at the same time using different spaces

❑ Vector processor: Instruction operates on multiple data

elements in consecutive time steps using the same space

10
Array vs. Vector Processors
ARRAY PROCESSOR VECTOR PROCESSOR

Instruction Stream Same op @ same time

Different ops @ time
LD VR  A[3:0] LD0 LD1 LD2 LD3 LD0
ADD VR  VR, 1 AD0 AD1 AD2 AD3 LD1 AD0
MUL VR  VR, 2
ST A[3:0]  VR MU0 MU1 MU2 MU3 LD2 AD1 MU0
ST0 ST1 ST2 ST3 LD3 AD2 MU1 ST0
Different ops @ same space AD3 MU2 ST1
MU3 ST2
Time Same op @ space ST3

Space Space

11
SIMD Array Processing vs. VLIW
◼ VLIW: Multiple independent operations packed together by the compiler

12
SIMD Array Processing vs. VLIW
◼ Array processor: Single operation on multiple (different) data elements

13
Vector Processors (I)
◼ A vector is a one-dimensional array of numbers
◼ Many scientific/commercial programs use vectors
for (i = 0; i<=49; i++)
C[i] = (A[i] + B[i]) / 2

◼ A vector processor is one whose instructions operate on

vectors rather than scalar (single data) values
◼ Basic requirements
❑ Need to load/store vectors → vector registers (contain vectors)
❑ Need to operate on vectors of different lengths → vector length
register (VLEN)
❑ Elements of a vector might be stored apart from each other in
memory → vector stride register (VSTR)
◼ Stride: distance in memory between two elements of a vector

14
Vector Processors (II)
◼ A vector instruction performs an operation on each element
in consecutive cycles
❑ Vector functional units are pipelined
❑ Each pipeline stage operates on a different data element

◼ Vector instructions allow deeper pipelines

❑ No intra-vector dependencies → no hardware interlocking
needed within a vector
❑ No control flow within a vector
❑ Known stride allows easy address calculation for all vector
elements
◼ Enables prefetching of vectors into registers/cache/memory

15
Vector Processor Advantages
+ No dependencies within a vector
❑ Pipelining & parallelization work really well
❑ Can have very deep pipelines, no dependencies!

+ Each instruction generates a lot of work

❑ Reduces instruction fetch bandwidth requirements

+ Highly regular memory access pattern

+ No need to explicitly code loops

❑ Fewer branches in the instruction sequence

16
Vector Processor Disadvantages
-- Works (only) if parallelism is regular (data/SIMD parallelism)
++ Vector operations
-- Very inefficient if parallelism is irregular
-- How about searching for a key in a linked list?

Fisher, “Very Long Instruction Word architectures and the ELI-512,” ISCA 1983. 17
Vector Processor Limitations
-- Memory (bandwidth) can easily become a bottleneck,
especially if
1. compute/memory operation balance is not maintained
2. data is not mapped appropriately to memory banks

18
Vector Processing in More Depth
Vector Registers
◼ Each vector data register holds N M-bit values
◼ Vector control registers: VLEN, VSTR, VMASK
◼ Maximum VLEN can be N
❑ Maximum number of elements stored in a vector register
◼ Vector Mask Register (VMASK)
❑ Indicates which elements of vector to operate on

❑ Set by vector test instructions

◼ e.g., VMASK[i] = (Vk[i] == 0)

M-bit wide M-bit wide
V0,0 V1,0
V0,1 V1,1

V0,N-1 V1,N-1

20
 Vector Functional Units
◼ Use a deep pipeline to execute
element operations
V V V
→ fast clock cycle
1 2 3

◼ Control of deep pipeline is

simple because elements in
vector are independent

Six stage multiply pipeline

V1 * V2 → V3

Slide credit: Krste Asanovic 21

Vector Machine Organization (CRAY-1)
◼ CRAY-1
◼ Russell, “The CRAY-1
computer system,”
CACM 1978.

◼ Scalar and vector modes

◼ 8 64-element vector
registers
◼ 64 bits per element
◼ 16 memory banks
◼ 8 64-bit scalar registers
◼ 8 24-bit address registers

22
CRAY X-MP-28 @ ETH (CAB, E Floor)

23
CRAY X-MP System Organization

Cray Research Inc., “The

CRAY X-MP Series of
Computer Systems,” 1985
24
CRAY X-MP Design Detail

Cray Research Inc., “The

CRAY X-MP Series of
Computer Systems,” 1985
25
CRAY X-MP CPU Functional Units

Cray Research Inc., “The

CRAY X-MP Series of
Computer Systems,” 1985
26
CRAY X-MP System Configuration

Cray Research Inc., “The

CRAY X-MP Series of
Computer Systems,” 1985
27
Seymour Cray, the Father of Supercomputers

"If you were plowing a field, which would you

rather use: Two strong oxen or 1024 chickens?"

© amityrebecca / Pinterest. https://www.pinterest.ch/pin/473018767088408061/

© Scott Sinklier / Corbis. http://america.aljazeera.com/articles/2015/2/20/the-short-brutal-life-of-male-chickens.html

28
Vector Machine Organization (CRAY-1)
◼ CRAY-1
◼ Russell, “The CRAY-1
computer system,”
CACM 1978.

◼ Scalar and vector modes

◼ 8 64-element vector
registers
◼ 64 bits per element
◼ 16 memory banks
◼ 8 64-bit scalar registers
◼ 8 24-bit address registers

29
Loading/Storing Vectors from/to Memory
◼ Requires loading/storing multiple elements

◼ Elements separated from each other by a constant distance

(stride)
❑ Assume stride = 1 for now

◼ Elements can be loaded in consecutive cycles if we can

start the load of one element per cycle
❑ Can sustain a throughput of one element per cycle

◼ Question: How do we achieve this with a memory that

takes more than 1 cycle to access?
◼ Answer: Bank the memory; interleave the elements across
banks
30
 Memory Banking
◼ Memory is divided into banks that can be accessed independently;
banks share address and data buses (to minimize pin cost)
◼ Can start and complete one bank access per cycle
◼ Can sustain N parallel accesses if all N go to different banks

Bank Bank Bank Bank

0 1 2 15

MDR MAR MDR MAR MDR MAR MDR MAR

Data bus

Address bus

CPU
Picture credit: Derek Chiou 31
 Vector Memory System
◼ Next address = Previous address + Stride
◼ If (stride == 1) && (consecutive elements interleaved
across banks) && (number of banks >= bank latency), then
❑ we can sustain 1 element/cycle throughput
Base Stride
Vector Registers

Address
Generator +

0 1 2 3 4 5 6 7 8 9 A B C D E F
Memory Banks
Picture credit: Krste Asanovic 32
Scalar Code Example: Element-Wise Avg.
◼ For I = 0 to 49
❑ C[i] = (A[i] + B[i]) / 2

◼ Scalar code (instruction and its latency)

MOVI R0 = 50 1
MOVA R1 = A 1 304 dynamic instructions
MOVA R2 = B 1
MOVA R3 = C 1
X: LD R4 = MEM[R1++] 11 ;autoincrement addressing
LD R5 = MEM[R2++] 11
ADD R6 = R4 + R5 4
SHFR R7 = R6 >> 1 1
ST MEM[R3++] = R7 11
DECBNZ R0, X 2 ;decrement and branch if NZ
33
Scalar Code Execution Time (In Order)
◼ Scalar execution time on an in-order processor with 1 bank
❑ First two loads in the loop cannot be pipelined: 2*11 cycles
❑ 4 + 50*40 = 2004 cycles

◼ Scalar execution time on an in-order processor with 16

banks (word-interleaved: consecutive words are stored in
consecutive banks)
❑ First two loads in the loop can be pipelined
❑ 4 + 50*30 = 1504 cycles

◼ Why 16 banks?
❑ 11-cycle memory access latency
❑ Having 16 (>11) banks ensures there are enough banks to
overlap enough memory operations to cover memory latency
34
Vectorizable Loops
◼ A loop is vectorizable if each iteration is independent of any
other

◼ For I = 0 to 49
❑ C[i] = (A[i] + B[i]) / 2
◼ Vectorized loop (each instruction and its latency):
MOVI VLEN = 50 1
7 dynamic instructions
MOVI VSTR = 1 1
VLD V0 = A 11 + VLEN – 1
VLD V1 = B 11 + VLEN – 1
VADD V2 = V0 + V1 4 + VLEN – 1
VSHFR V3 = V2 >> 1 1 + VLEN – 1
VST C = V3 11 + VLEN – 1
35
Basic Vector Code Performance
◼ Assume no chaining (no vector data forwarding)
❑ i.e., output of a vector functional unit cannot be used as the
direct input of another
❑ The entire vector register needs to be ready before any
element of it can be used as part of another operation
◼ One memory port (one address generator)
◼ 16 memory banks (word-interleaved)

1 1 11 49 11 49 4 49 1 49 11 49

V0 = A[0..49] V1 = B[0..49] ADD SHIFT STORE

◼ 285 cycles
36
 Vector Chaining
◼ Vector chaining: Data forwarding from one vector
functional unit to another

V V V V V
LV v1 1 2 3 4 5
MULV v3,v1,v2
ADDV v5, v3, v4

Chain Chain

Load
Unit
Mult. Add

Memory

Slide credit: Krste Asanovic 37

Vector Code Performance - Chaining
◼ Vector chaining: Data forwarding from one vector
functional unit to another
1 1 11 49 11 49

Strict assumption:
Each memory bank
4 49 has a single port
(memory bandwidth
bottleneck)
These two VLDs cannot be 1 49
pipelined. WHY?

11 49

VLD and VST cannot be

◼ 182 cycles pipelined. WHY?
38
Vector Code Performance – Multiple Memory Ports
◼ Chaining and 2 load ports, 1 store port in each bank
1 1 11 49

1 11 49

4 49

1 49

11 49
◼ 79 cycles
◼ 19X perf. improvement!
39
Questions (I)
◼ What if # data elements > # elements in a vector register?
❑ Idea: Break loops so that each iteration operates on #
elements in a vector register
◼ E.g., 527 data elements, 64-element VREGs
◼ 8 iterations where VLEN = 64
◼ 1 iteration where VLEN = 15 (need to change value of VLEN)
❑ Called vector stripmining

40
 (Vector) Stripmining

Source: https://en.wikipedia.org/wiki/Surface_mining 41
Questions (II)
◼ What if vector data is not stored in a strided fashion in
memory? (irregular memory access to a vector)
❑ Idea: Use indirection to combine/pack elements into vector
registers
❑ Called scatter/gather operations

42
Gather/Scatter Operations

Want to vectorize loops with indirect accesses:

for (i=0; i<N; i++)
A[i] = B[i] + C[D[i]]

Indexed load instruction (Gather)

LV vD, rD # Load indices in D vector
LVI vC, rC, vD # Load indirect from rC base
LV vB, rB # Load B vector
ADDV.D vA,vB,vC # Do add
SV vA, rA # Store result

43
Gather/Scatter Operations
◼ Gather/scatter operations often implemented in hardware
to handle sparse vectors (matrices)
◼ Vector loads and stores use an index vector which is added
to the base register to generate the addresses

◼ Scatter example
Index Vector Data Vector (to Store) Stored Vector (in Memory)

0 3.14 Base+0 3.14

2 6.5 Base+1 X
6 71.2 Base+2 6.5
7 2.71 Base+3 X
Base+4 X
Base+5 X
Base+6 71.2
Base+7 2.71
44
 Conditional Operations in a Loop
◼ What if some operations should not be executed on a vector
(based on a dynamically-determined condition)?
loop: for (i=0; i<N; i++)
if (a[i] != 0) then b[i]=a[i]*b[i]

◼ Idea: Masked operations

❑ VMASK register is a bit mask determining which data element
should not be acted upon
VLD V0 = A
VLD V1 = B
VMASK = (V0 != 0)
VMUL V1 = V0 * V1
VST B = V1
❑ This is predicated execution. Execution is predicated on mask bit.
45
Another Example with Masking
for (i = 0; i < 64; ++i)
if (a[i] >= b[i]) Steps to execute the loop in SIMD code
c[i] = a[i]
1. Compare A, B to get
else
VMASK
c[i] = b[i]
2. Masked store of A into C

A B VMASK 3. Complement VMASK

1 2 0
2 2 1 4. Masked store of B into C
3 2 1
4 10 0
-5 -4 0
0 -3 1
6 5 1
-7 -8 1

46
 Masked Vector Instructions
Simple Implementation Density-Time Implementation
– execute all N operations, turn off – scan mask vector and only execute
result writeback according to mask elements with non-zero masks

M[7]=1 A[7] B[7] M[7]=1

M[6]=0 A[6] B[6] M[6]=0 A[7] B[7]
M[5]=1 A[5] B[5] M[5]=1
M[4]=1 A[4] B[4] M[4]=1
M[3]=0 A[3] B[3] M[3]=0 C[5]

M[2]=0 C[4]
M[1]=1
M[2]=0 C[2]
M[0]=0
M[1]=1 C[1] C[1]

Write data port

M[0]=0 C[0]
Which one is better?
Write Enable Write data port
Tradeoffs?
Slide credit: Krste Asanovic 47
Some Issues
◼ Stride and banking
❑ As long as they are relatively prime to each other and there
are enough banks to cover bank access latency, we can
sustain 1 element/cycle throughput

◼ Storage of a matrix
❑ Row major: Consecutive elements in a row are laid out
consecutively in memory
❑ Column major: Consecutive elements in a column are laid out
consecutively in memory
❑ You need to change the stride when accessing a row versus
column

48
Matrix Multiplication
◼ A and B, both in row-major order

A4x6 B6x10 → C4x10

Dot products of rows and columns
of A and B

◼ A: Load A0 into vector register V1

❑ Each time, increment address by one to access the next column
❑ Accesses have a stride of 1
◼ B: Load B0 into vector register V2 Different strides can lead
❑ Each time, increment address by 10 to bank conflicts
❑ Accesses have a stride of 10
How do we minimize them?
49
Minimizing Bank Conflicts
◼ More banks

◼ Better data layout to match the access pattern

❑ Is this always possible?

◼ Better mapping of address to bank

❑ E.g., randomized mapping
❑ Rau, “Pseudo-randomly interleaved memory,” ISCA 1991.

50
Array vs. Vector Processors, Revisited
◼ Array vs. vector processor distinction is a “purist’s”
distinction

◼ Most “modern” SIMD processors are a combination of both

❑ They exploit data parallelism in both time and space
❑ GPUs are a prime example we will cover in a bit more detail

51
Recall: Array vs. Vector Processors
ARRAY PROCESSOR VECTOR PROCESSOR

Instruction Stream Same op @ same time

Different ops @ time
LD VR  A[3:0] LD0 LD1 LD2 LD3 LD0
ADD VR  VR, 1 AD0 AD1 AD2 AD3 LD1 AD0
MUL VR  VR, 2
ST A[3:0]  VR MU0 MU1 MU2 MU3 LD2 AD1 MU0
ST0 ST1 ST2 ST3 LD3 AD2 MU1 ST0
Different ops @ same space AD3 MU2 ST1
MU3 ST2
Time Same op @ space ST3

Space Space

52
 Vector Instruction Execution
VADD A,B → C

Execution using Execution using

one pipelined four pipelined
functional unit functional units

A[6] B[6] A[24] B[24] A[25] B[25] A[26] B[26] A[27] B[27]
A[5] B[5] A[20] B[20] A[21] B[21] A[22] B[22] A[23] B[23]
A[4] B[4] A[16] B[16] A[17] B[17] A[18] B[18] A[19] B[19]
A[3] B[3] A[12] B[12] A[13] B[13] A[14] B[14] A[15] B[15]

C[2] C[8] C[9] C[10] C[11]

C[1] C[4] C[5] C[6] C[7]

Time Time
C[0] C[0] C[1] C[2] C[3]
Space
Slide credit: Krste Asanovic 53
 Vector Unit Structure
Functional Unit

Partitioned
Vector
Registers
Elements 0, Elements 1, Elements 2, Elements 3,
4, 8, … 5, 9, … 6, 10, … 7, 11, …

Lane

Memory Subsystem

Slide credit: Krste Asanovic 54

 Vector Instruction Level Parallelism
Can overlap execution of multiple vector instructions
❑ Example machine has 32 elements per vector register and 8 lanes
❑ Completes 24 operations/cycle while issuing 1 vector instruction/cycle

Load Unit Multiply Unit Add Unit

load
mul
add
time
load
mul
add

Instruction
issue

Slide credit: Krste Asanovic 55

We did not cover the following slides.
They are for your preparation for the
next lecture.

56
 Automatic Code Vectorization
for (i=0; i < N; i++)
C[i] = A[i] + B[i];
Scalar Sequential Code Vectorized Code

load load load

Iter. 1 load load load

add Time add add

store store store

load
Iter. Iter.
Iter. 2 load 1 2 Vector Instruction

add
Vectorization is a compile-time reordering of
operation sequencing
 requires extensive loop dependence analysis
store
Slide credit: Krste Asanovic 57
Vector/SIMD Processing Summary
◼ Vector/SIMD machines are good at exploiting regular data-
level parallelism
❑ Same operation performed on many data elements
❑ Improve performance, simplify design (no intra-vector
dependencies)

◼ Performance improvement limited by vectorizability of code

❑ Scalar operations limit vector machine performance
❑ Remember Amdahl’s Law
❑ CRAY-1 was the fastest SCALAR machine at its time!

◼ Many existing ISAs include (vector-like) SIMD operations

❑ Intel MMX/SSEn/AVX, PowerPC AltiVec, ARM Advanced SIMD
58
SIMD Operations in Modern ISAs
SIMD ISA Extensions
◼ Single Instruction Multiple Data (SIMD) extension
instructions
❑ Single instruction acts on multiple pieces of data at once
❑ Common application: graphics
❑ Perform short arithmetic operations (also called packed
arithmetic)
◼ For example: add four 8-bit numbers
◼ Must modify ALU to eliminate carries between 8-bit values
padd8 $s2, $s0, $s1
32 24 23 16 15 8 7 0 Bit position

a3 a2 a1 a0 $s0

+ b3 b2 b1 b0 $s1

a3 + b3 a2 + b2 a1 + b1 a0 + b0 $s2

60
Intel Pentium MMX Operations
◼ Idea: One instruction operates on multiple data elements
simultaneously
❑ À la array processing (yet much more limited)

❑ Designed with multimedia (graphics) operations in mind

No VLEN register
Opcode determines data type:
8 8-bit bytes
4 16-bit words
2 32-bit doublewords
1 64-bit quadword

Stride is always equal to 1.

Peleg and Weiser, “MMX Technology

Extension to the Intel Architecture,”
IEEE Micro, 1996.
61
MMX Example: Image Overlaying (I)
◼ Goal: Overlay the human in image 1 on top of the background in image 2

Peleg and Weiser, “MMX Technology Extension to the Intel Architecture,” IEEE Micro, 1996. 62
MMX Example: Image Overlaying (II)
Y = Blossom image X = Woman’s image

Peleg and Weiser, “MMX Technology Extension to the Intel Architecture,” IEEE Micro, 1996. 63
Digital Design & Computer Arch.
Lecture 19: SIMD Processors

Prof. Onur Mutlu

ETH Zürich
Spring 2020
7 May 2020