Advanced Caching Techniques

Approaches to improving memory system performance

• eliminate memory operations
• decrease the number of misses
• decrease the miss penalty
• decrease the cache/memory access times
• hide memory latencies
• increase cache throughput
• increase memory bandwidth

Spring 2003 CSE P548 1

 Handling a Cache Miss the Old Way

(1) Send the address to the next level of the hierarchy

(2) Signal a read operation
(3) Wait for the data to arrive
(4) Update the cache entry with data*, rewrite the tag, turn the valid bit on, clear
the dirty bit (unless an instruction cache)
(5) Resend the memory address; this time there will be a hit.

* There are variations:

• fill the cache block, then send the requested word to the CPU
• send the requested word to the CPU as soon as it arrives at the cache
(early restart)
• requested word is sent from memory first; then the rest of the block
follows (requested word first)
Early restart and requested word first have a lower miss penalty, because they
return execution control to the CPU earlier.

Spring 2003 CSE P548 2

 Non-blocking Caches

Non-blocking cache (lockup-free cache)

• allows the CPU to continue executing instructions while a miss is
handled
• some processors allow only 1 outstanding miss (“hit under miss”)
• some processors allow multiple misses outstanding (“miss under miss”)
• miss status holding registers (MSHR)
• hardware structure for tracking outstanding misses
• physical address of the block
• which word in the block
• destination register number (if data)
• mechanism to merge requests to the same block
• mechanism to insure accesses to the same location execute in
program order

Spring 2003 CSE P548 3

 Non-blocking Caches

Non-blocking cache (lockup-free cache)

• can be used with both in-order and out-of-order processors
• in-order processors stall when an instruction that uses the load
data is the next instruction to be executed (non-blocking loads)
• out-of-order processors can execute instructions after the load
consumer
• how do non-blocking caches improve memory system performance?

Spring 2003 CSE P548 4

 Victim Cache

Victim cache
• small fully-associative cache
• contains the most recently replaced blocks of a direct-mapped
cache
• alternative to 2-way set-associative cache
• check it on a cache miss
• swap the direct-mapped block and victim cache block
• how do victim caches improve memory system performance?

• why do victim caches work?

Spring 2003 CSE P548 5

 Sub-block Placement

Divide a block into sub-blocks

tag I data V data V data I data

tag V data V data V data V data
tag V data V data V data V data
tag I data I data I data I data

• sub-block = unit of transfer on a cache miss

• valid bit/sub-block
• Misses:
• block-level miss: tags didn’t match
• sub-block-level miss: tags matched, valid bit was clear
+ the transfer time of a sub-block
+ fewer tags than if each sub-block were a block
- less implicit prefetching
• how does sub-block placement improve memory system performance?

Spring 2003 CSE P548 6

 Pseudo-set associative Cache

Pseudo-set associative cache

• access the cache
• if miss, invert the high-order index bit & access the cache again
+ miss rate of 2-way set associative cache
+ close to access time of direct-mapped cache
- increase in hit time (relative to 2-way associative) if always try slow hit
first
• predict which is the fast-hit block
• put the fast hit block in the same location; swap blocks if wrong
• how does pseudo-set associativity improve memory system
performance?

Spring 2003 CSE P548 7

 Pipelined Cache Access

Pipelined cache access

• simple 2-stage pipeline
• access
• data transfer
• tag check & hit/miss logic with the shorter
• how do pipelined caches improve memory system performance?

Spring 2003 CSE P548 8

 Mechanisms for Prefetching

Prefetching
+ reduces the number of misses
• stream buffers
• where prefetched instructions/data held
• if requested block in the stream buffer, then cache access is
cancelled for a prefetch

Spring 2003 CSE P548 9

 Trace Cache
Trace cache contents
• cache blocks contain instructions from the dynamic instruction stream
+ a more efficient use of I-cache space
+ reduces the number of misses by fetching statically noncontiguous
instructions in a single cycle
• cache tag is high branch address bits + predictions for all branches
within the block
• 2 next addresses (target & fall-through code)

Assessing a trace cache

• assess trace cache + branch predictor, BTB, TLB, I-cache in parallel
• compare PC & prediction history of the current branch instruction to the
trace cache tag
• hit: I-cache fetch ignored
• miss: use the I-cache
start constructing a new trace
Spring 2003 CSE P548 10
 Cache-friendly Compiler Optimizations

Exploit spatial locality

• schedule for array misses
• hoist first load to a cache block
Improve spatial locality
• group & transpose
• makes portions of vectors that are accessed together lie in memory
together
• loop interchange
• so inner loop follows memory layout
Improve temporal locality
• loop fusion
• do multiple computations on the same portion of an array
• tiling (also called blocking)
• do all computation on a small block of memory that will fit in the
cache

Spring 2003 CSE P548 11

 Effect of Tiling

Spring 2003 CSE P548 12

 Tiling Example
/* before */
for (i=0; i<n; i=i+1)
for (j=0; j<n; j=j+1){
r = 0;
for (k=0; k<n; k=k+1) {
r = r + y[i,k] * z[k,j]; }
x[i,j] = r;
};
/* after */
for (jj=0; jj<n; jj=jj+B)
for (kk=0; kk<n; kk=kk+B)
for (i=0; i<n; i=i+1)
for (j=jj; j<min(jj+B-1,n); j=j+1) {
r = 0;
for (k=kk; k<min(kk+B-1,n); k=k+1)
{r = r + y[i,k] * z[k,j]; }
x[i,j] = x[i,j] + r;
};

Spring 2003 CSE P548 13

 Memory Banks

Interleaved memory:
• multiple memory banks
• word locations are assigned across banks
• interleaving factor: number of banks
• a single address can access all banks at once

+ get more data for one transfer: increases memory bandwidth

• data is probably used (why?)
+ similar implementation to not banking memory
- larger chip capacity means fewer banks
- power issue
Spring 2003 CSE P548 14
 Memory Banks

Independent memory banks

• different banks can be accessed at once, with different addresses
• allows parallel access, possibly parallel data transfer
• multiple memory controllers, one to do each access
• different controllers cannot access the same bank
• cheaper than dual porting
• how do independent memory banks improve memory system
performance?

Spring 2003 CSE P548 15

 Machine Comparison

Spring 2003 CSE P548 16

 Advanced Caching Techniques

Approaches to improving memory system performance

• eliminate memory operations

• decrease the number of misses

• decrease the miss penalty

• decrease the cache/memory access times

• hide memory latencies

• increase cache throughput

• increase memory bandwidth

Spring 2003 CSE P548 17

 Wrap-up

Victim cache
TLB
Hardware or compiler-based prefetching
Cache-conscious compiler optimizations
Coupling a write-through memory update policy with a write buffer
Handling the read miss before replacing a block with a write-back memory
update policy
Sub-block placement
Non-blocking caches
Merging requests to the same cache block in a non-blocking cache
Requested word first or early restart
Cache hierarchies
Virtual caches
Pipelined cache accesses
Pseudo-set associative cache
Banked or interleaved memories
Independent memory banks
Wider bus