Exploiting Memory Hierarchy
Locality
Locality is a principle that makes having a memory hierarchy a
good idea
If an item is referenced then because of
temporal locality: it will tend to be again referenced soon
spatial locality: nearby items will tend to be referenced soon
why does code have locality – consider instruction and data?
Hit and Miss
Focus on any two adjacent levels – called, upper (closer to CPU)
and lower (farther from CPU) – in the memory hierarchy,
because each block copy is always between two adjacent levels
Terminology:
block: minimum unit of data to move between levels
hit: data requested is in upper level
miss: data requested is not in upper level
hit rate: fraction of memory accesses that are hits (i.e.,
found at upper level)
miss rate: fraction of memory accesses that are not hits
miss rate = 1 – hit rate
hit time: time to determine if the access is indeed a hit +
time to access and deliver the data from the upper level to
the CPU
miss penalty: time to determine if the access is a miss + time
to replace block at upper level with corresponding block at
lower level + time to deliver the block to the CPU
Caches
By simple example
assume block size = one word of data
X4 X4
Reference to Xn
X1 X1
causes miss so
Xn – 2 Xn – 2
it is fetched from
memory
Xn – 1 Xn – 1
X2 X2
Xn
X3 X3
a. Before the reference to Xn b. After the reference to Xn
Issues:
how do we know if a data item is in the cache?
if it is, how do we find it?
if not, what do we do?
Solution depends on cache addressing scheme…
Direct Mapped Cache
MIPS style:
A d dre ss ( s h ow ing b it p os itio ns )
31 30 13 12 11 2 1 0
Byte
offse t
20 10
H it D a ta
Tag
Ind e x
In de x V a lid T ag D a ta
0
1
2
1021
1022
1023
20 32
Cache with 1024 1-word blocks: byte offset
(least 2 significant bits) is ignored and
next 10 bits used to index into cache
What kind of locality are we taking advantage of?
DECStation 3100 Cache(MIPS R2000 processor)
Address (showing bit positions)
3 1 30 17 1 6 15 54 32 10
16 14 Byte
offset
H it D ata
16 bits 32 bits
Valid Tag D ata
16K
entries
16 32
Cache with 16K 1-word blocks: byte offset
(least 2 significant bits) is ignored and
next 14 bits used to index into cache
Cache Read Hit/Miss
Cache read hit: no action needed
Instruction cache read miss:
1. Send original PC value (current PC – 4, as PC has already
been incremented in first step of instruction cycle) to
memory
2. Instruct main memory to perform read and wait for
memory to complete access – stall on read
3. After read completes write cache entry
4. Restart instruction execution at first step to refetch
instruction
Data cache read miss:
Similar to instruction cache miss
To reduce data miss penalty allow processor to execute
instructions while waiting for the read to complete until
the word is required – stall on use
Cache Write Hit/Miss
Write-through scheme
on write hit: replace data in cache and memory with every
write hit to avoid inconsistency
on write miss: write the word into cache and memory –
obviously no need to read missed word from memory!
Write-through is slow because of always required memory write
performance is improved with a write buffer where words
are stored while waiting to be written to memory –
processor can continue execution until write buffer is full
when a word in the write buffer completes writing into main
memory that buffer slot is freed and becomes available for
future writes
DEC 3100 write buffer has 4 words
Write-back scheme
write the data block only into the cache and write-back the
block to main only when it is replaced in cache
more efficient than write-through, more complex to implement
Direct Mapped Cache: Taking Advantage of Spatial Locality
Taking advantage of spatial locality with larger blocks:
Address (showing bit positions)
31 16 1 5 4 32 1 0
16 12 2 Byte
H it T ag D a ta
offset
Index Block offset
1 6 bits 12 8 bits
V T ag D ata
4K
entrie s
16 32 32 32 32
M ux
32
Cache with 4K 4-word blocks: byte offset (least 2 significant bits) is ignored,
next 2 bits are block offset, and the next 12 bits are used to index into cache
Direct Mapped Cache: Taking Advantage of
Spatial Locality
Cache replacement in large (multiword) blocks:
word read miss: read entire block from main memory
word write miss: cannot simply write word and tag! Why?!
writing in a write-through cache:
if write hit, i.e., tag of requested address and cache
entry are equal, continue as for 1-word blocks by
replacing word and writing block to both cache and
memory
if write miss, i.e., tags are unequal, fetch block from
memory, replace word that caused miss, and write block
to both cache and memory
therefore, unlike case of 1-word blocks, a write miss with
a multiword block causes a memory read
Direct Mapped Cache: Taking Advantage of
Spatial Locality
Miss rate falls at first with increasing block size as expected,
but, as block size becomes a large fraction of total cache size,
miss rate may go up because
there are few blocks
competition for blocks increases
blocks get ejected before most of their words are accessed
(thrashing in cache)
40%
35%
30%
Miss rate vs. block size for 25%
Miss rate
various cache sizes 20%
15%
10%
5%
0%
4 16 64 256
Block size (bytes) 1 KB
8 KB
16 KB
64 KB
256 KB
Example
How many total bits are required for a direct-mapped cache
with 128 KB of data and 1-word block size, assuming a 32-bit
address?
Example
How many total bits are required for a direct-mapped cache
with 128 KB of data and 1-word block size, assuming a 32-bit
address?
Cache data = 128 KB = 217 bytes = 215 words = 215 blocks
Cache entry size = block data bits + tag bits + valid bit
= 32 + (32 – 15 – 2) + 1 = 48 bits
Therefore, cache size = 215 48 bits =
215 (1.5 32) bits = 1.5 220 bits = 1.5 Mbits
data bits in cache = 128 KB 8 = 1 Mbits
total cache size/actual cache data = 1.5
Example Problem
How many total bits are required for a direct-mapped cache with
128 KB of data and 4-word block size, assuming a 32-bit
address?
Cache size = 128 KB = 217 bytes = 215 words = 213 blocks
Cache entry size = block data bits + tag bits + valid bit
= 128 + (32 – 13 – 2 – 2) + 1 = 144 bits
Therefore, cache size = 213 144 bits =
213 (1.25 128) bits = 1.25 220 bits = 1.25 Mbits
data bits in cache = 128 KB 8 = 1 Mbits
total cache size/actual cache data = 1.25
Example Problem
Consider a cache with 64 blocks and a block size of 16 bytes. What
block number does byte address 1200 map to?
As block size = 16 bytes:
byte address 1200 block address 1200/16 = 75
As cache size = 64 blocks:
block address 75 cache block (75 mod 64) = 11
Block Size Considerations
Larger blocks should reduce miss rate
Due to spatial locality
But in a fixed-sized cache
Larger blocks fewer of them
More competition increased miss rate
Larger miss penalty
Can override benefit of reduced miss rate
Early restart and critical-word-first can help
Performance
Simplified model assuming equal read and write miss penalties:
CPU time = (execution cycles + memory stall cycles) cycle
time
memory stall cycles = number of memory accesses miss
rate miss penalty
Therefore, two ways to improve performance in cache:
decrease miss rate
decrease miss penalty
what happens if we increase block size?
Example
Assume for a given machine and program:
instruction cache miss rate 2%
data cache miss rate 4%
miss penalty always 40 cycles
CPI of 2 without memory stalls
frequency of load/stores 36% of instructions
1. How much faster is a machine with a perfect cache that never
misses?
2. What happens if we speed up the machine by reducing its CPI to 1
without changing the clock rate?
3. What happens if we speed up the machine by doubling its clock rate,
but if the absolute time for a miss penalty remains same?
Solution
1.
Assume instruction count = I
Instruction miss cycles = I 2% 40 = 0.8 I
Data miss cycles = I 36% 4% 40 = 0.576 I
So, total memory-stall cycles = 0.8 I + 0.576 I = 1.376 I
in other words, 1.376 stall cycles per instruction
Therefore, CPI with memory stalls = 2 + 1.376 = 3.376
Assuming instruction count and clock rate remain same for a
perfect cache and a cache that misses:
CPU time with stalls / CPU time with perfect cache
= 3.376 / 2 = 1.688
Performance with a perfect cache is better by a factor of 1.688
Solution (cont.)
2. What happens if we speed up the machine by reducing its CPI to 1
without changing the clock rate?
CPI without stall = 1
CPI with stall = 1 + 1.376 = 2.376 (clock has not changed so
stall cycles per instruction
remains same)
CPU time with stalls / CPU time with perfect cache
= CPI with stall / CPI without stall
= 2.376
Performance with a perfect cache is better by a factor of 2.376
Conclusion: Lower the CPI more pronounced is the impact of
stall cycles
Solution (cont.)
3. What happens if we speed up the machine by doubling its clock rate, but if
the absolute time for a miss penalty remains same?
With doubled clock rate, miss penalty = 2 40 = 80 clock cycles
Stall cycles per instruction = (I 2% 80) + (I 36% 4% 80)
= 2.752 I
So, faster machine with cache miss has CPI = 2 + 2.752 = 4.752
CPU time with stalls / CPU time with perfect cache
= CPI with stall / CPI without stall
= 4.752 / 2 = 2.376
Performance with a perfect cache is better by a factor of 2.376
Conclusion: with higher clock rate cache misses “hurt more” than
with lower clock rate
Decreasing Miss Rates with Associative Block Placement
Direct mapped: one unique cache location for each memory block
cache block address = memory block address mod cache size
Fully associative: each memory block can locate anywhere in cache
all cache entries are searched (in parallel) to locate block
Set associative: each memory block can place in a unique set of
cache locations – if the set is of size n it is n-way set-associative
cache set address = memory block address mod number of sets in
cache
all cache entries in the corresponding set are searched (in parallel) to
locate block
Increasing degree of associativity
reduces miss rate
increases hit time because of the parallel search and then fetch
Decreasing Miss Rates with Associative Block Placement
Direct Mapped 2-way Set Associative Fully Associative
Direct mapped Set associative Fully associative
Block # 0 1 2 3 4 5 6 7 Set # 0 1 2 3
Data Data Data
12 mod 8 = 4 12 mod 4 = 0
1 1 1
Tag Tag Tag
2 2 2
Search Search Search
Location of a memory block with address 12 in a cache with 8
blocks with different degrees of associativity
Decreasing Miss Rates with Associative Block Placement
One-way set set
One-way associative
associative
(direct mapped)
Block Tag Data
0
Two-way set associative
1
Set Tag Data Tag Data
2
3 0
4 1
5 2
6 3
Four-way set associative
Set Tag Data Tag Data Tag Data Tag Data
0
1
Eight-way set associative (fully associative)
Tag Data Tag Data Tag Data Tag Data Tag Data Tag Data Tag Data Tag Data
Configurations of an 8-block cache with different degrees of associativity
Example
Find the number of misses for a cache with four 1-word blocks given
the following sequence of memory block accesses:
0, 8, 0, 6, 8,
for each of the following cache configurations
1. direct mapped
2. 2-way set associative (use LRU replacement policy)
3. fully associative
Note about LRU replacement
in a 2-way set associative cache LRU replacement can be
implemented with one bit at each set whose value
indicates the most recently referenced block
Solution
1 (direct-mapped)
Block address Cache block
0 0 (= 0 mod 4)
6 2 (= 6 mod 4)
8 0 (= 8 mod 4)
Block address translation in direct-mapped cache
Address of memory Hit or Contents of cache blocks after reference
block accessed miss 0 1 2 3
0 miss Memory[0]
8 miss Memory[8]
0 miss Memory[0]
6 miss Memory[0] Memory[6]
8 miss Memory[8] Memory[6]
Cache contents after each reference – red indicates new entry added
5 misses
Solution (cont.)
2 (two-way set-associative)
Block address Cache set
0 0 (= 0 mod 2)
6 0 (= 6 mod 2)
8 0 (= 8 mod 2)
Block address translation in a two-way set-associative cache
Address of memory Hit or Contents of cache blocks after reference
block accessed miss Set 0 Set 0 Set 1 Set 1
0 miss Memory[0]
8 miss Memory[0] Memory[8]
0 hit Memory[0] Memory[8]
6 miss Memory[0] Memory[6]
8 miss Memory[8] Memory[6]
Cache contents after each reference – red indicates new entry added
Four misses
Solution (cont.)
3 (fully associative)
Address of memory Hit or Contents of cache blocks after reference
block accessed miss Block 0 Block 1 Block 2 Block 3
0 miss Memory[0]
8 miss Memory[0] Memory[8]
0 hit Memory[0] Memory[8]
6 miss Memory[0] Memory[8] Memory[6]
8 hit Memory[0] Memory[8] Memory[6]
Cache contents after each reference – red indicates new entry added
3 misses
Implementation of a Set-Associative Cache
A d dr es s
31 3 0 1 2 11 10 9 8 3 2 1 0
22 8
In d ex V Tag D a ta V Tag D a ta V T ag D ata V T ag D ata
0
1
2
Set
253
254
255
22 32
4 - to - 1 m ultip le xo r
H it D a ta
4-way set-associative cache with 4 comparators and one 4-to-1
multiplexor:size of cache is 1K blocks = 256 sets * 4-block set size
Performance with Set-Associative Caches
1 5%
1 2%
9%
Miss rate
6%
3%
0%
O n e -w a y T w o -w a y F ou r-w a y E ig h t-w a y
A sso c ia tivity 1 KB 16 KB
2 KB 32 KB
4 KB 64 KB
8 KB 1 28 KB
Miss rates for each of eight cache sizes
with increasing associativity:
data generated from SPEC92 benchmarks
with 32 byte block size for all caches
Replacement Policy
Direct mapped: no choice
Set associative
Prefer non-valid entry, if there is one
Otherwise, choose among entries in the set
Least-recently used (LRU)
Choose the one unused for the longest time
Simple for 2-way, manageable for 4-way, too hard
beyond that
Random
Gives approximately the same performance as
LRU for high associativity
Multilevel Caches
Primary cache attached to CPU
Small, but fast
Level-2 cache services misses from primary
cache
Larger, slower, but still faster than main memory
Main memory services L-2 cache misses
Some high-end systems include L-3 cache
Decreasing Miss Penalty with Multilevel Caches
Add a second-level cache
primary cache is on the same chip as the processor
use SRAMs to add a second-level cache, between main
memory and the first-level cache
if miss occurs in primary cache second-level cache is
accessed
if data is found in second-level cache miss penalty is access
time of second-level cache which is much less than main
memory access time
if miss occurs again at second-level then main memory
access is required and large miss penalty is incurred
Design considerations using two levels of caches:
try and optimize the hit time on the 1
st level cache to reduce
clock cycle
try and optimize the miss rate on the 2
nd level cache to
reduce memory access penalties
In other words, 2
nd level allows 1st level to go for speed
without “worrying” about failure…
Example Problem
Assume a 500 MHz machine with
base CPI 1.0
main memory access time 200 ns.
miss rate 5%
How much faster will the machine be if we add a second-level
cache with 20ns access time that decreases the miss rate to 2%?
Solution
Miss penalty to main = 200 ns / (2 ns / clock cycle) = 100 clock cycles
Effective CPI with one level of cache
= Base CPI + Memory-stall cycles per instruction
= 1.0 + 5% 100 = 6.0
With two levels of cache, miss penalty to second-level cache
= 20 ns / (2 ns / clock cycle) = 10 clock cycles
Effective CPI with two levels of cache
= Base CPI + Primary stalls per instruction
+ Secondary stall per instruction
= 1 + 5% 10 + 2% 100 = 3.5
= 1 + (5%-2%)x10 + 2%x(100+10)
Therefore, machine with secondary cache is faster by a factor of
6.0 / 3.5 = 1.71
Multilevel On-Chip Caches
Intel Nehalem 4-core processor
Per core: 32KB L1 I-cache, 32KB L1 D-cache, 512KB L2 cache
3-Level Cache Organization
Intel Nehalem AMD Opteron X4
L1 caches L1 I-cache: 32KB, 64-byte L1 I-cache: 32KB, 64-byte
(per core) blocks, 4-way, approx LRU blocks, 2-way, LRU
replacement, hit time n/a replacement, hit time 3 cycles
L1 D-cache: 32KB, 64-byte L1 D-cache: 32KB, 64-byte
blocks, 8-way, approx LRU blocks, 2-way, LRU
replacement, write- replacement, write-
back/allocate, hit time n/a back/allocate, hit time 9 cycles
L2 unified 512KB, 64-byte blocks, 8-way, 512KB, 64-byte blocks, 16-way,
cache approx LRU replacement, write- approx LRU replacement, write-
(per core) back/allocate, hit time n/a back/allocate, hit time n/a
L3 unified 8MB, 64-byte blocks, 16-way, 2MB, 64-byte blocks, 32-way,
cache replacement n/a, write- replace block shared by fewest
(shared) back/allocate, hit time n/a cores, write-back/allocate, hit
time 32 cycles
n/a: data not available
Sources of Misses (3C’s)
Compulsory misses (aka cold start misses)
First access to a block
Capacity misses
Due to finite cache size
A replaced block is later accessed again
Conflict misses (aka collision misses)
In a non-fully associative cache
Due to competition for entries in a set
Would not occur in a fully associative cache of the
same total size
Cache Design Trade-offs
Design change Effect on miss rate Negative performance
effect
Increase cache size Decrease capacity May increase access
misses time
Increase associativity Decrease conflict May increase access
misses time
Increase block size Decrease compulsory Increases miss
misses penalty.
