25/03/2020 1

BVM ENGG. COLLEGE

AY: 2019-20 Sem: even Div: 9(EC)

Cache Memory

Microprocessor and computer

architecture
Dr. Bhargav Goradiya
 The Memory Hierarchy
This storage organization can be thought of as a pyramid:

25/03/2020 Cache Memory 2

 The Memory Hierarchy
To access a particular piece of data, the CPU
first sends a request to its nearest memory,
usually cache.

If the data is not in cache, then main memory

is queried. If the data is not in main memory,
then the request goes to disk.

Once the data is located, then the data, and a

number of its nearby data elements are
fetched into cache memory.

25/03/2020 Cache Memory 3

 Principle of Locality of Reference
Programs tend to reuse data and instructions
they have used recently

Instructions in localized areas are executed

repeatedly

“Make the common case fast”  favor

accesses to such data

Keep recently accessed data in the fastest

memory
25/03/2020 Cache Memory 4
 Principle of Locality of Reference
Most program execution time spent on routines in
which many instructions are executed repeatedly

Temporal
– Recently executed instruction is likely to be executed again
very soon
– Whenever an item is first needed, it is first brought to the
cache, where it will hopefully remain until it is needed
again. Also influences choice on which item to discard when
cache is full

Spatial
– Instructions in close proximity to a recently executed
instruction are also likely to be executed soon
– Instead of fetching just one item into the cache, fetch
several adjacent data items as well (block/cache line)
25/03/2020 Cache Memory 5
 Cache Memory
Cache Memory is intended to give:
– Memory speed approaching that of the fastest
memories available.
– Large memory size at the price of less expensive
types of semiconductor memories.

Small amount of fast memory.

Sits between normal main memory and CPU.

May be located on CPU chip or module.

25/03/2020 Cache Memory 6
 Conceptual Operation
Relatively large and slow main memory together
with faster, smaller cache.
Cache contains a copy of portions of main memory.
When processor attempts to read a word from
memory, a check is made to determine if the word
exists in cache.
– If it is, the word is delivered to the processor.
– If not, a block of main memory is read into the cache, then the
word is delivered to the processor.
Word Block
Transfer Transfer
Cache Main
CPU Memory
Memory

25/03/2020 Cache Memory 7

 Hit Ratio
A measure of the efficiency of the cache
structure.
– When the CPU refers to memory and the word is
found in the cache, this called a hit.
– When the word is not found in cache, this is called
a miss.

Hit ratio is the total number of hits divided

by the total number of access attempts (hits
+ misses).
– It has been shown practically that hit rations
higher than 0.9 are possible.

25/03/2020 Cache Memory 8

 Cache vs. Main Memory Structure
0
1
Tag Block 2 Block
0 (K words)
1
2
3
.
.
C-1
.
Block Length
(K Words)

Cache
Main
2n-1 Memory
Word Length

25/03/2020 Cache Memory 9

 Main Memory and Cache Memory
Main Memory consists of 2n words.
– Each word has a unique n-bit address.

We can consider that main memory is made

up of blocks of K words each.

Cache consists of C lines of K words each.

A block of main memory is copied into a line

of Cache.
– The “tag” field of the line identifies which main
memory block each cache line represents

25/03/2020 Cache Memory 10

 Memory Hierarchy Design
Block placement:
– Where can a block be placed in the upper
level?
Block identification:
– How is a block found if it is in the upper
level?
Block replacement:
– Which block should be replaced on a miss?
Write strategy:
– What happens on a write?
25/03/2020 Cache Memory 11
 Where can a block be placed in a cache?

Mapping function determines how a

block is placed in the cache

25/03/2020 Cache Memory 12

 Mapping Functions
Three Types
– Direct Mapping
– Associative Mapping
– Set-Associative Mapping
Examples assume 64K words main
memory and 2K words cache
1 Block consists of 16 words
65536/16 = 4096 Blocks in Memory
2048/16 = 128 Blocks in Cache

25/03/2020 Cache Memory 13

 Where can a block be placed in a cache?
How is a block found?
MAIN MEMORY
CACHE
Block 0
TAG Block 0 Block 1

TAG Block 1 MAPPING

FUNCTION Block 127
Block 128
Block 129
TAG Block 126

TAG Block 127 Block 255

Block 256
Word Block 257
16-Bit
Address 12 4

Block 4095
25/03/2020 Cache Memory 14
 Direct Mapping
Block j of main memory maps onto block (j modulo
128) of the cache.
Example:
– Block 2103 of main memory maps to block (2103 mod 128)
= block 55
Each main memory block has only one place in
cache
Cache Block 0
– Memory Blocks 0, 128, 256...
Cache Block 1
– Memory Blocks 1, 129, 257...
Contention may occur even when cache is not full
Replacement algorithm is trivial
Simplest
– Very inflexible

25/03/2020 Cache Memory 15

 Direct Mapping
16-bit address (64K words)
16 words per block  lower 4 bits
Cache block position  middle 7 bits
32 blocks are mapped to the same
word
Higher 5 bits tell which of the 32 blocks
are mapped
Higher 5 bits are stored in 5 tag bits
associated with cache location

25/03/2020 Cache Memory 16

 How is a block found if it is in the cache?
Direct Mapping
Middle 7 bits select determine which
location in cache is used
Higher-order 5 bits are matched with
tag bits in cache to check if desired
block is the one stored in the cache

25/03/2020 Cache Memory 17

 Direct Mapping
MAIN MEMORY
CACHE
Block 0
TAG Block 0 Block 1

TAG Block 1 MAPPING

FUNCTION Block 127
Block 128
Block 129
TAG Block 126

TAG Block 127 Block 255

Block 256
Tag Block Word Block 257
16-Bit
Address 5 7 4

Block 4095
25/03/2020 Cache Memory 18
 Direct Mapping Cache Organization

25/03/2020 Cache Memory 19

 Example: Direct Mapping
Main Memory: 2M words = 221 Words
Cache: 64K words = 216 Words
Words per Block: 8 words = 23 Words
No. of Blocks in a Cache:
– Cache Size / Block Size = 216 / 23 = 213
No. of bits in Tag field:
Total Bits – Block Bits – Word Bits
21 - 13 - 3 = 5
Tag Block Word

5 13 3

25/03/2020 Cache Memory 20

 Associative Mapping
To improve the hit ratio of the cache,
another mapping techniques is often utilized,
“associative mapping”.

A block of main memory may be mapped

into ANY line of the cache.
– A block of memory is no longer restricted to a
single line of cache.

Higher 12 bits are stored in tag bits

25/03/2020 Cache Memory 21

 How is a block found if it is in the cache?
Associative Mapping
Tag bits (Higher-order 12 bits) of an
address are compared with tag bits of
each block to check if desired block is
present
Higher cost than direct mapping due to
need to search all 128 tags
Tags must be searched in parallel for
performance reasons

25/03/2020 Cache Memory 22

 Associative Mapping
MAIN MEMORY
CACHE
Block 0
TAG Block 0 Block 1

TAG Block 1 MAPPING

FUNCTION Block 127
Block 128
Block 129
TAG Block 126

TAG Block 127 Block 255

Block 256
Word Block 257
16-Bit
Address 12 4

Block 4095
25/03/2020 Cache Memory 23
 Associative Mapping

25/03/2020 Cache Memory 24

 Set-Associative Mapping
Cache blocks are grouped into sets
A main memory block can reside in any
block of a specific set
Less contention than direct mapping
Less cost than associative mapping
Set = (Block Address) MOD (Number of
Sets in Cache)
k-way set associative cache: k blocks
per set

25/03/2020 Cache Memory 25

 How is a block found if it is in the cache?
Set-Associative Mapping
Example: Cache groups two blocks per
set  64 sets (6-bit set field)
64 blocks can be mapped onto one set
Tag bits in each cache block store
upper 6 bits of address to tell which of
the 64 blocks are currently in the cache

25/03/2020 Cache Memory 26

 Set-Associative Mapping
MAIN MEMORY
CACHE
Block 0
TAG Block 0 Block 1
Set 0
TAG Block 1 MAPPING
FUNCTION Block 127
Block 128
Block 129
TAG Block 126
Set 63
TAG Block 127 Block 255
Block 256
Tag Set Word Block 257
16-Bit
Address 6 6 4

Block 4095
25/03/2020 Cache Memory 27
25/03/2020 Cache Memory 28
 Set-Associative Mapping

25/03/2020 Cache Memory 29

 Example: Set-Associative Mapping
Main Memory: 32M words = 225 Words
Cache: 128K words = 217 Words
Words per Block: 16 words = 24 Words
8 Blocks per Set = 23 blocks
No of Sets in Cache:
– 217 / (23 *24) = 210
No of tag bits: 25 – 10 – 4 = 11
Tag Set Word

11 10 4

25/03/2020 Cache Memory 30

 Levels of Set Associativity
Direct Mapping: 1 block per set  128
sets
Fully Associative Mapping: 128 blocks
per set  1 set
Set Associative Mapping is in between
Direct and Fully Associative
Different mappings are just different
degrees of set associativity

25/03/2020 Cache Memory 31

Which block should be replaced on a cache
miss?
Replacement Algorithm
– Determines which block in the cache is to
be replaced in the event of a cache miss
and the cache is full
Trivial for direct mapped caches

25/03/2020 Cache Memory 32

Which block should be replaced on a cache
miss?
Replacement algorithms
– Random Replacement
– First-In First-Out (FIFO)
– Optimal Algorithm
– Least Recently Used (LRU)
– Least Frequently Used
– Most Frequently Used

25/03/2020 Cache Memory 33

 What happens on a write?
Write policies
– Write-through
– Write-back

25/03/2020 Cache Memory 34

 Write-Through
Cache location and main memory
location are updated simultaneously
Simpler but results in unnecessary
write operations if word is updated
many times during its cache residency
Requires only valid bit

25/03/2020 Cache Memory 35

 Valid Bit
Indicate if block stored in cache is still valid
Set to 1 when block is initially loaded to cache
Transfers from disk to main memory use DMA,
bypass cache
When main memory block is updated by a
source that bypasses the cache, if block is also
in cache, its valid bit is set to 0

25/03/2020 Cache Memory 36

 Write-Back
Update only the cache location and mark it
updated using dirty bit/modified bit
Main memory location is updated later when
block is replaced
Writes at the speed of the cache
Also results in unnecessary writes because
whole block is written back to memory even
if only one word is updated
Requires valid bit and dirty bit

25/03/2020 Cache Memory 37

 Dirty Bit
Tells whether block in cache has been
modified/has newer data than main
memory block
Problem: Transfer from main memory to
disk bypassing the cache
Solution: Flush the cache (write back all
dirty blocks) before DMA transfer begins

25/03/2020 Cache Memory 38

 What happens on a write miss?
No-write allocate: Data is directly
written to main memory
Write allocate: Block is first loaded
from cache, then cache block is written
to

25/03/2020 Cache Memory 39

 Write Buffer
Used as temporary holding location for
data to be written to memory
Processor need not wait for write to
finish
Data in write buffer will be written
when memory is available for writing
Works for both write-through and
write-back caches

25/03/2020 Cache Memory 40

 Example of Mapping Techniques
Consider data cache with 8 blocks of data
Each block of data consists of only one word
These are greatly simplified parameters
Consider 4 x 10 array of numbers, arranged
in column order
40 elements = 28h stored from 7A00h to
7A27h

25/03/2020 Cache Memory 41

 Example of Mapping Techniques
Address Array
7A00h A(0,0)
7A01h A(1,0)
7A02h A(2,0)
7A03h A(3,0) Direct Mapped 13

7A04h A(0,1)
Set-Associative 15
7A05h A(1,1)
7A06h A(2,1)
Associative 16
7A07h A(3,1)
… … 16-Bit
16
7A24h A(0,9) Address

7A25h A(1,9)
7A26h A(2,9)
7A27h A(3,9)

25/03/2020 Cache Memory 42

 Example of Mapping Techniques
Consider the following algorithm
SUM := 0
for j:= 0 to 9 do
SUM := SUM + A(0,j)
end
AVE := SUM / 10
for i:= 9 downto 0 do
A(0,i) := A(0,i) / AVE
end
This gets the average of the first row (0),
and stores the value of the element divided
by the average of all the elements

25/03/2020 Cache Memory 43

 Example of Mapping Techniques

Array Address
A(0,0) 7A00h
A(0,1) 7A04h
SUM := 0 A(0,2) 7A08h
for j:= 0 to 9 do A(0,3) 7A0Ch
SUM := SUM + A(0,j) A(0,4) 7A10h
end
A(0,5) 7A14h
AVE := SUM / 10
A(0,6) 7A18h
for i:= 9 downto 0 do
A(0,i) := A(0,i) / AVE A(0,7) 7A1Ch
end A(0,8) 7A20h
A(0,9) 7A24h

25/03/2020 Cache Memory 44

 Direct-Mapped Cache

j=1 j=3 j=5 j=7 j=9 i=6 i=4 i=2 i=0

0 A(0,0) A(0,2) A(0,4) A(0,6) A(0,8) A(0,6) A(0,4) A(0,2) A(0,0)

1
2
3
4 A(0,1) A(0,3) A(0,5) A(0,7) A(0,9) A(0,7) A(0,5) A(0,3) A(0,1)

5
6
7

25/03/2020 Cache Memory 45

 Direct-Mapped Cache
Only two cache locations are used due
to the way the array is arranged
First loop (0 to 9): all references result
in cache miss
Second loop (9 to 0): reference to 9
and 8 are in the cache, the rest are not
in the cache

25/03/2020 Cache Memory 46

 Associative-Mapped Cache

j=7 j=8 j=9 i=1 i=0

0 A(0,0) A(0,8) A(0,8) A(0,8) A(0,0)

1 A(0,1) A(0,1) A(0,9) A(0,1) A(0,1)

2 A(0,2) A(0,2) A(0,2) A(0,2) A(0,2)

3 A(0,3) A(0,3) A(0,3) A(0,3) A(0,3)

4 A(0,4) A(0,4) A(0,4) A(0,4) A(0,4)

5 A(0,5) A(0,5) A(0,5) A(0,5) A(0,5)

6 A(0,6) A(0,6) A(0,6) A(0,6) A(0,6)

7 A(0,7) A(0,7) A(0,7) A(0,7) A(0,7)

25/03/2020 Cache Memory 47

 Associative-Mapped Cache
All cache locations are used
First loop (0 to 9): all references result
in cache miss
Second loop (9 to 0): references from
9 to 2 are in the cache, the rest are not
in the cache
Good utilization of cache because of
the order of the loop

25/03/2020 Cache Memory 48

 Set-Associative-Mapped Cache

j=3 j=7 j=9 i=4 i=2 i=0

0 A(0,0) A(0,4) A(0,8) A(0,4) A(0,4) A(0,0)

Set 0 1 A(0,1) A(0,5) A(0,9) A(0,5) A(0,5) A(0,1)

2 A(0,2) A(0,6) A(0,6) A(0,6) A(0,2) A(0,2)

3 A(0,3) A(0,7) A(0,7) A(0,7) A(0,3) A(0,3)

4
Set 1 5
6
7

25/03/2020 Cache Memory 49

 Associative-Mapped Cache
Half of the cache was used  only one
set
First loop (0 to 9): all references result
in cache miss
Second loop (9 to 0): references 9 to 6
are in the cache, the rest are not in the
cache

25/03/2020 Cache Memory 50

 Example of Mapping Techniques
In general, full associative mapping
performs best
Second is set-associative mapping
Worst performance: direct mapping
But full associative mapping is
expensive to implement
Compromise: set-associative mapping

25/03/2020 Cache Memory 51

 Measuring Cache Performance
Hit Rate: h
– Ratio of number of hits to number of all
attempted accesses
Miss Rate
– Ratio of number of misses to number of all
attempted acccesses
Hit Rate + Miss Rate = 1
Miss Penalty: M
– Extra time to bring desired block into cache
Hit time: C
– time to hit in the cache; time to access data from
the cache

25/03/2020 Cache Memory 52

 Measuring Cache Performance
Average memory access time: tavg
– Hit Rate x Hit Time + Miss Rate x Miss
Penalty

tavg = hC + (1–h)M

25/03/2020 Cache Memory 53

 Measuring Cache Performance
Example
– 10 clock cycles for memory access
– 17 clock cycles to load a block into the cache
(miss penalty)
– 1 clock cycle to load word from cache (hit time)
– Assume 30% of instructions perform read/write 
130 memory access for every 100 instructions
– Hit rates: 0.95 for instructions, 0.9 for data
Time without cache = 130 * 10 = 1300
Time with cache = 100 (0.95*1 + 0.05*17)
+ 30 (0.9*1 + 0.1*17) = 258
Ratio: 5.04

25/03/2020 Cache Memory 54

 Cache Examples
Intel 386 has no internal cache
Intel 486 – on-chip 8KB unified cache
Intel Pentium – 8KB/8KB split L1 cache
Intel Pentium II – 8KB/16KB split L1
cache / 256KB L2 Cache

25/03/2020 Cache Memory 55

 Pentium III Cache
Two cache levels L1 and L2
L1
– 16-KB data cache
• Four-way set-associative / write-back or write-
through
– 16-KB instruction cache
• Two-way set-associative
L2
– Unified cache
– 256K (Coppermine), 512K (Tualatin)

25/03/2020 Cache Memory 56

 Pentium III Cache

Processing Units

LI Instruction LI Data
Cache Cache

Bus interface unit

System Bus
Cache Bus

L2 Cache Main Memory Input/Output

25/03/2020 Cache Memory 57

 Pentium IV Cache
Up to 3 levels of caches
L1
– 8K data cache, 4-way set-associative,
write-through
– 12K instruction cache
L2
– Unified cache: 256 KB, 512 KB, write-back
L1 and L2 implemented on chip

25/03/2020 Cache Memory 58

 Memory Interleaving
Each memory module has its own ABR
(address buffer register) and DBR
(data buffer register)
Consecutive addresses are located in
successive modules
Lower-order address bits select a
module
Higher-order address bits select a
location within the module

25/03/2020 Cache Memory 59

 Memory Interleaving
CONSECUTIVE WORDS IN A MODULE

k m

Module Address in Module Main Memory Address

ABR DBR ABR DBR ABR DBR

Module Module Module

0 1 2

25/03/2020 Cache Memory 60

 Memory Interleaving
CONSECUTIVE WORDS IN CONSECUTIVE MODULES
(INTERLEAVED)

m k

Address in Module Module Main Memory Address

ABR DBR ABR DBR ABR DBR

Module Module Module

0 1 2

25/03/2020 Cache Memory 61

 Example
Assume a cache with 8-word blocks is
used.
On a read miss the block that contains
the desired word must be copied from
the main memory into Cache.
Hardware takes 1 cycle to put address.
Main memory is, Say, DRAM allow the
first word to be accessed in 8 cycle, but
subsequent words in 4 cycles.

25/03/2020 Cache Memory 62

 If consecutive words are in single
module, then the time required to load
the desired block in to cache is
1 + 8 + (7 x 4) + 1 = 38
If consecutive words are in Consecutive
module, then the time required to load
the desired block in to cache is
1 + 8 + 4 + 4 = 17

25/03/2020 Cache Memory 63

 Example Based on Cache Memory
A Program consists of two nested
loops. The general structure of a
program is given below. The program
is to be rum on that has instruction
cache organized in the direct mapped
manner that has following parameters:
Main Memory Size: 64K words
Cache Size: 1K Words
Block Size: 128 Words

25/03/2020 Cache Memory 64

START 17 The Cycle time of the
main memory is 10τ s
23 and that of the cache is
1τ s.
165 (a) Specify the no of
20
Times 10 bits in TAG, BLOCK and
239 Times WORD fields in main
memory address.
1200
(b) Compute the total
END 1500 time needed for
instruction fetching
during the execution of
the program.
25/03/2020 Cache Memory 65
 Solution
Main memory address length is 16 bits.
BLOCK field is 3 bits (8 blocks).
WORD field is 7 bits (128 words/block).
TAG field is 6 bits.

25/03/2020 Cache Memory 66

25/03/2020 Cache Memory 67
 Solution
Hence, the sequence of reads from the main
memory blocks into cache blocks is

As this sequence shows, both the beginning

and the end of the outer loop use blocks 0
and 1 in the cache. They overwrite each
other on each pass through the loop. Blocks
2 to 7 remain resident in the cache until the
outer loop is completed.

25/03/2020 Cache Memory 68

 Solution
The total time for reading the blocks from
the main memory into the cache is therefore
(10 + 4 x 9 + 2) x 128 x 10τ = 61,440τ

Executing the program out of the cache:

Outer loop - inner loop =
[(1200 - 22) - (239 - 164)]10 1τ = 11,030τ
Inner loop = (239 - 164)200 x1τ = 15,000τ
End section of program =
1500 - 1200 = 300 x 1τ
Total execution time = 87,770τ

25/03/2020 Cache Memory 69

 Example Set-associative Mapping
Problem 5.10

25/03/2020 Cache Memory 70

 Solution
TAG field is 10 bits.
SET field is 4 bits.
WORD field is 6 bits.

Words 0, 1, 2,…., 4351 occupy blocks 0 to

67 in the main memory (MM).
After blocks 0, 1, 2,…., 63 have been read
from MM into the cache on the first pass, the
cache is full. Because of the fact that the
replacement algorithm is LRU, MM blocks
that occupy the first four sets of the 16
cache sets are always overwritten before
they can be used on a successive pass.

25/03/2020 Cache Memory 71

 Solution
In particular, MM blocks 0, 16, 32, 48, and 64
continually displace each other in competing for the
4 block positions in cache set 0. The same thing
occurs in cache set 1 (MM blocks, 1, 17, 33, 49, 65),
cache set 2 (MM blocks 2, 18, 34, 50, 66) and cache
set 3 (MM blocks 3, 19, 35, 51, 67).
MM blocks that occupy the last 12 sets (sets 4
through 15) are fetched once on the first pass and
remain in the cache for the next 9 passes.
On the first pass, all 68 blocks of the loop must be
fetched from the MM. On each of the 9 successive
passes, blocks in the last 12 sets of the cache (4 x
12 = 48) are found in the cache, and the remaining
20 (68-48) blocks must be fetched from the MM

25/03/2020 Cache Memory 72

 Solution
Time with Cache:
= 1 x 68 x 11τ + 9(20x11τ + 48x1τ)
=

Time without Cache:

= 10 x 68 x 10τ =

Improvement Factor = 2.15

25/03/2020 Cache Memory 73