The memory hierarchy can be characterized by a number of parameters.

Access type,

the actions that physically take place during a read or

write operation

Capacity,

usually measured in bytes

Cycle time,

the time elapsed from the start of a read operation to

the start of a subsequent read

Latency,

The latency is defined as the time interval between the

request for information and the access to the first bit of
that information.

Bandwidth,

The bandwidth provides a measure of the number of

bits per second that can be accessed

Cost.

Random access any access to any memory location takes the same fixed amount of
time
Write operation to memory location 100 takes 15 ns then
Read operation to memory location 3000, also take 15 ns.
Compared to sequential access in which if
Access to location 100 takes 500 ns,
Access to location 101 takes 505 ns,
Access to location 300 may take 1500 ns.
The effectiveness of a memory hierarchy depends on the principle of moving
information into the fast memory infrequently and accessing it many times before
replacing it with new information. This principle is possible due to a phenomenon
called locality of reference; that is, within a given period of time, programs tend to
reference a relatively confined area of memory repeatedly. There exist two forms of
locality: spatial and temporal locality.

Spatial locality refers to:

When a given address has been referenced, it is most likely that

addresses near it will be referenced within a short period of time,
Example: consecutive instructions in a straight line program.

Temporal locality refers to:

Once a particular memory item has been referenced, it is most likely that
it will be referenced next,
Example: an instruction in a program loop.

The sequence of events that takes place when the processor makes a
request for an item is as follows.
First, the item is sought in the first memory level of the memory
hierarchy.
The probability of finding the requested item in the first level is
called the hit ratio, h1.
The probability of not finding (missing) the requested item in the
first level of the memory hierarchy is called the miss ratio, (1- h1).
When the requested item causes a miss, it is sought in the next
subsequent memory level.
The probability of finding the requested item in the second memory
level, the hit ratio of the second level, is h2.
The miss ratio of the second memory level is (1 - h2).
The process is repeated until the item is found. Upon finding the
requested item, it is brought and sent to the processor.
In a memory hierarchy that consists of three levels, the average memory
access time can be expressed as follows:

The average access time of a memory level is defined as the time

required to access one word in that level. In this equation, t1, t2, t3
represent, respectively, the access times of the three levels.

CACHE MEMORY
Distinguish between two types of main memory:
The conventional and the slave memory.
Slave memory is a second level of unconventional high-speed memory, (cache
memory)
(The term cache means a safe place for hiding or storing things).
The idea behind using a cache as the first level of the memory hierarchy is to keep
the information expected to be used more frequently by the CPU in the cache (a
small high-speed memory that is near the CPU).
The end result is that at any given time some active portion of the main memory is duplicated in the
cache.

When the processor makes a request for a memory reference, the request is first
sought in the cache.

If the request corresponds to an element that is currently residing in the

cache, we call that a cache hit.

If the request corresponds to an element that is not currently in the cache, we

call that a cache miss.

A cache hit ratio, hc, is defined as the probability of finding the requested element in
the cache. A cache miss ratio (1 - hc) is defined as the probability of not finding the
requested element in the cache.

In the case that the requested element is not found in the cache, then it has to be
brought from a subsequent memory level in the memory hierarchy.
If the element exists in the next memory level, that is, the main memory, then it has
to be brought and placed in the cache.
In expectation that the next requested element will be residing in the neighboring
locality of the current requested element (spatial locality), then upon a cache miss
what is actually brought to the main memory is a block of elements that contains the
requested element.

The advantage of transferring a block from the main memory to the cache will be
most visible if it could be possible to transfer such a block using one main memory
access time. Such a possibility could be achieved by increasing the rate at which
information can be transferred between the main memory and the cache. One
possible technique that is used to increase the bandwidth is memory interleaving. To
achieve best results, we can assume that the block brought from the main memory
to the cache, upon a cache miss, consists of elements that are stored in different
memory modules, that is, whereby successive memory addresses are stored in
successive memory modules. Figure 6.2 illustrates the simple case of a main
memory consisting of eight memory modules. It is assumed in this case that the
block consists of 8 bytes. Having introduced the basic idea leading to the use of a
cache memory, we would like to assess the impact of temporal and spatial locality
on the performance of the memory hierarchy. In order to make such an assessment,
we will limit our deliberation to the simple case of a hierarchy consisting only of two
levels, that is, the cache and the main memory. We assume that the main memory
access time is tm and the cache access time is tc. We will measure the impact of
locality in terms of the average access time, defined as the average time required to
access an element (a word) requested by the processor in such a two-level
hierarchy.

Impact of Temporal Locality

assume instructions in program loops, which are executed many times,
for example, n times, once loaded into the cache, are used more than
once before they are replaced by new instructions. The average access
time, tav, is given by

It was assumed that the requested memory element has created a

cache miss, thus leading to the transfer of a main memory block in time
tm. Following that, n accesses were made to the same requested
element, each taking tc. The above expression reveals that as the
number of repeated accesses, n, increases, the average access time
decreases, a desirable feature of the memory hierarchy
Impact of Spatial Locality

Assume that:
The size of the block transferred from the main memory to the cache,
upon a cache miss, is m elements.
All m elements were requested, one at a time, by the processor.
Based on these assumptions, the average access time, tav, is given by

The above expression reveals that as the number of elements in a

block, m, increases, the average access time decreases, a desirable
feature of the memory hierarchy.

Impact of Combined Temporal and Spatial Locality

Assume that:
The element requested by the processor created a cache miss.

leading to:

Transfer of a block, consisting of m elements, to the cache (that take tm).

due to spatial locality, all m elements constituting a block were requested, one at a
time, by the processor (requiring mtc).
Following that,
The originally requested element was accessed (n - 1) times (temporal locality), that
is, total of n times access to that element.
Based on these assumptions, the average access time, tav, is given by

The above expression reveals that

As the number of repeated accesses n increases, the
average access time will approach tc.
as more requests for items that do not exist in the cache (cache miss)
occur, more blocks would have to be brought to the cache.
This should raise two basic questions:
Where to place an incoming main memory block in the cache?
And in the case where the cache is totally filled,
which cache block should the incoming main memory block replace?

Cache-Mapping Function
Interface between the cache and the main memory.
A request for accessing a memory element is made by the processor
through issuing the address of the requested element.
The address issued by the processor may correspond to that of an
element that exists currently in the cache (cache hit); otherwise,
It may correspond to an element that is currently residing in the main
memory.
Therefore, address translation has to be made in order to determine the
whereabouts of the requested element.
This is one of the functions performed by the memory management unit
(MMU).
A schematic of the address mapping function is shown in Figure 6.3.

The system address represents the address issued by the processor

This address is used by an address translation function inside the MMU.
If address translation reveals that the issued address corresponds to an
element currently residing in the cache,
o Then the element will be made available to the processor.

o If the element is not currently in the cache, then it will be brought (as part of a
block) from the main memory and placed in the cache and the element
requested is made available to the processor.

Cache Memory Organization

1-

Direct Mapping
It places an incoming main memory block into a specific fixed
cache block location.
The placement is done based on a fixed relation between the
incoming block number, i, the cache block number, j, and the
number of cache blocks, N:
j = i mod N

Example 1
A main memory consisting of 4K blocks, a cache memory consisting of
128 blocks, and a block size of 16 words.
Figure 6.4 shows the division of the main memory and the cache
according to the direct-mapped cache technique.
There are a total of 32 main memory blocks that map to a given cache
block. For example, main memory blocks 0, 128, 256, 384, . . . , 3968
map to cache block 0.

It is simple in determining where to place an incoming main memory

block in the cache

It is the inefficient use of the cache.

A number of main memory blocks may compete for a given cache block
even if there exist other empty cache blocks. -----> Low cache hit ratio.
According to the direct-mapping technique the MMU interprets the
address issued by the processor by dividing the address into three fields
as shown in Figure 6.5. The lengths, in bits, of each of the fields in
Figure 6.5 are:

The protocol used by the MMU to satisfy a request made by the

processor for accessing a given element.
1. Use the Block field to determine the cache block that should
contain the element requested by the processor. The Block field is
used directly to determine the cache block sought
2. Check the corresponding Tag memory to see whether there is a
match between its content and that of the Tag field. A match
between the two indicates that the targeted cache block
determined in step 1 is currently holding the main memory element
requested by the processor, that is, a cache hit.
3. Among the elements contained in the cache block, the targeted
element can be selected using the Word field.

10

4. If in step 2, no match is found, then this indicates a cache miss.

Therefore, the required block has to be brought from the main
memory, deposited in the cache, and the targeted element is
made available to the processor. The cache Tag memory and the
cache block memory have to be updated accordingly

11

Fully Associative Mapping

An incoming main memory block can be placed in any available cache
block. Therefore,
The address issued by the processor need only have two fields. These
are the Tag and Word fields.
The first uniquely identifies the block while residing in the cache. The
second field identifies the element within the block that is requested by
the processor.

The MMU interprets the address issued by the processor by dividing it

into two fields

The length, in bits, of each of the fields in bits

Word field = log2 B, where B is the size of the block in words
Tag field = log2 M, where M is the size of the main memory in blocks
The number of bits in the main memory address = log2 (B x M)

Protocol used by the MMU to satisfy a request made by the processor

for accessing a given element.
1. Use the Tag field to search in the Tag memory for a match with any of
the tags stored.
2. A match in the tag memory indicates that the corresponding targeted
cache block determined in step 1 is currently holding the main memory
element requested by the processor, that is, a cache hit.

12

3. Among the elements contained in the cache block, the targeted

element can be selected using the Word field.
5. If in step 2, no match is found, then this indicates a cache miss.
Therefore, the required block has to be brought from the main
memory, deposited in the first available cache block, and the
targeted element (word) is made available to the processor.
The cache Tag memory and the cache block memory have to be
updated accordingly.

13

The search made in step 1 above requires matching the tag field of the
address with each and every entry in the tag memory.
Such a search, if:
done sequentially, could lead to a long delay. Therefore, the tags are
stored in an associative (content addressable) memory. This allows the
entire contents of the tag memory to be searched in parallel
(associatively), hence the name, associative mapping.

Each line of the cache contains its own compare circuitry that is able to
discern in an instant whether or not the block is contained at that line.
With all of the lines performing this compare in parallel, the correct line is
identified quickly.

This mechanism is needed to ensure that any accessed cache block

contains valid information.
It uses of a single bit for each cache block, called the valid bit.
The valid bit of a cache block should be
updated in such a way that if valid bit = 1, then the corresponding cache
block carries valid information; otherwise, the information in the cache
block is invalid.

When a computer system is powered up, all valid bits are made equal to
0, indicating that they carry invalid information.
As blocks are brought to the cache, their statuses are changed
accordingly to indicate the validity of the information contained.

The main advantage of the associative-mapping technique is:

Efficient use of the cache, this stems from the fact that there exists
no restriction on where to place incoming main memory blocks,

14

The main disadvantage of is:

The hardware overhead required to perform the associative search
conducted in order to find a match between the tag field and the
tag memory.

Set-Associative Mapping
The cache is divided into a number of sets. Each set consists of a
number of blocks.
A given main memory block maps to a specific cache set based on the
equation
s = i mod S,
Where S is the number of sets in the cache, i is the main memory block
number, and s is the specific cache set to which block i maps.
An incoming block maps to any block in the assigned cache set.
Therefore, the address issued by the processor is divided into three
distinct fields. These are the Tag, Set, and Word fields.
The Set field is used to uniquely identify the specific cache set that
ideally should hold the targeted block.
The Tag field uniquely identifies the targeted block within the determined
set.
The Word field identifies the element (word) within the block that is
requested by the processor.
According to the set-associative mapping technique, the MMU interprets
the address issued by the processor by dividing it into three fields as
shown in Figure 6.9. The length, in bits, of each of the fields of Figure
6.9 is given by
1. Word field = log2 B, where B is the size of the block in words
2. Set field = log2 S, where S is the number of sets in the cache
3. Tag field = log2 (M/S), where M is the size of the main memory in
blocks.

15

S = N/Bs, where N is the number of cache blocks and Bs is the

number of blocks per set
4. The number of bits in the main memory address = log2 (B x M)
Ex
Compute the above three parameters (Word, Set, and Tag) for a
memory system having the following specification: size of the main
memory is 4K blocks, size of the cache is 128 blocks, and the block size
is 16 words. Assume that the system uses set-associative mapping with
four blocks per set.

Protocol used by the MMU to satisfy a request made by the processor

for accessing a given element.

1. Use the Set field (5 bits) to determine (directly) the specified set (1 of
the 32 sets).
2. Use the Tag field to find a match with any of the (four) blocks in the
determined set. A match in the tag memory indicates that the
specified set determined in step 1 is currently holding the targeted
block, that is, a cache hit.
3. among the 16 words (elements) contained in hit cache block, the
requested word is selected using a selector with the help of the Word
field.
4. If in step 2, no match is found, then this indicates a cache miss.
Therefore, the required block has to be brought from the main
memory, deposited in the specified set first, and the targeted element
(word) is made available to the processor. The cache Tag memory
and the cache block memory have to be updated accordingly.

16

17

Example1
Assume a cache system has been designed such that each block contains 4 words and the
cache has 1024 lines, i.e., the cache can store up to 1024 blocks. What line of the cache is
supposed to hold the block that contains the word from the twenty-bit address 3A45616? In
addition, what is the tag number that will be stored with the block?

Solution
Start by dividing the address into its word id, line id, and tag bits. Since 4=22, then the two
least significant bits identify the word, i.e., w = 2. Since the cache has 1024=210 lines, then the
next 10 bits identify the line number where the data is supposed to be stored in the cache, i.e.,
l = 10. The remaining t = 20 w l = 8 bits are the tag bits. This partitions the address
3A45616 = 001110100100010101102 as follows:

Therefore, the block from address 3A45416 to 3A45716 will be stored in line 01000101012 =
27710 of the cache with the tag 001110102.

Example2
The first 10 lines of a 256 line cache are shown in the table below. Identify the address of the
data that is shaded (D816). For this cache, a block contains 4 words. The tags are given in
binary in the table.

18

2 bits represent the positions of each of the four words in a block

6 bits are used to represent the tag.
Since the cache has 256=28 lines, then the line number in the cache is represented
with 8 bits, and the address is partitioned as follows:

The shaded cell in the table has a

Tag number of 1100112.
The line number is 4, which in 8 bit binary is 000001002.
The word is in the third column which means that it is the 102 word within the block.
(Remember to start counting from 002.) Putting the tag, line id,
word id bits together gives us:

Example3
Using the table from the previous example, determine if the data stored in main memory at
address 101C16 is contained in this cache, and if it is, retrieve the data.
Converting 101C16 to binary gives us 0001 0000 0001 11002.

From this we see that the line in the cache where this data should be stored is 000001112 =
710. The tag currently stored in this line is 0001002 which equals the tag from the above
partitioned address.
The word is FE16.

19

Example4
Using the same table from the previous two examples, determine if the data from address
982716 is in the cache.
982716 = 1001 1000 0010 01112.

Tag is 1001102,
The line number is 000010012 = 910, and the word offset into the block is 112.

Looking at
Line number 9 we see that the tag stored there equals 1010002.
Since this does not equal 1001102, the data from that address is not contained in this cache,
and we will have to get it from the main memory.

Example5
The table below represents five lines from a cache that uses fully associative mapping with a
block size of eight. Identify the address of the shaded data (C916).

The tag for C916 is 01000110101012. When combining this with the word id of 0012, the
address in main memory from which C916 was retrieved is 01000110101010012 = 46A916.

20

Example 6
Is the data from memory address 1E6516 contained in the table from the previous example?
Since 1E6516 = 00011110011001012,
The word is 9E

Example7
Identify the set number where the block containing the address 29ABCDE816 will be stored.
In addition, identify the tag and the lower and upper addresses of the block. Assume the
cache is a 4-way set associative cache with 4K lines(block), each block containing 16 words,
with the main memory of size 1 Gig memory space.
Solution
First, we need to identify the partitioning of the bits in the memory address.
1 Gig = 230 memory space requires 30 address lines (bit).
Four of those address lines will be used to identify one out of the 16 words within the block.

Notice: A set associative cache that has k lines per set is referred to as a k-way set associative
cache

Since the cache is a 4-way set associative cache,

The number of sets = 4K lines divided by four lines per set, i.e., 1K = 210 sets.
Therefore, ten address lines will be needed to identify the set.
The figure below represents the partitioning of the 30 bit address.

29ABCDE816 to a 30-bit binary value gives us

001010011010101111001101111010002.

21

The first sixteen bits, 0010100110101011112, represent the tag.

The next ten bits, 00110111102 = 22210, represent the set in which the block will be stored.
The last four bits, 10002 = 810, represent the word position within the block.

The lowest address will be the one where word 00002 is stored and the highest address will be
the one where the word 11112 is stored.

Replacing the last four bits of 29ABCDE816 with 00002 gives us a low address of
29ABCDE016 while replacing them with 11112 gives us a high address of 29ABCDEF16

Following figure shows an example of a 1 Meg memory space divided into four word blocks.

22

Cash Memory Sheet

1. How many blocks of 16 words are there in a 256 Gig memory space? Draw the logical
organization of the full address identifying the block ID portion and the word offset
portion.
2.

Identify the line number, tag, and word position for each of the 30- bit addresses shown
below if they are stored in a cache using the direct mapping method.
a.) Address: 23D94EA616 Lines in cache: 4K Block size: 2
b.) Address: 1A54387F6 Lines in cache: 8K Block size: 4
c.) Address: 3FE9704A16 Lines in cache: 16K Block size: 16
d.) Address: 54381A516 Lines in cache: 1K Block size: 8

4. True or False: A block from main memory could possibly be stored in any line of a cache
using fully associative mapping.
5.

What problem is the fully or set-associative mapping methods for caches supposed to
solve over the direct mapping method?

6.

What is the easiest replacement algorithm to use with a 2-way set associative cache?

7. The table below represents five lines from a cache that uses fully associative mapping with
a block size of eight. Identify the address of the shaded data (3B16).

8. Using the table from the previous problem, identify the data value represented by each of
the following addresses.
a.) 7635916

b.) 386AF16

c.) BC5CC16

9. Identify the set number, tag, and word position for each of the 30- bit addresses stored in
an 8K line set associative cache.
a.) Address: 23D94EA616
b.) Address: 1A54387F6
c.) Address: 3FE9704A16
d.) Address: 54381A516

2-way cache Block size: 2

2-way cache Block size: 4
8-way cache Block size: 16
4-way cache Block size: 8

23