Plaksha Univ

The Memory Hierarchy

CS2011: Fundamentals of Computer Systems
Lectures 34th-36th, Nov. 8-13, 2024

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 1

 Plaksha Univ

Today
● Storage technologies and trends
● Locality of reference
● Caching in the memory hierarchy

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 2

 Plaksha Univ

Random-Access Memory (RAM)

● Key features
○ RAM is traditionally packaged as a chip.
○ Basic storage unit is normally a cell (one bit per cell).
○ Multiple RAM chips form a memory.

● RAM comes in two varieties:

○ SRAM (Static RAM)
○ DRAM (Dynamic RAM)

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 3

 Plaksha Univ

SRAM vs DRAM Summary

Trans. Access Needs Needs

per bit time refresh? EDC? Cost Applications

SRAM 4 or 6 1X No Maybe 100x Cache memories

DRAM 1 10X Yes Yes 1X Main memories, frame buffers

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 4

 Plaksha Univ

Nonvolatile Memories
● DRAM and SRAM are volatile memories
○ Lose information if powered off.
● Nonvolatile memories retain value even if powered off
○ Read-only memory (ROM): programmed during production
○ Programmable ROM (PROM): can be programmed once
○ Eraseable PROM (EPROM): can be bulk erased (UV, X-Ray)
○ Electrically eraseable PROM (EEPROM): electronic erase capability
○ Flash memory: EEPROMs. with partial (block-level) erase capability
■ Wears out after about 100,000 erasings
● Uses for Nonvolatile Memories
○ Firmware programs stored in a ROM (BIOS, controllers for disks, network cards,
graphics accelerators)
○ Solid state disks
○ Disk caches
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 5
 Plaksha Univ

Traditional Bus Structure Connecting CPU and Memory

● A bus is a collection of parallel wires that carry address, data, and control
signals.
● Buses are typically shared by multiple devices.

CPU chip
Register file

ALU
System bus Memory bus

I/O Main
Bus interface
bridge memory

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 6

 Plaksha Univ

Memory Read Transaction (1)

● CPU places address A on the memory bus.

Register file
Load operation: movq A, %rax
%rax ALU

Main memory
I/O bridge A 0
Bus interface x A

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 7

 Plaksha Univ

Memory Read Transaction (2)

● Main memory reads A from the memory bus, retrieves word x, and places it on
the bus.
Register file
Load operation: movq A, %rax
ALU
%rax
Main memory
I/O bridge x 0
Bus interface x A

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 8

 Plaksha Univ

Memory Read Transaction (3)

● CPU read word x from the bus and copies it into register %rax.

Register file
Load operation: movq A, %rax
%rax ALU
x
Main memory
I/O bridge 0
Bus interface x A

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 9

 Plaksha Univ

Memory Write Transaction (1)

● CPU places address A on bus. Main memory reads it and waits for the
corresponding data word to arrive.
Register file
Store operation: movq %rax, A
%rax ALU
y
Main memory
I/O bridge A 0
Bus interface
A

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 10

 Plaksha Univ

Memory Write Transaction (2)

● CPU places data word y on the bus.
Register file
Store operation: movq %rax, A
ALU
%rax y
Main memory
I/O bridge y 0
Bus interface A

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 11

 Plaksha Univ

Memory Write Transaction (3)

● Main memory reads data word y from the bus and stores it at address A.

Register file
Store operation: movq %rax, A
ALU
%rax y
main memory
I/O bridge 0
Bus interface y A

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 12

 Plaksha Univ

What’s Inside A Disk Drive?

Arm Spindle
Platters

Actuator

Electronics
(including a
processor
SCSI and memory!)
connector

Image courtesy of Seagate Technology

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 13
 Plaksha Univ

Disk Geometry
● Disks consist of platters, each with two surfaces.
● Each surface consists of concentric rings called tracks.
● Each track consists of sectors separated by gaps.

Tracks
Surface
Track k Gaps

Spindle

Sectors
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 14
 Plaksha Univ

Disk Geometry (Multiple-Platter View)

● Aligned tracks form a cylinder.
Cylinder k
Surface 0
Surface 1 Platter 0
Surface 2
Surface 3 Platter 1
Surface 4
Platter 2
Surface 5

Spindle

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 15

 Plaksha Univ

Disk Capacity
● Capacity: maximum number of bits that can be stored.
○ Vendors express capacity in units of gigabytes (GB), where
1 GB = 109 Bytes.
● Capacity is determined by these technology factors:
○ Recording density (bits/in): number of bits that can be squeezed into a 1
inch segment of a track.
○ Track density (tracks/in): number of tracks that can be squeezed into a 1
inch radial segment.
○ Areal density (bits/in2): product of recording and track density.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 16

 Plaksha Univ

Recording zones
● Modern disks partition tracks into disjoint
subsets called recording zones
○ Each track in a zone has the same

…
number of sectors, determined by
the circumference of innermost
track. Spindle
○ Each zone has a different number of
sectors/track, outer zones have more
sectors/track than inner zones.
○ So we use average number of
sectors/track when computing
capacity.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 17

 Plaksha Univ

Computing Disk Capacity

Capacity = (# bytes/sector) x (avg. # sectors/track) x
(# tracks/surface) x (# surfaces/platter) x
(# platters/disk)
Example:
▪ 512 bytes/sector
▪ 300 sectors/track (on average)
▪ 20,000 tracks/surface
▪ 2 surfaces/platter
▪ 5 platters/disk

Capacity = 512 x 300 x 20000 x 2 x 5

= 30,720,000,000
= 30.72 GB

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 18

 Plaksha Univ

Disk Operation (Single-Platter View)

The disk surface

The read/write head
spins at a fixed
is attached to the end
rotational rate
of the arm and flies over
the disk surface on
a thin cushion of air.

dle
spin
spin
spindle
spindle

dle
By moving radially, the arm can
position the read/write head over any
track.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 19

 Plaksha Univ

Disk Operation (Multi-Platter View)

Read/write heads
move in unison
from cylinder to cylinder

Arm

Spindle

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 20

 Plaksha Univ

Disk Structure - top view of single platter

Surface organized into tracks

Tracks divided into sectors

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 21

 Plaksha Univ

Disk Access

Head in position above a track

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 22

 Plaksha Univ

Disk Access

Rotation is counter-clockwise

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 23

 Plaksha Univ

Disk Access – Read

About to read blue sector

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 24

 Plaksha Univ

Disk Access – Read

After BLUE read

After reading blue sector

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 25

 Plaksha Univ

Disk Access – Read

After BLUE read

Red request scheduled next

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 26

 Plaksha Univ

Disk Access – Seek

After BLUE read Seek for RED

Seek to red’s track

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 27

 Plaksha Univ

Disk Access – Rotational Latency

After BLUE read Seek for RED Rotational latency

Wait for red sector to rotate around

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 28

 Plaksha Univ

Disk Access – Read

After BLUE read Seek for RED Rotational latency After RED read

Complete read of red

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 29

 Plaksha Univ

Disk Access – Service Time Components

After BLUE read Seek for RED Rotational latency After RED read

Data transfer Seek Rotational Data transfer

latency

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 30

 Plaksha Univ

Disk Access Time

● Average time to access some target sector approximated by :
○ Taccess = Tavg seek + Tavg rotation + Tavg transfer
● Seek time (Tavg seek)
○ Time to position heads over cylinder containing target sector.
○ Typical Tavg seek is 3—9 ms
● Rotational latency (Tavg rotation)
○ Time waiting for first bit of target sector to pass under r/w head.
○ Tavg rotation = 1/2 x 1/RPMs x 60 sec/1 min
○ Typical Tavg rotation = 7200 RPMs
● Transfer time (Tavg transfer)
○ Time to read the bits in the target sector.
○ Tavg transfer = 1/RPM x 1/(avg # sectors/track) x 60 secs/1 min.
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 31
 Plaksha Univ

Disk Access Time Example

● Given:
○ Rotational rate = 7,200 RPM
○ Average seek time = 9 ms.
○ Avg # sectors/track = 400.
● Derived:
○ Tavg rotation = 1/2 x (60 secs/7200 RPM) x 1000 ms/sec = 4 ms.
○ Tavg transfer = 60/7200 RPM x 1/400 secs/track x 1000 ms/sec = 0.02 ms
○ Taccess = 9 ms + 4 ms + 0.02 ms
● Important points:
○ Access time dominated by seek time and rotational latency.
○ First bit in a sector is the most expensive, the rest are free.
○ SRAM access time is about 4 ns/doubleword, DRAM about 60 ns
■ Disk is about 40,000 times slower than SRAM,
■ 2,500 times slower then DRAM.
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 32
 Plaksha Univ

Logical Disk Blocks

● Modern disks present a simpler abstract view of the complex sector geometry:
○ The set of available sectors is modeled as a sequence of b-sized logical
blocks (0, 1, 2, ...)
● Mapping between logical blocks and actual (physical) sectors
○ Maintained by hardware/firmware device called disk controller.
○ Converts requests for logical blocks into (surface,track,sector) triples.
● Allows controller to set aside spare cylinders for each zone.
○ Accounts for the difference in “formatted capacity” and “maximum
capacity”.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 33

 Plaksha Univ

I/O Bus
CPU chip
Register file
ALU
System bus Memory bus

I/O Main
Bus interface memory
bridge

I/O bus Expansion slots for

USB other devices such
Graphics Disk
controller as network adapters.
adapter controller

Mouse Keyboard Monitor

Disk
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 34
 Plaksha Univ

Reading a Disk Sector (1)

CPU chip
Register file CPU initiates a disk read by writing a command,
logical block number, and destination memory
ALU address to a port (address) associated with disk
controller.

Main
Bus interface
memory

I/O bus

USB Graphics Disk

controller adapter controller

Mouse keyboard Monitor

Disk
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 35
 Plaksha Univ

Reading a Disk Sector (2)

CPU chip
Register file Disk controller reads the sector and performs
ALU a direct memory access (DMA) transfer into
main memory.

Main
Bus interface
memory

I/O bus

USB Graphics Disk

controller adapter controller

Mouse Keyboard Monitor

Disk
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 36
 Plaksha Univ

Reading a Disk Sector (3)

CPU chip
Register file When the DMA transfer completes, the disk
controller notifies the CPU with an interrupt (i.e.,
ALU asserts a special “interrupt” pin on the CPU)

Main
Bus interface
memory

I/O bus

USB Graphics Disk

controller adapter controller

Mouse Keyboard Monitor

Disk
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 37
 Plaksha Univ

Solid State Disks (SSDs)

I/O bus
Solid State Disk (SSD) Requests to read and
write logical disk blocks
Flash
translation layer
Flash memory
Block 0 Block B-1
Page 0 Page 1 … Page P-1
… Page 0 Page 1 … Page P-1

● Pages: 512KB to 4KB, Blocks: 32 to 128 pages

● Data read/written in units of pages.
● Page can be written only after its block has been erased
● A block wears out after about 100,000 repeated writes.
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 38
 Plaksha Univ

SSD Performance Characteristics

Sequential read tput 550 MB/s Sequential write tput 470 MB/s
Random read tput 365 MB/s Random write tput 303 MB/s
Avg seq read time 50 us Avg seq write time 60 us

● Sequential access faster than random access

○ Common theme in the memory hierarchy
● Random writes are somewhat slower
○ Erasing a block takes a long time (~1 ms)
○ Modifying a block page requires all other pages to be copied to new block
○ In earlier SSDs, the read/write gap was much larger.

Source: Intel SSD 730 product specification.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 39
 Plaksha Univ

SSD Tradeoffs vs Rotating Disks

● Advantages
○ No moving parts 🡪 faster, less power, more rugged
● Disadvantages
○ Have the potential to wear out
■ Mitigated by “wear leveling logic” in flash translation layer
■ E.g. Intel SSD 730 guarantees 128 petabyte (128 x 1015 bytes) of writes
before they wear out
○ In 2015, about 30 times more expensive per byte
● Applications
○ MP3 players, smart phones, laptops
○ Beginning to appear in desktops and servers

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 40

 Plaksha Univ

The CPU-Memory Gap

The gap widens between DRAM, disk, and CPU speeds.

Disk

SSD

DRAM

CPU

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 41

 Plaksha Univ

Locality to the Rescue!

● The key to bridging this CPU-Memory gap is a fundamental property of

computer programs known as locality

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 42

 Plaksha Univ

Today
● Storage technologies and trends
● Locality of reference
● Caching in the memory hierarchy

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 43

 Plaksha Univ

Locality
● Principle of Locality: Programs tend to use data and instructions with
addresses near or equal to those they have used recently

● Temporal locality:
○ Recently referenced items are likely
to be referenced again in the near future

● Spatial locality:
○ Items with nearby addresses tend
to be referenced close together in time

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 44

 Plaksha Univ

Locality Example

sum = 0;
for (i = 0; i < n; i++)
sum += a[i];
return sum;

● Data references
○ Reference array elements in succession
(stride-1 reference pattern).
Spatial locality
○ Reference variable sum each iteration. Temporal locality
● Instruction references
○ Reference instructions in sequence. Spatial locality
○ Cycle through loop repeatedly. Temporal locality
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 45
 Plaksha Univ

Qualitative Estimates of Locality

● Claim: Being able to look at code and get a qualitative sense of its locality is a
key skill for a professional programmer.

● Question: Does this function have good locality with respect to array a?

int sum_array_rows(int a[M][N])

{
int i, j, sum = 0;

for (i = 0; i < M; i++)

for (j = 0; j < N; j++)
sum += a[i][j];
return sum;
}
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 46
 Plaksha Univ

Locality Example
● Question: Does this function have good locality with respect to array a?

int sum_array_cols(int a[M][N])

{
int i, j, sum = 0;

for (j = 0; j < N; j++)

for (i = 0; i < M; i++)
sum += a[i][j];
return sum;
}

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 47

 Plaksha Univ

Locality Example
● Question: Can you permute the loops so that the function scans the 3-d array a
with a stride-1 reference pattern (and thus has good spatial locality)?

int sum_array_3d(int a[M][N][N])

{
int i, j, k, sum = 0;

for (i = 0; i < M; i++)

for (j = 0; j < N; j++)
for (k = 0; k < N; k++)
sum += a[k][i][j];
return sum;
}

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 48

 Plaksha Univ

Memory Hierarchies
● Some fundamental and enduring properties of hardware and software:
○ Fast storage technologies cost more per byte, have less capacity, and
require more power (heat!).
○ The gap between CPU and main memory speed is widening.
○ Well-written programs tend to exhibit good locality.

● These fundamental properties complement each other beautifully.

● They suggest an approach for organizing memory and storage systems known
as a memory hierarchy.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 49

 Plaksha Univ

Today
● Storage technologies and trends
● Locality of reference
● Caching in the memory hierarchy

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 50

 Plaksha Univ

Example Memory Hierarchy

Smaller, CPU registers hold words retrieved
L0: Regs from the L1 cache.
faster,
and
L1 cache L1 cache holds cache lines
costlier L1: (SRAM) retrieved from the L2 cache.
(per byte)
storage L2 cache L2 cache holds cache lines
devices L2: (SRAM) retrieved from L3 cache

L3: L3 cache L3 cache holds cache lines

Larger, (SRAM) retrieved from main memory.
slower,
and Main memory Main memory holds disk blocks
cheaper L4: (DRAM) retrieved from local disks.
(per byte)
storage Local secondary storage Local disks hold files retrieved
devices L5: from disks on remote servers
(local disks)
Remote secondary storage
L6: (e.g., Web servers)
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 51
 Plaksha Univ

Caches
● Cache: A smaller, faster storage device that acts as a staging area for a subset of the
data in a larger, slower device.
● Fundamental idea of a memory hierarchy:
○ For each k, the faster, smaller device at level k serves as a cache for the larger,
slower device at level k+1.
● Why do memory hierarchies work?
○ Because of locality, programs tend to access the data at level k more often than
they access the data at level k+1.
○ Thus, the storage at level k+1 can be slower, and thus larger and cheaper per bit.
● Big Idea: The memory hierarchy creates a large pool of storage that costs as much as
the cheap storage near the bottom, but that serves data to programs at the rate of
the fast storage near the top.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 52

 Plaksha Univ

General Cache Concepts

Smaller, faster, more expensive

Cache 8
4 9 14
10 3 memory caches a subset of
the blocks
Data is copied in block-sized
10
4
transfer units

Larger, slower, cheaper memory

Memory 0 1 2 3 viewed as partitioned into “blocks”
4 5 6 7
8 9 10 11
12 13 14 15

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 53

 Plaksha Univ

General Cache Concepts: Hit

Request: 14 Data in block b is needed

8 9 14 3 Block b is in cache:
Cache Hit!

Memory 0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 54

 Plaksha Univ

General Cache Concepts: Miss

Request: 12 Data in block b is needed

Cache 8 9
12 14 3 Block b is not in cache:
Miss!

12 Block b is fetched from

Request: 12
memory

Memory 0 1 2 3 Block b is stored in cache

•Placement policy:
4 5 6 7
determines where b goes
8 9 10 11
•Replacement policy:
12 13 14 15
determines which block
gets evicted (victim)
Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 55
 Plaksha Univ

General Caching Concepts: Types of Cache Misses

● Cold (compulsory) miss
○ Cold misses occur because the cache is empty.
● Conflict miss
○ Most caches limit blocks at level k+1 to a small subset (sometimes a singleton) of
the block positions at level k.
■ E.g. Block i at level k+1 must be placed in block (i mod 4) at level k.
○ Conflict misses occur when the level k cache is large enough, but multiple data
objects all map to the same level k block.
■ E.g. Referencing blocks 0, 8, 0, 8, 0, 8, ... would miss every time.
● Capacity miss
○ Occurs when the set of active cache blocks (working set) is larger than the cache.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 56

 Plaksha Univ

Examples of Caching in the Mem. Hierarchy

Cache Type What is Cached? Where is it Cached? Latency (cycles) Managed By
Registers 4-8 bytes words CPU core 0 Compiler
TLB Address translations On-Chip TLB 0 Hardware MMU

L1 cache 64-byte blocks On-Chip L1 4 Hardware

L2 cache 64-byte blocks On-Chip L2 10 Hardware
Virtual Memory 4-KB pages Main memory 100 Hardware + OS
Buffer cache Parts of files Main memory 100 OS
Disk cache Disk sectors Disk controller 100,000 Disk firmware
Network buffer cache Parts of files Local disk 10,000,000 NFS client

Browser cache Web pages Local disk 10,000,000 Web browser

Web cache Web pages Remote server disks 1,000,000,000 Web proxy server

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 57

 Plaksha Univ

Summary
● The speed gap between CPU, memory and mass storage continues to widen.

● Well-written programs exhibit a property called locality.

● Memory hierarchies based on caching close the gap by exploiting locality.

Bryant and O’Hallaron, Computer Systems: A Programmer’s Perspective, Third Edition 58