Cache algorithms - Wikipedia, the free encyclopedia

1 of 5

https://en.wikipedia.org/wiki/Cache_algorithms

Cache algorithms

From Wikipedia, the free encyclopedia

In computing, cache algorithms (also frequently called cache replacement algorithms or cache replacement
policies) are optimizing instructionsor algorithmsthat a computer program or a hardware-maintained
structure can follow in order to manage a cache of information stored on the computer. When the cache is full,
the algorithm must choose which items to discard to make room for the new ones.

Contents
1
2
3
4
5

Overview
Examples
See also
References
External links

Overview
The average memory reference time is[1]
where

= average memory reference time

= miss ratio = 1 - (hit ratio)
= time to make a main memory access when there is a miss (or, with multi-level cache, average
memory reference time for the next-lower cache)
= the latency: the time to reference the cache when there is a hit
= various secondary effects, such as queuing effects in multiprocessor systems

There are two primary figures of merit of a cache: The latency, and the hit rate. There are also a number of
secondary factors affecting cache performance.[1]

The "hit ratio" of a cache describes how often a searched-for item is actually found in the cache. More efficient
replacement policies keep track of more usage information in order to improve the hit rate (for a given cache
size).

The "latency" of a cache describes how long after requesting a desired item the cache can return that item (when
there is a hit). Faster replacement strategies typically keep track of less usage informationor, in the case of
direct-mapped cache, no informationto reduce the amount of time required to update that information.
Each replacement strategy is a compromise between hit rate and latency.

Measurements of "the hit ratio" are typically performed on benchmark applications. The actual hit ratio varies
widely from one application to another. In particular, video and audio streaming applications often have a hit
4/18/2016 12:22 PM

Cache algorithms - Wikipedia, the free encyclopedia

2 of 5

https://en.wikipedia.org/wiki/Cache_algorithms

ratio close to zero, because each bit of data in the stream is read once for the first time (a compulsory miss),
used, and then never read or written again. Even worse, many cache algorithms (in particular, LRU) allow this
streaming data to fill the cache, pushing out of the cache information that will be used again soon (cache
pollution).[2]

Examples

Bldy's Algorithm
The most efficient caching algorithm would be to always discard the information that will not be needed
for the longest time in the future. This optimal result is referred to as Bldy's optimal algorithm or the
clairvoyant algorithm. Since it is generally impossible to predict how far in the future information will be
needed, this is generally not implementable in practice. The practical minimum can be calculated only
after experimentation, and one can compare the effectiveness of the actually chosen cache algorithm.

Least Recently Used (LRU)

Discards the least recently used items first. This algorithm requires keeping track of what was used when,
which is expensive if one wants to make sure the algorithm always discards the least recently used item.
General implementations of this technique require keeping "age bits" for cache-lines and track the "Least
Recently Used" cache-line based on age-bits. In such an implementation, every time a cache-line is used,
the age of all other cache-lines changes. LRU is actually a family of caching algorithms with members
including 2Q by Theodore Johnson and Dennis Shasha,[3] and LRU/K by Pat O'Neil, Betty O'Neil and
Gerhard Weikum.[4]

Most Recently Used (MRU)

Discards, in contrast to LRU, the most recently used items first. In findings presented at the 11th VLDB
conference, Chou and DeWitt noted that "When a file is being repeatedly scanned in a [Looping
Sequential] reference pattern, MRU is the best replacement algorithm."[5] Subsequently other researchers
presenting at the 22nd VLDB conference noted that for random access patterns and repeated scans over
large datasets (sometimes known as cyclic access patterns) MRU cache algorithms have more hits than
LRU due to their tendency to retain older data.[6] MRU algorithms are most useful in situations where the
older an item is, the more likely it is to be accessed.
Pseudo-LRU (PLRU)
For CPU caches with large associativity (generally >4 ways), the implementation cost of LRU becomes
prohibitive. In many CPU caches, a scheme that almost always discards one of the least recently used
items is sufficient. So many CPU designers choose a PLRU algorithm which only needs one bit per cache
item to work.
PLRU typically has a slightly worse miss ratio, has a slightly better latency, and uses slightly less power
than LRU.

Random Replacement (RR)

Randomly selects a candidate item and discards it to make space when necessary. This algorithm does not
require keeping any information about the access history. For its simplicity, it has been used in ARM
processors.[7] It admits efficient stochastic simulation.[8]
Segmented LRU (SLRU)
An SLRU cache is divided into two segments, a probationary segment and a protected segment. Lines in

4/18/2016 12:22 PM

Cache algorithms - Wikipedia, the free encyclopedia

3 of 5

https://en.wikipedia.org/wiki/Cache_algorithms

each segment are ordered from

the most to the least recently
accessed. Data from misses is
added to the cache at the most
recently accessed end of the
probationary segment. Hits are
removed from wherever they
currently reside and added to
the most recently accessed end
of the protected segment.
Lines in the protected segment
have thus been accessed at
Which memory locations can be cached by which cache locations
least twice. The protected
segment is finite, so migration
of a line from the probationary segment to the protected segment may force the migration of the LRU line
in the protected segment to the most recently used (MRU) end of the probationary segment, giving this
line another chance to be accessed before being replaced. The size limit on the protected segment is an
SLRU parameter that varies according to the I/O workload patterns. Whenever data must be discarded
from the cache, lines are obtained from the LRU end of the probationary segment.[9]"

2-way set associative

Used for high-speed CPU caches where even PLRU is too slow. The address of a new item is used to
calculate one of two possible locations in the cache where it is allowed to go. The LRU of the two is
discarded. This requires one bit per pair of cache lines, to indicate which of the two was the least recently
used.
Direct-mapped cache
Used for the highest-speed CPU caches where even 2-way set associative caches are too slow. The
address of the new item is used to calculate the one location in the cache where it is allowed to go.
Whatever was there before is discarded.
Least-Frequently Used (LFU)
Counts how often an item is needed. Those that are used least often are discarded first.

Low Inter-reference Recency Set (LIRS)

A page replacement algorithm with an improved performance over LRU and many other newer
replacement algorithms. This is achieved by using reuse distance as a metric for dynamically ranking
accessed pages to make a replacement decision. The algorithm was developed by Song Jiang and
Xiaodong Zhang.
Adaptive Replacement Cache (ARC)
Constantly balances between LRU and LFU, to improve the combined result.[10] ARC improves on
SLRU by using information about recently-evicted cache items to dynamically adjust the size of the
protected segment and the probationary segment to make the best use of the available cache space.

Clock with Adaptive Replacement (CAR)

Combines Adaptive Replacement Cache (ARC) and CLOCK. CAR has performance comparable to ARC,
and substantially outperforms both LRU and CLOCK. Like ARC, CAR is self-tuning and requires no
4/18/2016 12:22 PM

Cache algorithms - Wikipedia, the free encyclopedia

4 of 5

https://en.wikipedia.org/wiki/Cache_algorithms

user-specified magic parameters.

Multi Queue (MQ)

By Zhou, Philbin, and Li.[11]
The MQ cache contains multiple LRU queues, Q0, Q1, ..., Qm-1. Blocks stay in the LRU queues for a
given lifetime, which is defined dynamically by the MQ algorithm to be the maximum temporal distance
between two accesses to the same file or the number of cache blocks, whichever is larger. If a block has
not been referenced within its lifetime, it is demoted from to Qi to Qi1 or evicted from the cache if it is
in Q0. Each queue also has a maximum access count; if a block in queue Qi is accessed more than 2i
times, this block is promoted to Qi+1 until it is accessed more than 2i+1 times or its lifetime expires.
Within a given queue, blocks are ranked by the recency of access, according to LRU.[12]

Pannier: Container-based caching algorithm for compound objects

Pannier [13] is a container-based flash caching mechanism that identifies divergent (heterogeneous)
containers where blocks held therein have highly varying access patters. Pannier uses a priority-queue
based survival queue structure to rank the containers based on their survival time, which is proportional to
the live data in the container. Pannier is built based on Segmented LRU(S2LRU), which segregates hot
and cold data. Pannier also uses a multi-step feedback controller to throttle flash writes to ensure flash
lifespan.
Other things to consider:

Items with different cost: keep items that are expensive to obtain, e.g. those that take a long time to get.
Items taking up more cache: If items have different sizes, the cache may want to discard a large item to
store several smaller ones.
Items that expire with time: Some caches keep information that expires (e.g. a news cache, a DNS cache,
or a web browser cache). The computer may discard items because they are expired. Depending on the
size of the cache no further caching algorithm to discard items may be necessary.

Various algorithms also exist to maintain cache coherency. This applies only to situation where multiple
independent caches are used for the same data (for example multiple database servers updating the single shared
data file).

See also

Cache-oblivious algorithm
Locality of reference
Distributed cache

References

1. Alan Jay Smith. "Design of CPU Cache Memories". Proc. IEEE TENCON, 1987. [1] (http://www.eecs.berkeley.edu
/Pubs/TechRpts/1987/CSD-87-357.pdf)
2. Paul V. Bolotoff. "Functional Principles of Cache Memory" (http://alasir.com/articles/cache_principles/). 2007.
3. http://www.vldb.org/conf/1994/P439.PDF

4/18/2016 12:22 PM

Cache algorithms - Wikipedia, the free encyclopedia

5 of 5

https://en.wikipedia.org/wiki/Cache_algorithms

4. O'Neil, Elizabeth J.; O'Neil, Patrick E.; Weikum, Gerhard (1993). "The LRU-K Page Replacement Algorithm for
Database Disk Buffering". Proceedings of the 1993 ACM SIGMOD International Conference on Management of
Data. SIGMOD '93 (New York, NY, USA: ACM): 297306. doi:10.1145/170035.170081. ISBN 0-89791-592-5.
5. Hong-Tai Chou and David J. DeWitt. An Evaluation of Buffer Management Strategies for Relational Database
Systems. (http://www.vldb.org/conf/1985/P127.PDF) VLDB, 1985.
6. Shaul Dar, Michael J. Franklin, Bjrn r Jnsson, Divesh Srivastava, and Michael Tan. Semantic Data Caching and
Replacement. (http://www.vldb.org/conf/1996/P330.PDF) VLDB, 1996.
7. ARM Cortex-R series processors manual (http://infocenter.arm.com/help/topic/com.arm.doc.set.cortexr/index.html)
8. An Efficient Simulation Algorithm for Cache of Random Replacement Policy [2] (http://www.springerlink.com/index
/L324G2U075540681.pdf)
9. Ramakrishna Karedla, J. Spencer Love, and Bradley G. Wherry. Caching Strategies to Improve Disk System
Performance. In Computer, 1994.
10. Nimrod Megiddo and Dharmendra S. Modha. ARC: A Self-Tuning, Low Overhead Replacement Cache.
(http://www.usenix.org/events/fast03/tech/full_papers/megiddo/megiddo.pdf) FAST, 2003.
11. Yuanyuan Zhou, James Philbin, and Kai Li. The Multi-Queue Replacement Algorithm for Second Level Buffer
Caches. (http://static.usenix.org/event/usenix01/zhou.html) USENIX, 2002.
12. Eduardo Pinheiro , Ricardo Bianchini, Energy conservation techniques for disk array-based servers, Proceedings of
the 18th annual international conference on Supercomputing, June 26-July 01, 2004, Malo, France
13. Cheng Li, Philip Shilane, Fred Douglis and Grant Wallace. Pannier: A Container-based Flash Cache for Compound
Objects. (http://dl.acm.org/citation.cfm?id=2814734) ACM/IFIP/USENIX Middleware, 2015.

External links
Definitions of various cache algorithms (http://www.usenix.org/events/usenix01/full_papers
/zhou/zhou_html/node3.html)
Fully associative cache (http://www.cs.umd.edu/class/spring2003/cmsc311/Notes/Memory/fully.html)
Set associative cache (http://www.cs.umd.edu/class/spring2003/cmsc311/Notes/Memory/set.html)
Direct mapped cache (http://www.cs.umd.edu/class/spring2003/cmsc311/Notes/Memory/direct.html)
Slides on various page replacement schemes including LRU (http://www.powershow.com/view/95163NzkyO/4_4_Page_replacement_algorithms_powerpoint_ppt_presentation)
Caching algorithm for flash/SSDs (http://dl.acm.org/citation.cfm?id=2814734)
Python DiskCache Library (http://www.grantjenks.com/docs/diskcache/) - An implementation of leastrecently-used and least-frequently-used cache eviction policies in Python.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Cache_algorithms&oldid=711914548"
Categories: Cache (computing) Memory management algorithms
This page was last modified on 25 March 2016, at 17:57.
Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may
apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia is a registered
trademark of the Wikimedia Foundation, Inc., a non-profit organization.

4/18/2016 12:22 PM