23CST302-COMPUTER ARCHITECTURE

NOTES

Module II- Cache Design and Pipelining

The Memory System:
Characteristics of Memory Systems - Cache Memory Principles-- Elements of
Cache Design - Mapping Function - Example of Mapping Technique -
Replacement Algorithms - Performance Consideration.

CHARACTERISTICS OF COMPUTER MEMORY

Characteristics of Computer Memory

Location: either internal or external to the processor.
• Forms of internal memory: registers; cache; and others;
• Forms of external memory:
• disk; magnetic tape
• devices that are accessible to the processor via I/O controllers.
Capacity: amount of information the memory is capable of holding.
• Typically expressed in terms of bytes (1 byte = 8 bits) or words
• A word represents each addressable block of the memory
• Common word lengths are 8, 16, and 32 bits; Word length = number of bits
used to represent an integer
• External memory capacity is typically expressed in terms of bytes
Unit of transfer: number of bytes read / written into memory at a time.
• Need not equal a word or an addressable unit
• Addressable unit – Word / byte
• If length in bits of an address is A then number of addressable units N = 2A
• Also possible to transfer blocks:
• Sets of words;
• Used in external memory as External memory is slow
• Idea: minimize number of accesses, optimize amount of data transfer
Access Method
• Sequential Method: Memory is organized into units of data, called records.
• Access must be made in a specific linear sequence;
• Stored addressing information is used to assist in the retrieval process.
• A shared read-write head is used
• The head must be moved from its one location to the another passing and
rejecting each intermediate record
• Access time varies

Magnetic Tape

Direct Access Memory:

• Involves a shared read-write mechanism;
• Difference - Individual records have a unique address;
• Requires accessing general record vicinity plus sequential searching,
counting, or waiting to reach the final location;
• Access time is also variable

Access Method
Random Access: Each addressable location in memory has a unique, physically
wired-in addressing mechanism.
• Constant time
• Independent of the sequence of prior accesses
• Any location can be selected at random and directly accessed
• Main memory and some cache systems are random access.
Associative: RAM that enables one to make a comparison of desired bit locations
within a word for a specified match
• Word is retrieved based on a portion of its contents rather than its address
• Retrieval time is constant independent of location or prior access patterns
Performance
• Access time (latency)
• For RAM: time to perform a read or write operation
• Others: time to position the read-write head at desired location
• Memory cycle time: Primarily applied to RAM
• Access time + additional time required before a second access
• Required for electrical signals to be terminated/regenerated
• Concerns the system bus.
• Transfer time: Rate at which data can be transferred in/out of memory
• For RAM: 1 / Cycle time
For Others:
Tn : Average time to read or write n bits;
TA : Average access time;
n: Number of bits
R: Transfer rate, in bits per second (bps)

Physical characteristics
• Volatile: information decays naturally or is lost when powered off;
• Nonvolatile: information remains without deterioration until changed:
• no electrical power is needed to retain information;
• E.g.: Magnetic-surface memories are nonvolatile;
• Semiconductor memory (memory on integrated circuits) may be either
volatile or non-volatile.

Memory Hierarchy
• Design constraints on memory can be summed up by three questions:
• How much?
• If memory exists, applications will likely be developed to use it.
• How fast?
• Best performance achieved when memory keeps up with the processor i.e. as
the processor executes instructions, memory should minimize pausing /
waiting for instructions or operands.
• How expensive?
• Cost of memory must be reasonable in relationship to other components;

Memory Hierarchy
• Trade-off among 3 characteristics: Capacity, Access time and Cost
• Faster access time, greater cost per bit
• Greater capacity – smaller cost per bit
• Greater capacity – slower access time
• Conclusion – Use a memory hierarchy instead of a single type of memory
• Supplement smaller, more expensive, faster memories with Larger, cheaper,
slower memories

As one goes down the hierarchy:

• Decreasing cost per bit;
• Increasing capacity;
• Increasing access time;
• Decreasing frequency of access of memory by processor
CACHE MEMORY
Need for Cache Memory
• Space and Time locality of reference principle:
• Space:
• if we access a memory location, close by addresses will very likely be
accessed soon
• Time:
• if we access a memory location, we will very likely access it again;
• This is a consequence of using iterative loops and subroutines
• instructions and data will be accessed multiple times

Example
• Suppose that the processor has access to two levels of memory:
• Level 1 - L1:
• contains 1000 words and has an access time of 0.01µs;
• Level 2 - L2:
• contains 100,000 words and has an access time of 0.1µs.
• Assume that:
• if word ∈ L1, then the processor accesses it directly;
• If word ∈ L2, then word is transferred to L1 and then accessed by the
processor.

For simplicity:
Ignore time required for processor to determine whether word is in L1 or L2. Also,
let:
• H define the fraction of all memory accesses that are found in L1
• T1 is the access time of L1
• T2 is the access time of L2
• Now consider the following scenario:
• Suppose 95% of the memory accesses are found in L1.
• Average time to access a word is:
• (0.95)(0.01µs) + (0.05)(0.01µs + 0.1µs) = 0.0095 + 0.0055 = 0.015µs
• Average access time is much closer to 0.01µs than to 0.1µs, as desired.
Example
General shape of the curve that covers this situation:

For high percentages of L1 access, the average total access time is much closer to
that of L1 than that of L2
Figure: Performance of accesses involving only L1
Example
• Strategy to minimize access time should be:
• Organize data across the hierarchy such that
• % of accesses to lower levels is substantially less than that of upper levels
• i.e. L2 memory contains all program instructions and data:
• Data that is currently being used should be in L1;
• Eventually:
• Data ∈ L1 will be swapped to L2 to make room for new data
• On average, most references will be to data contained in L1.
Example
• This principle can be applied across more than two levels of memory:
• Processor registers:
• Fastest, smallest, and most expensive type of memory
• Followed immediately by the cache:
• Stages data movement between registers and main memory;
• Improves perfomance;
• Is not usually visible to the processor;
• Is not usually visible to the programmer.
• Followed by main memory:
• Principal internal memory system of the computer;
• Each location has a unique address.

Cache Memory Principles

• Cache memory is designed to combine :
• Memory access time of expensive, high-speed memory combined with...
• ...the large memory size of less expensive, lower-speed memory.

Figure: Cache and main memory - single cache approach

When the processor attempts to read a word of memory:
• Check is made to determine if the word is in the cache
• If so - Cache Hit - word is delivered to the processor.
• If the word is not in cache -Cache Miss -
• Block of main memory is read into the cache;
• Word is delivered to the processor.
• Because of the locality of reference principle:
• When a block of data is fetched into the cache...
• ...it is likely that there will be future references to that same memory location
Way of improving the cache concept
• What if we introduce multiple levels of cache?
• L2 cache is slower and typically larger than the L1 cache
• L3 cache is slower and typically larger than the L2 cache.

Figure: Cache and main memory - three-level cache organization

ELEMENTS OF CACHE MEMORY
Few basic design elements that serve to classify and differentiate cache architectures.
They are listed down:
• Cache Size
• Block Size
• Mapping Function
• Replacement Algorithm
• Write Policy
Cache Size:
It seems that moderately tiny caches will have a big impact on performance.

Block Size:
Block size is the unit of information changed between cache and main memory.

Mapping Function:
When a replacement block of data is scan into the cache, the mapping performs
determines that cache location in which the block will occupy.

Replacement Algorithm:
The replacement algorithmic rule chooses, at intervals, the constraints of the mapping
perform, which block to interchange once a replacement block is to be loaded into the
cache and also the cache already has all slots full of alternative blocks.

Write Policy:
• If the contents of a block within the cache square measure altered, then it’s
necessary to write down it back to main memory before exchange it.
• The written policy dictates once the memory write operation takes place. At
one extreme, the writing will occur whenever the block is updated. At the
opposite extreme, the writing happens only if the block is replaced.
• The latter policy minimizes memory write operations however leaves the main
memory in associate obsolute state.
• This can interfere with the multiple-processor operation and with direct
operation by I/O hardware modules.
Cache Mapping

Cache has fewer lines than main memory blocks

Mapping function is needed to map main memory blocks into cache lines.
• Three techniques can be used for mapping blocks into cache lines:
o Direct
o Associative
o Set associative
Direct Mapping
Maps each block of main memory into only one possible cache line as: i = j mod m
where:
• i = cache line number;
• j = main memory block number;
• m = number of lines in the cache
Direct Mapping

• First m main memory blocks map into each line of the cache;
• Next m blocks of main memory map in the following manner:
o Bm maps into line L0 of cache;
o Bm+1 maps into line L1; …
• Modulo operation implies repetitive structure;
• Each line can have a different main memory block
• We need the ability to distinguish between these
• Most significant bits - the tag, serve this purpose

Direct Mapping- Cache blocks mapping

Each main memory address (s + w bits) can be viewed as:

• Block (s bits): identifies the memory block;
• Offset (w bits): identifies a word within a block of main memory;
If the cache has m = 2r lines:
• Line (r bits): specify one of the 2r cache lines;
• Tag (s − r bits): to distinguish blocks that are mapped to the same line;
Direct Mapping- Concepts

• The number of addressable units = 2s+w words or bytes

 • The block size (cache line width not including tag) = 2w words or bytes
• Line size = Block size

• The number of blocks in main memory = 2s

• The number of lines in cache = m = 2r
• The size of the tag stored in each line of the cache = (s - r) bits
• Why tag needs only s-r bits? Cache lines 2r ≪ 2s blocks of memory;

Direct Mapping- Example

Tag t= s-r bits Line identifier r bits Word w bits

8 14 2

In the example:
• 24 bit address (main memory)
• w = 2 bit word identifier (22= 4 Bytes in a block)
• s = (s-r)+r
• =(8)+14
• =22 bit block identifier
• r = 14 bit line identifier
• Tag =22-14 (s-r= 8 bits)
• Number of Cache lines = 214 = 16384
Direct Mapping- Organization of Cache
Direct Mapping- Cache hit or Miss

• To determine whether a block is in the cache:

• 1 Use the line field of the memory address to index the cache line;
• 2 Compare the tag from the memory address with the line tag;
o If both match, then Cache Hit: Retrieve the corresponding word from
the cache line;
o If both do not match, then Cache Miss: Fetch and Update the cache
line (word + tag);

Direct Mapping- Example 1

Consider a direct mapped cache of size 16 KB with block size 256 bytes. The size of
main memory is 128 KB. Find- Number of bits in tag and Tag directory size?

Number of bits in tag = Number of bits in physical address – (Number of bits in line
number + Number of bits in block offset)
Given-
Cache memory size = 16 KB
Block size = Line size = 256 bytes
Main memory size = 128 KB

Step 1:
• Main memory = 128 KB = 217 bytes
• Thus, Number of bits in physical address = 17 bits (s+w)
Step 2:
• Block size = 256 bytes = 28 bytes
• Number of bits to access a word in a block = 8 bits
• w = 8 bits

Given:Cache memory size = 16 KB; Block size = Line size = 256 bytes
Main memory size = 128 KB
Calculated: s + w = 17; w = 8

Step 3:
Total number of lines in cache = Cache size / Line size
= 16 KB / 256 bytes
= 214 bytes / 28 bytes
= 26 lines
So r = 6 bits

Step 4:
Number of bits in tag = (s+w)– (r+ w)
= 17 bits – (6 bits + 8 bits)
= 17 bits – 14 bits
= 3 bits

Thus, Number of bits in tag = 3 bits

Given:Cache memory size = 16 KB; Block size = Line size = 256 bytes
Main memory size = 128 KB
Calculated: s + w = 17; w = 8; r = 6; # of tag bits = 3

Step 5:
Tag directory size
= Number of tags x Tag size
= Number of lines in cache x Number of bits in tag
= 26 x 3 bits = 192 bits
= 24 bytes ( 8 bits= 1 byte)
Thus, size of tag directory = 24 bytes

Direct Mapping- Example 2

Consider a direct mapped cache of size 512 KB with block size 1 KB. There are 7 bits
in the tag. Find-1.Size of main memory. 2.Tag directory size.

Given:
Cache memory size = 512 KB
Block size = Line size = 1 KB
Number of bits in tag = 7 bits
We consider that the memory is byte addressable.
Number of Bits in Block Offset-
Block size = 1 KB = 210 bytes
Number of bits in block offset = 10 bits

Number of Bits in Line Number-

Total number of lines in cache = Cache size / Line size
= 512 KB / 1 KB
= 29 lines
Thus, Number of bits in line number = 9 bits
Number of Bits in Physical Address-
Number of bits in physical address= Number of bits in tag + Number of bits in line
no. + Number of bits in block offset
= 7 bits + 9 bits + 10 bits
= 26 bits
Thus, Number of bits in physical address = 26 bits

Size of Main Memory-

Number of bits in physical address = 26 bits
Thus, Size of main memory
= 226 bytes
= 64 MB

Tag Directory Size-

= Number of tags x Tag size
= Number of lines in cache x Number of bits in tag
= 29 x 7 bits
= 3584 bits
= 448 bytes
Thus, size of tag directory = 448 bytes

Direct Mapping- Pros and Cons

• Advantage: simple and inexpensive to implement

• Disadvantage: There is a fixed cache location for any given block;
o If a program happens to reference words repeatedly from two different
blocks that map into the same line, then the blocks will be continually
swapped in the cache;
o Hit ratio will be low .

ASSOCIATIVE MAPPING
In full associative mapping, any block can go into any line of the cache.

Figure: Associative mapped caches

Full Associative Mapping
(Associative Mapping)
• Permits each block to be loaded into any cache line
• Cache interprets a memory address as a Tag and a Word field:
• Tag: (s bits) uniquely identifies a block of main memory
• Word: (w bits) uniquely identifies a word within a block

Tag Word id
s bits w bits

Associative Mapping- Organization

Figure: Associative mapped caches (Source: [Stallings, 2015])

Associative Mapping- Cache hit or miss

• To determine whether a block is in the cache: simultaneously compare every
line’s tag for a match.
• If a match exists, then Cache Hit:
• Use the tag field of the memory address to index the cache line
• Retrieve the corresponding word from the cache line;
• If a match does not exist, then Cache Miss:
• Fetch the required block
• Choose a cache line for replacement
• Update the cache line (word + tag)
Associative Mapping- Example 1
Consider a fully associative mapped cache of size 16 KB with block size 256 bytes.
The size of main memory is 128 KB. Find- Number of bits in tag and Tag directory
size.

Given:
Cache memory size = 16 KB
Block size = Frame size = Line size = 256 bytes
Main memory size = 128 KB
Main memory = 128 KB = 217 bytes
Number of bits in physical address = 17 bits
Number of Bits in Block Offset-
Block size = 256 bytes = 28 bytes
Thus, Number of bits in block offset = 8 bits
Tag/Block number Block offset
8

17 bits
Number of bits in tag
= Number of bits in physical address – Number of bits in block offset
= 17 bits – 8 bits
= 9 bits

Tag/Block number Block offset

9 8

Tag directory size

= Number of tag bits x Number of Cache lines
Total number of lines in cache
= Cache size / Line size
= 16 KB / 256 bytes
= 214 bytes / 28 bytes
= 26 lines
Tag directory size
= Number of tags x Tag size
= Number of lines in cache x Number of bits in tag
= 26 x 9 bits = 576 bits
= 72 bytes (8 bits= 1 byte)

Associative Mapping- Pros and Cons

Advantage
• Flexibility as to which block to replace when a new block is read into the
cache
Disadvantage
• Complex circuitry required to examine the tags of all cache lines in
parallel

SET ASSOCIATIVE MAPPING

Combination of direct and associative approaches
Cache consists of a number of sets, each consisting of a number of lines.
From direct mapping:
• Each block can only be mapped into a single set;
• i.e. Block Bj always maps to a particular set
From associative mapping:
• Each block can be mapped into any cache line of a particular set.

The relationships are:

• m=v×k
 • i = j mod v
• i = cache set number
• j = main memory block number
• m = number of lines in the cache
• v = number of sets
• k = number of lines in each set(k-way Associative)

Figure: v associative mapped caches

Idea:
1 memory block → 1 single set, but to any row of that set.Can be physically
implemented as v associative caches.

• Cache interprets a memory address as a Tag, a Set and a Word field:

o Word: identifies a word within a block
o Set: identifies a set (d bits, v = 2d sets)
o Tag: used in conjunction with the set bits to identify a block (s − d bits)

Tag Set Word

s-d bits d bits W bits

Set Associative Mapping - Organization

Figure: k way set associative mapped caches organization

Set Associative Mapping
Cache hit or Miss

o To determine whether a block is in the cache:

o Determine the set through the set fields
o Compare address tag simultaneously with all cache line tags
o If a match exists, then Cache Hit: Retrieve the corresponding word from the
cache line;
o If a match does not exist, then Cache Miss: Fetch the required block; Choose a
cache line for replacement and Update the cache line (word + tag)

A 4-way set associative mapped cache has block size of 4 KB. The size of main
memory is 16 GB and there are 10 bits in the tag. Find the size of cache memory and
the tag directory size.

Size of main memory = 16 GB = 2^34 bytes

Number of bits in address = 34 bits
Number of bits is block offset w = 12 as 2^12 = 4KB
Number of bits in set field = 34 – 12 – 10 = 12
Cache size = 4 KB x 4 x 2^12 = 2 ^26 = 64 MB
Tag directory size = 10 x Number of cache lines = 10 x 2^14 = 163840 bits or
20840bytes.

Set associative cache – Example 1

• Consider a set-associative cache consisting of: 64 lines divided into four-line
sets; Main memory contains 4K blocks of 128 words each;Find
• How many bits are required for encoding words, sets and tag?
• What is the format of main memory addresses?
• Given:
• Main memory has 4K blocks
• Main memory block size = 128 words
• Each cache set has 4 lines
• Total cache lines = 64
Number of lines in memory address
• 4K blocks of 128 words
• 4K = 212
• 128 = 27
• So 19 bits are required to identify 4K blocks of 128 words
Number of bits to identify a word
• Each block contains 128 words
• 128 = 27
• So 7 bits are required to identify 128 words
• Number of bits to identify a Set
• Each set contains four lines
• Cache has 64 lines in total
• Number of Sets = 64 / 4 = 16
• 16 = 24
• So 4 bits are required to identify 16 Sets
• Main memory address size = 19 bits
• Number of bits to identify a word = 7 bits
 • Number of bits to identify a Set = 4 bits
Number of Tag bits
= 19 – 7 – 4
= 8 bits
Set associative cache – Example 2
• Consider a 2-way set associative mapped cache of size 16 KB with block size
256 bytes. The size of main memory is 128 KB. Find Number of bits in tag
and Tag directory size
• Given:
• Main memory size = 128 KB
• Cache Size = 16 KB
• Block size = 256 bytes
• Number of lines per Cache set = k = 2
• Number of lines in memory address
• 128 KB
• 128 = 27
• 1 KB = 210
• So 17 bits are required to address main memory
• Number of bits to identify a word
• Block size = 256 bytes
• 256 = 28
• So 8 bits are required to identify 256 bytes (Byte-addressable)
• Number of bits to identify a Set
• Cache size = 16 KB = 24 . 210
• Number of Cache lines = Cache size / Block size
14 8 6
=2 /2 =2
• Number of Sets = Number of Cache lines / number of lines per set
= 26 / 2 = 25
• So 5 bits are required to identify 2 5 sets
• Main memory address size = 17 bits
• Number of bits to identify a word = 8 bits
• Number of bits to identify a Set = 5 bits
Number of Tag bits
= 17 – 8 – 5
= 4 bits
• Tag directory size = 4 x Number of Cache lines
= 4 x 26 = 4 x 64 = 256 bits
= 32 bytes

Set associative cache – Example 3

• Consider a 8-way set associative mapped cache of size 512 KB with block size
1 KB. There are 7 bits in the tag. Find Size of main memory and Tag
directory size.
• Given:
• Cache Size = 512 KB
• Block size = 1 KB
• Number of lines per Cache set = k = 8
• Number of tag bits = 7
• Number of bits to identify a word
• Block size = 1 KB
 • 1 KB = 210
• So 10 bits are required to a word within a block
• Number of bits to identify a Set
• Cache size = 512 KB = 2 9 . 210
• Number of Cache lines = Cache size / Block size
= 219 / 210 = 29
• Number of Sets = Number of Cache lines / number of lines per set
= 29 / 8 = 26
• So 6 bits are required to identify 2 6 sets
• Number of bits to identify a word = 10 bits
• Number of bits to identify a Set = 6 bits
• Number of Tag bits = 7 bits
• Main memory address size = 23 bits
• Main memory size = 223 = 8 GB
• Tag directory size = 7 x Number of Cache lines
= 7 x 29 = 7 x 512 = 3584 bits
= 448 bytes

Set Associative Mapping- Pros and Cons

Advantages of Set-Associative mapping

• Set-Associative cache memory has highest hit-ratio compared two previous
two cache memory discussed.
• So performance is considerably better.
Disadvantages of Set-Associative mapping
• Set-Associative cache memory is very expensive.
• As the set size increases the cost increases. More space is needed for the tag
field
• A replacement algorithm must be used to determine which line of cache to
swap out.

Cache Performance
Varying Associativity over Cache Size

• 2-way is significantly better than direct

• Beyond 32kB increase in cache size brings no significant increase in
performance.
Comparison between the three schemes
Direct Mapping Fully Associative Set associative Mapping
Mapping
The simplest technique, The associative memory is Set associative addresses
known as direct mapping, used to store content and the problem of possible
maps each block of main addresses of the memory thrashing in the direct
memory into only one word. Any block can go mapping method.
possible cache line into any line of the cache.
Direct mapping’s Enables placement of any Set associative cache
performance is directly word at any place in the mapping combines the best
proportional to the Hit cache memory. It is of direct and associative
ratio. considered to be the fastest cache mapping techniques
and the most flexible
mapping form.
Main memory= s + word Main memory= s + word Main memory= s + word
offset offset offset
Cache memory=(tag+ line Cache memory=(tag)+ Cache memory=(tag+ set
offset )+ word offset word offset offset )+ word offset

CACHE REPLACEMENT ALGORITHMS

Need for Replacement Algorithms

• Eventually cache will fill and blocks will need to be replaced.
For direct mapping, there is only one possible line for any particular block – So no
choice is possible
For the associative and set-associative mapping techniques:
• A policy is required to keep the block in the cache, when it is likely to be
referenced in near future.
• A replacement algorithm is needed to suggest the block to be replaced if all
the cache lines are occupied.
Replacement Algorithms
• Most commonly used replacement algorithms are:
o Least recently used (LRU)
o First in first out (FIFO)
o Least frequently used(LFU)
o Random line replacement

LRU Algorithm
• Replace block in the set that has been in the cache longest, with no references
to it.
o Maintains a list of indexes to all the lines in the cache:
o Whenever a line is used move it to the front of the list;
o Choose the line at the back of the list when replacing a block.
 • LRU replacement can be implemented by attaching a number to each CM
block to indicate how recent a block has been used.
o Every time a CPU reference is made all of these numbers are updated
in such a way that the smaller a number the more recent it was used,
i.e., the LRU block is always indicated by the largest number.
LRU Replacement -Example
Block L1 L2 L3 L4 Status
5 5 Miss
4 5 4 Miss
6 5 4 6 Miss
3 5 4 6 3 Miss
4 5 4 6 3 Hit
0 0 4 6 3 Miss
2 0 4 2 3 Miss
5 0 4 2 5 Miss
3 0 3 2 5 Miss
0 0 3 2 5 Hit
6 0 3 6 5 Miss
7 0 3 6 7 Miss
11 0 11 6 7 Miss
3 3 11 6 7 Miss
5 3 11 5 7 Miss

Consider 4 block cache with following main memory references: 5,4,

6,3,4,0,2,5,3,0,6,7,11,3,5. Identify the hit ratio with the given memory requests.

Hit ratio = 2 / 15 = 0.133

LRU Algorithm - Example

Given 8 Cache lines and the following reference string trace the block placement
in Cache: 4,3,25,8,19,6,25,8,16,35,45,22,7

Cache block number Data in the block

0 LRU Block
4 45
1 LRU Block
3 22
2 25 H
3 8 H
4 LRU Block
19 7
5 6
6 16
7 35

Disadvantages
1. High latency to evict an unused cache line.
2. It does not consider 'frequency' and 'spatial locality'.
FIFO Replacement
First-in-first-out (FIFO):
o Replace the block in the set that has been in the cache longest:
o Regardless of whether or not there exist references to the block;
o Easily implemented as a round-robin or circular buffer technique

FIFO Replacement -Example

Block L1 L2 L3 L4 Status
5 5 Miss
4 5 4 Miss
6 5 4 6 Miss
3 5 4 6 3 Miss
4 5 4 6 3 Hit
0 0 4 6 3 Miss
2 0 2 6 3 Miss
5 0 2 5 3 Miss
3 0 2 5 3 Hit
0 0 2 5 3 Hit
6 0 2 5 6 Miss
7 7 2 5 6 Miss
11 7 11 5 6 Miss
3 7 11 3 6 Miss
5 7 11 3 5 Miss
Consider 4 block cache with following main memory references: 5,4, 6,3,4,0,
2,5,3,0,6,7,11,3,5. Identify the hit ratio with the given memory requests.
Hit ratio = 3 / 15 = 0.2

Disadvantages:
o Not always good at performance.

LFU
Least frequently used (LFU):
o Replace the block in the set that has experienced the fewest references;
o Implemented by associating a counter with each line.

LFU cache elimination Process

LFU Replacement -Example
Block L1 L2 L3 L4 Status
5 5 Miss
0 5 0 Miss
1 5 0 1 Miss
3 5 0 1 3 Miss
2 2 0 1 3 Miss
4 2 4 1 3 Miss
1 2 4 1 3 Hit
0 2 4 1 0 Miss
5 5 4 1 0 Miss

Consider 4 block cache with following main memory references: 5,0, 1,3,2,4, 1,0, 5.
Identify the hit ratio with the given memory requests.

Hit ratio = 1 / 9 = 0.11

LFU Disadvantages
o A separate counter is needed.

PERFORMANCE CONSIDERATIONS

• Two key factors in the commercial success of a computer are performance and
cost.
• Objective: Best possible performance for a given cost
• A common measure of success is the
 price/ performance ratio
• The extent to which cache improves performance is dependent on how
frequently the requested instructions and data are found in the cache

Hit Rate and Miss Penalty

• Hit rate: Number of hits stated as a fraction of all attempted accesses
• Miss rate: Number of misses stated as a fraction of attempted accesses.
• Miss penalty: Total access time seen by the processor when a miss occurs
• Hit rate of 90% or more ensures good performance

Average access time for cache

• In a system with only one level of cache, miss penalty consists almost entirely
of the time to access a block of data in the main memory.
• Given: h - hit rate, M - miss penalty, C - time to access information in the
cache.
• Average access time experienced by the processor is
tavg = hC + (1 − h)M

Hit Rate Improvement

• One possibility is to make the cache larger, but this leads to increased cost.
• Another possibility is to increase the cache block size while keeping the total
cache size constant, to take advantage of spatial locality.
• High data rate is achievable during large block transfers.
 • Larger blocks take longer to transfer, and hence increase the miss penalty
• Block size should be neither too small nor too large.
• Block sizes in the range of 16 to 128 bytes are the most popular choices.

Miss Penalty Reduction

• The miss penalty can be reduced if the load-through approach is used when
loading new blocks into the cache.
• Instead of waiting for an entire block to be transferred, the processor resumes
execution as soon as the required word is loaded into the cache.

Caches on the Processor Chip

• Most processor chips include at least one L1 cache. Often there are two
separate L1 caches, one for instructions and another for data.
• In high-performance processors, two levels of caches are normally used,
separate L1 caches for instructions and data and a larger L2 cache.
• In this case, the L1 caches must be very fast, as they determine the memory
access time seen by the processor.
• The L2 cache can be slower, but it should be much larger than the L1 caches
to ensure a high hit rate.

Average access time

Average access time experienced by the processor in such a system is:
tavg = h1C1 + (1 − h1)(h2C2 + (1 − h2)M)
where
h1 is the hit rate in the L1 caches.
h2 is the hit rate in the L2 cache.
C1 is the time to access information in the L1 caches.
C2 is the miss penalty to transfer information from the L2 cache to an L1 cache.
M is the miss penalty to transfer information from the main memory to the L2
cache.

Possibilities for improving the performance

o Write Buffer
o Prefetching
o Lock up free cache

Write Buffer – Write through

• When the write-through protocol is used, each Write operation results in
writing a new value into the main memory.
• If processor waits for the memory function to be completed, then the processor
is slowed down by all Write requests.
• Processor typically does not need immediate access to the result of a Write
operation.
• So it is not necessary for it to wait for the Write request to be completed.
• Write buffer - temporary storage of Write requests
• Processor places each Write request into this buffer and continues execution of
the next instruction
• Write requests stored in Write buffer are sent to the main memory whenever
the memory is not responding to Read requests.
 • Read requests must be serviced quickly, because the processor usually cannot
proceed
• Write buffer may hold a number of Write requests. Subsequent Read request
may refer to data that are still in the Write buffer.
• Addresses of data to be read from the memory are always compared with the
addresses of the data in the Write buffer. In the case of a match, the data in
the Write buffer are used.

Write Buffer – Write back

• When a new block of data is to be brought into the cache, it may replace an
existing block that has some dirty data.
• The dirty block has to be written into the main memory.
• If the required write-back is performed first, then the processor has to wait for
this operation to be completed before the new block is read into the cache.
• The dirty block being ejected from the cache is temporarily stored in the Write
buffer and held there while the new block is being read.
• Afterwards, contents of buffer are written into the main memory.

Prefetching
• New data is bought into the cache when it is first needed. Processor has to
pause until new data arrives - miss penalty.
• To avoid stalling the processor, it is possible to prefetch the data into the cache
before they are needed.
• A special prefetch instruction may be provided in the instruction set of the
processor.
• Executing this instruction causes the addressed data to be loaded into the
cache, as in the case of a Read miss but before they are needed in the program.
This avoids miss penalty.
• Hardware or Software (Compiler or Programmer)

Lockup-Free Cache
• While servicing a miss, the cache is said to be locked.
• This problem can be solved by modifying the basic cache structure to allow
the processor to access the cache while a miss is being serviced
• A cache that can support multiple outstanding misses is called lockup-free.
• Such a cache must include circuitry that keeps track of all outstanding misses.
• This may be done with special registers that hold the pertinent information
about these misses.
 PIPELINING
Basic concept - Pipeline Organization and issues - Data Dependencies –Memory
Delays – Branch Delays – Resource Limitations - Performance Evaluation -
Superscalar operation –Pipelining in CISC Processors - Instruction Level
Parallelism –Parallel Processing Challenges – Flynn’s Classification – Hardware
multithreading –Multicore Processors: GPU, Multiprocessor Network
Topologies.

PIPELINING
• Overlaps the execution of successive instructions to achieve high performance
• Example: Manufacture of a product involving 3 processes
Time 1 2 3 4 5 6 7 8 9
P1 A B C
P2 A B C
P3 A B C

Time 1 2 3 4 5
P1 A B C
P2 A B C
P3 A B C

Pipelining
The speed of execution of programs is influenced by many factors
• Using faster circuit technology to implement the processor and the main
memory
• Arranging the hardware so that more than one operation can be performed
at the same time – overall completion time is speeded up – individual
operation time remains the same

What is Pipelining?
• Pipelining is an implementation technique whereby multiple instructions
are overlapped in execution.
• Pipe stage (pipe segment)
• Commonly known as an assembly-line operation.
• Automobile Manufacturing

Idea of pipelining
• Original Five-stage processor organization allows instructions to be fetched
and executed one at a time
• Overlapping of instructions
Simple implementation of A RISC ISA

Five-cycle implementation
• Instruction fetch cycle (IF)
• Instruction decode/register fetch cycle (ID)
o Operand fetches
o Sign-extending the immediate field;
o Decoding is done in parallel with reading registers. This technique is
known as fixed-field decoding;
o Test branch condition and computed branch address; finished
branching at the end of this cycle.
• Execution/Compute (EX)
o Memory reference;
o Register-Register ALU instruction;
o Register-Immediate ALU instruction;
• Memory access/branch completion cycle (MEM)
• Write-back cycle (WB)
o Register-Register ALU instruction;
o Register-Immediate ALU instruction;
o Load instruction;

5 stage Pipeline

Interstage buffer B1 feeds the

Decode stage with a newly-
fetched instruction

Interstage buffer B2 feeds the

Compute stage with the two
operands read from the register
file, immediate value derived
from the instruction etc.

Interstage buffer B3 holds the

result of the ALU operation

Interstage buffer B4 feeds the

Write stage with a value to be
written into the register file
Pipelining Issues
• Consider the case of two instructions Ij and Ij+1,
o where the destination register for instruction Ij is a source register for
instruction Ij+1. Result of instruction Ij is not written into the register file
until cycle 5
o If execution proceeds, Ij+1 would be incorrect because the arithmetic
operation would be performed using the old value of the register
o To obtain the correct result, it is necessary to wait until the new value is
written into the register by instruction Ij
• Hence, instruction Ij+1 cannot read its operand until cycle 6, which means it
must be stalled in the Decode stage for three cycles.
• While instruction Ij+1 is stalled, instruction Ij+2 and all subsequent
instructions are similarly delayed.
• New instructions cannot enter the pipeline, and the total execution time is
increased.
• Any condition that causes the pipeline to stall is called a hazard.

Pipelining Hazards
• Hazard - situation that prevents the next instruction in the instruction stream
from executing during its designated clock cycle.
• Three Types of hazards
o Structural hazard: Arises from resource conflicts.
o Data hazard: Arises when an instruction depends on the results of a
previous instruction.
o Control hazard: Arises from branches and other instructions that change
the PC.
• A pipeline can be stalled by a hazard.

Data Dependencies
LD R3,0(R2) LD R1,0(R2)
DSUB R1,R2,R5 DSUB R4,R1,R5
AND R6,R1,R7 AND R6,R1,R7
OR R8,R1,R9 OR R8,R1,R9
XOR R8,R2,R4 XOR R8,R2,R4

Add R2, R3, #100

Subtract R9, R2, #30

Subtract instruction is stalled for three cycles to delay reading register R2 until cycle 6

Add R2, R3, #100

Subtract R9, R2, #30
 • Control circuit must first recognize the data dependency when decoding
Subtract instruction in cycle 3 by comparing its source register with
destination of Add
• Subtract instruction must be held in interstage buffer B1 during cycles 3 to 5.
• In cycles 3 to 5, as the Add instruction moves ahead, control signals can be set
in interstage buffer B2 for an implicit NOP (No-operation) instruction.
• Each NOP creates one clock cycle of idle time, called a bubble.

Operand Forwarding
Add R2, R3, #100
Subtract R9, R2, #30

Pipeline is stalled for 3 cycles – but required value is available at end of Cycle 3
Operand forwarding – instead of Stalling the pipeline required value is sent to ALU
in Cycle
Add R2, R3, #100
Subtract R9, R2, #30

Operand Forwarding – Datapath modification

New multiplexer, MuxA, is inserted before input InA of the ALU,and multiplexer MuxB is
expanded with another input.
Handling Data Dependencies in Software
Add R2, R3, #100
NOP
NOP
NOP
Subtract R9, R2, #30

Insertion of NOP instructions for a data dependency

 Handling Data Dependencies in Software
• Simplifies the hardware implementation of the pipeline.
• However, the code size increases, and the execution time is not reduced as it
would be with operand forwarding
• Compiler can attempt to optimize the code to improve performance and reduce
the code size by reordering instructions to move useful instructions into the
NOP slots.

MEMORY DELAY
• Delays arising from memory accesses are another cause of pipeline stalls
• Load instruction may require more than one clock cycle to obtain its operand
from memory.
• This may occur because the requested instruction or data are not found in the
cache, resulting in a cache miss. A memory access may take ten or more
cycles.

Stalling by 3 Cycles

• A cache miss causes all subsequent instructions to be delayed.

• A similar delay can be caused by a cache miss when fetching an instruction
• Additional type of memory-related stall occurs when there is a data
dependency involving a Load instruction.
o Load R2, (R3)
o Subtract R9, R2, #30
• Operand forwarding cannot be used because the data from Cache is not
available until it is loaded into register RY at the beginning of cycle 5.

• Subtract instruction must be stalled for one cycle

• Compiler can reorder instructions to avoid stall
• If a useful instruction cannot be found by the compiler, then the hardware
introduces the one-cycle stall automatically.
• If the processor hardware does not deal with dependencies, then the compiler
must insert an explicit NOP instruction.
Data Dependencies

No DEPENDENCE
DEPENDENCE OVERCOME
BY FORWARDING LD R1, 45(R2)
LD R1, 45(R2) DADD R5, R6, R7
DADD R5, R6, R7 DSUB R8, R6, R7
DSUB R8, R1, R7 OR R9, R6, R7
OR R9, R6, R7
DEPENDENCE REQUIRING STALL
LD R1, 45(R2)
DADD R5, R1, R7
DSUB R8, R6, R7
OR R9, R6, R7

DEPENDENCE WITH ACCESS IN ORDER

LD R1, 45(R2)
DADD R5, R6, R7
DSUB R8, R6, R7
OR R9, R1, R7

BRANCH DELAYS
• In an ideal pipeline a new instruction is fetched every cycle, while the
preceding instruction is still being decoded.
• Branch instructions can alter the sequence of execution but they must first be
executed to determine whether and where to branch
• The number of stalls introduced during branch operations in the pipelined
processor is known as branch penalty
• Various techniques can be used for mitigating impact of branch delays :
o Unconditional Branches
o Conditional Branches
o The Branch Delay Slot
o Branch Prediction
o Static Branch Prediction
o Dynamic Branch Prediction
o Branch Target Buffer for Dynamic Prediction
Unconditional Branches

• Ij – Branch instruction
• Ik – Branch target – computed only in Cycle 3
• So Ik is fetched in Cycle 4
• Two – cycle delay
Branch instructions represent about 20 % of the dynamic instruction count of most
programs.
Dynamic count – number of instruction executions – some instructions may get
executed multiple times.
Two-cycle branch penalty – increases execution time by nearly 40%.
Reducing the branch penalty requires the branch target address to be computed earlier
– Decode stage
• Decode stage: instruction decoder determines that the instruction is a branch
instruction
• Computed target address will be available before the end of the cycle 2

Datapath must be modified by placing an additional adder in Decode stage to compute

branch target.
Conditional Branches
o Branch_if_[R5]=[R6] LOOP
o The result of the comparison in the third step determines whether the branch is
taken.
o For pipelining, the branch condition must be tested as early as possible to limit
the branch penalty.
o The comparator that tests the branch condition can also be moved to the
Decode stage enabling the conditional branch decision to be made at the same
time that the target address is determined to limit the branch penalty.

Branch Delay Slot

• Location that follows a branch instruction is called branch delay slot.
• Assume that the branch target address and the branch decision are determined
in the Decode stage, at the same time that instruction Ij+1 is fetched.
• Branch instruction may cause instruction Ij+1 to be discarded, after the branch
condition is evaluated.
• If the condition is true, then there is a branch penalty of one cycle before the
correct target instruction Ik is fetched.
• If the condition is false, then instruction Ij+1 is executed, and there is no
penalty.

Add R7, R8, R9

Branch_if_[R3]=0 TARGET
Ij+1
..
..
TARGET: Ik

• Branch delay slot can be filled with a useful instruction which will be
executed irrespective of whether the branch is taken or not
• Move one of the instructions preceding the branch to the branch delay slot
• Logically, execution proceeds as though the branch instruction were placed
after the ADD instruction – Delayed branching
• If no useful instruction is found – NOP is placed and branch penalty of 1 is
incurred.

Add R7, R8, R9

Branch_if_[R3]=0 TARGET
Ij+1
..
..
TARGET: Ik

Branch_if_[R3]=0 TARGET
Add R7, R8, R9
Ij+1
..
..
TARGET: Ik
Branch Prediction
• To reduce the branch penalty further, the processor needs to anticipate that an
instruction being fetched is a branch instruction and predict its outcome to
determine which instruction should be fetched in cycle 2.
• Types of Branch Prediction
o Static Branch Prediction
o Dynamic Branch Prediction
o LT - Branch is likely to be taken
o LNT - Branch is likely not to be taken

Static Branch Prediction

• Simplest form of branch prediction which assumes that the branch will not be
taken and fetches the next instruction
• If the prediction is correct, the fetched instruction is allowed to complete and
there is no penalty.
• However, if it is determined that the branch is to be taken, the instruction that
has been fetched is discarded and the correct branch target instruction is
fetched.
• Mis-prediction incurs the full branch penalty.
• Backward branches at the end of a loop are taken most of the time. For such a
branch, better accuracy can be achieved by predicting that the branch is likely
to be taken
• For a forward branch at the beginning of a loop, the not-taken prediction leads
to good prediction accuracy
• Processor can determine the static prediction of taken or not-taken by
checking the sign of the branch offset.
• Alternatively, the machine encoding of a branch instruction may include one
bit that indicates whether the branch should be predicted as taken or nor taken
– compiler sets this bit

Dynamic Branch Prediction

• To improve prediction accuracy further, we can use actual branch behavior to
influence the prediction, resulting in dynamic branch prediction.
• Better prediction accuracy can be achieved by keeping more information about
execution history.
• The four states are:
o ST - Strongly likely to be taken
o LT - Likely to be taken
o LNT - Likely not to be taken
o SNT - Strongly likely not to be taken

2-State-machine representation of branch prediction algorithms

• Algorithm is started in state LNT.
• When branch instruction is executed and branch is taken, the machine moves
to state LT. Otherwise, it remains in state LNT.
• The next time the same instruction is encountered, the branch is predicted as
taken if the state machine is in state LT. Otherwise it is predicted as not taken.
• A single bit is used represent the history of execution for a branch instruction
• Works well for program loops
 • Once a loop is entered, the decision for the branch instruction that controls
looping will always be the same except for the last pass through the loop
• Last pass – wrong prediction and branch history changes to opposite state
LNT
• Wrong prediction for first time in next execution of same loop

4-State-machine representation of branch prediction algorithms

• Algorithm is started in state LNT.

• Only one wrong prediction in last execution of program loop

Branch Target Buffer

• Processor stores history of execution in a small, fast memory called the branch
target buffer.
• The information is organized in the form of a lookup table, in which each
entry includes:
o The address of the branch instruction
o One or two state bits for the branch prediction algorithm
o The branch target address
• The processor is able to identify branch instructions & obtain the
corresponding branch prediction state bits based on the address of the
instruction being fetched.
• Limited size table - contains information for only the most recently executed
branch instructions
Resource Limitation:

In the context of computer organization and parallel processing, resource limitations

refer to the finite availability of key system components like memory, processing power,
bandwidth, and storage. These limitations can significantly affect the performance and
scalability of a system, especially in complex, parallel, and distributed systems.

1. Processor Limitations

 Clock Speed: The clock speed of a processor, which defines how quickly it can
execute instructions, is often a limiting factor. Higher clock speeds lead to increased
power consumption and heat generation, which creates challenges for energy-efficient
designs.
 Instruction Throughput: As the demand for processing power increases, modern
processors use techniques like superscalar execution and pipelining to increase
instruction throughput. However, these techniques have their limits in terms of how
many instructions can be processed simultaneously without causing issues like
instruction dependencies or pipeline stalls.

2. Memory Limitations

 Cache Memory: The speed gap between the CPU and main memory (RAM) is a
significant bottleneck. Processors rely on caches (L1, L2, L3) to store frequently used
data for faster access. However, cache sizes are limited due to space and cost
constraints.
 Memory Hierarchy: Efficient memory usage depends on memory hierarchy design
(registers, cache, main memory, and storage). Larger memory hierarchies can lead to
increased power consumption, and managing data flow between different levels of
memory presents a design challenge, as well as issues like cache coherence in multi-
core systems.

3. Interconnect Limitations

 Communication Bandwidth: In parallel and distributed systems, communication

between processors, memory units, and other resources must be fast enough to avoid
bottlenecks. As the number of processors or devices increases, the interconnect
network’s bandwidth becomes a key limiting factor. Efficient interconnection
networks, such as crossbar switches, mesh networks, and ring networks, are
essential to overcome these limitations.
 Latency: The time it takes for data to travel from one processor to another can
become a major limitation. Communication latency is critical in parallel processing
systems, especially when using distributed memory (such as in multi-node systems
or GPU-based computing) where data has to be transferred over the network.

4. Storage Limitations

 Data Storage: Storage systems must accommodate large datasets, especially in

modern applications like big data processing and machine learning. Hard disk
drives (HDDs) and solid-state drives (SSDs) are commonly used, but storage
 bandwidth and access speeds are limited. More advanced storage systems like
distributed file systems and cloud storage provide solutions, but they also introduce
their own limitations in terms of access speed, reliability, and redundancy.
 Persistent Storage: For long-term data retention, systems use non-volatile memory
(e.g., SSDs). However, the rate at which data can be written to non-volatile storage is
generally slower compared to volatile memory (RAM), which impacts performance
during data-heavy operations.

5. Power and Energy Constraints

 Energy Consumption: Power efficiency is increasingly becoming a major concern in

modern processors. Techniques such as dynamic voltage and frequency scaling
(DVFS) are employed to manage energy consumption. However, high-performance
systems require significant power, and managing power consumption becomes more
difficult as systems scale up.
 Thermal Management: As processors become faster and more efficient, managing
heat dissipation becomes a limiting factor. Excessive heat can damage components
and slow down processing speeds (due to thermal throttling). Specialized cooling
systems, including fans, liquid cooling, and heat sinks, are used to maintain optimal
temperatures, but these systems add cost and complexity to hardware design.

6. Scalability Limitations

 Amdahl’s Law: One of the primary scalability limitations in parallel processing is

Amdahl’s Law, which suggests that the speedup gained from parallelizing a task is
limited by the portion of the task that remains sequential. As more processors are
added, the benefits of parallelism decrease if the system is bottlenecked by sequential
operations or communication overhead.
 Memory Scaling: The scalability of memory is a critical issue in multiprocessor
systems. As the number of processors increases, memory access contention can
increase unless systems employ efficient memory-sharing and distribution
mechanisms, like non-uniform memory access (NUMA).

7. Bandwidth-Delay Product

 Bandwidth-Delay Product (BDP) is a concept that refers to the product of a

network’s bandwidth and the latency of the connection between two nodes. High
bandwidth and low latency are essential for efficient communication in distributed
systems. In high-performance computing systems, optimizing the BDP is essential to
ensure that the system doesn’t suffer from communication bottlenecks as data transfer
scales up.

8. Software and Algorithmic Limitations

 Parallel Software Design: Efficient parallel computing requires software to be

designed to exploit available hardware resources. Writing software that efficiently
scales across multiple cores and processors is difficult, especially for legacy
applications not originally designed with parallelism in mind.
 Load Balancing Algorithms: The effectiveness of parallel systems also depends on
how well tasks are distributed across processors. Poor load balancing can result in
 some processors being idle while others are overloaded, reducing the overall
efficiency of the system.
 Synchronization: Effective synchronization is critical to avoid data races and ensure
correctness in parallel programs. However, excessive synchronization can lead to
bottlenecks and reduce the system’s overall performance.

9. Specialized Resource Limitations in GPUs

 GPU Memory: Graphics Processing Units (GPUs) are designed to handle highly
parallel workloads but are constrained by their local memory (often much smaller
than CPU memory). Techniques like memory paging and streaming
multiprocessors (SMs) help mitigate this issue, but resource limitations can still
affect the performance of GPU-based systems.
 Compute Units: While GPUs have many smaller cores that are optimized for parallel
execution, the total compute power is still constrained by the number of compute
units available, especially in workloads that do not fit well into the SIMD execution
model.

Performance Evaluation in Computer Systems

Performance evaluation in computer systems refers to assessing and quantifying how well a
system or component performs under specific workloads or conditions. This is essential for
understanding the efficiency, throughput, and overall capability of hardware and software.

1. Key Performance Metrics

1.1 Throughput

 Throughput refers to the amount of work a system can perform in a given period,
typically measured in terms of tasks completed, data processed, or instructions
executed per unit of time. For processors, this is often quantified as instructions per
cycle (IPC), operations per second (OPS), or flops (floating-point operations per
second).
 Parallel systems, throughput is particularly important in determining the system’s
capacity to handle multiple tasks or data streams simultaneously.

1.2 Latency

 Latency is the time taken to complete a single operation or task. It is especially

important in time-sensitive applications, such as real-time systems and interactive
computing. Low latency is desirable, as it minimizes the delay in system response.
 In multicore processors or distributed systems, latency can be affected by factors
like memory access time, data transfer time, and interprocessor communication
time. High latency can lead to bottlenecks that degrade overall system performance.

1.3 Execution Time

 Execution time refers to the total time a system takes to execute a given program or
workload. It is a direct measure of the time taken to perform operations. Execution
time can be broken down into:
 o CPU time: Time spent on processing.
o I/O time: Time spent waiting for input/output operations.
o Memory access time: Time spent waiting for data to be fetched from memory
or caches.

1.4 Speedup

 Speedup is a measure of how much faster a parallel system or algorithm performs

relative to a sequential system. It is calculated as:

 Amdahl’s Law describes the theoretical maximum speedup achievable in parallel

computing, which is limited by the fraction of the program that cannot be parallelized.

1.5 Efficiency

 Efficiency measures the utilization of the system’s resources relative to the maximum
possible utilization. In parallel systems, efficiency is often calculated as:

 High efficiency means that adding more processors leads to significant improvements
in performance, whereas low efficiency implies that resources are not being fully
utilized, often due to overheads such as synchronization or communication.
 CPI (Cycles Per Instruction): This metric reflects the number of clock cycles
required to execute a single instruction. A CPU with a lower CPI is more efficient in
processing instructions.
o CPI Calculation: The CPU's overall performance is linked to its CPI, which
is determined by the instruction set architecture, data hazards, and instruction
scheduling.
o Example: A CPU with a CPI of 2 will take 2 clock cycles to complete each
instruction on average.
 MIPS (Million Instructions Per Second): This measures the number of millions of
instructions a processor can execute in one second. However, MIPS alone does not
 provide an accurate performance measure because it doesn’t account for instruction
complexity.
o Formula:

 Benchmarks: Performance is also evaluated using industry-standard benchmarks.

These are representative workloads that help in comparing the efficiency of different
processors. Benchmarks can be specific to particular tasks such as SPEC (Standard
Performance Evaluation Corporation) benchmarks.

2. Benchmarks for Performance Evaluation

2.1 Synthetic Benchmarks

 Synthetic benchmarks are specifically designed tests that simulate specific aspects
of system performance, such as memory access patterns, processor throughput, or I/O
performance. These benchmarks help identify how well a system performs in isolated
tasks but may not reflect real-world application performance.
 Patterson & Hennessey mention that synthetic benchmarks can provide insights into
the raw capabilities of the system components but may fail to represent complex, real-
world workloads.

2.2 Application Benchmarks

 Application benchmarks evaluate system performance using real-world applications,

such as databases, scientific simulations, and web servers. These benchmarks reflect
the actual usage patterns of the system and provide a better approximation of how the
system performs in practical scenarios.
 For example, in high-performance computing, benchmarks like the LINPACK
benchmark are used to measure the floating-point computing power of
supercomputers.

2.3 SPEC Benchmarks

 The Standard Performance Evaluation Corporation (SPEC) provides standardized

benchmarks for evaluating processor performance, memory performance, and system
efficiency. The SPEC CPU benchmark is commonly used to assess the computational
power of CPUs.
 These benchmarks provide comparative metrics across different systems, helping
organizations choose the appropriate hardware based on their workload needs.
3. Performance in Multicore and Parallel Systems

3.1 Scalability and Amdahl’s Law

 Scalability is a crucial consideration when evaluating the performance of multicore

or multiprocessor systems. Scalability measures how well the system’s performance
improves as more processors or cores are added. However, the scalability of parallel
systems is often limited by Amdahl’s Law, which states that the maximum speedup
of a program using multiple processors is limited by the fraction of the program that
cannot be parallelized.
o Amdahl’s Law: If a portion of the program is sequential, no matter how many
processors are added, the speedup is limited by that sequential portion.

3.2 Load Balancing and Scheduling

 In parallel processing systems, load balancing is essential for efficient performance.

Uneven distribution of tasks can lead to some processors being overburdened while
others remain idle, leading to poor performance. Patterson & Hennessey discuss
load balancing algorithms like dynamic scheduling and static scheduling to ensure
that processors are kept busy with work evenly distributed.

3.3 Communication Overhead

 In distributed systems and multicore processors, communication overhead is

another performance bottleneck. This includes the time required for data to be
transferred between processors, cores, or nodes in a cluster. Effective communication
strategies, like direct memory access (DMA) or high-speed interconnects, are vital
to reducing overhead and improving performance.

4. Performance in Specialized Systems (GPUs, Accelerators)

4.1 Graphics Processing Units (GPUs)

 GPUs are highly parallel computing devices optimized for specific workloads, such
as graphics rendering and matrix computations. The performance evaluation of GPUs
often focuses on their ability to handle large amounts of data in parallel.
o Patterson & Hennessey discuss how GPU performance is typically
measured by the number of cores (thousands of small cores in GPUs),
memory bandwidth, and the efficiency of execution in parallel tasks.

4.2 Specialized Hardware Accelerators

 Other hardware accelerators, such as FPGA (Field-Programmable Gate Arrays) and

TPU (Tensor Processing Units), are used for specific computational tasks. Evaluating
their performance involves understanding the task-specific throughput, latency, and
energy efficiency of operations like deep learning or signal processing.
5. Energy Efficiency in Performance Evaluation

5.1 Power Consumption and Thermal Management

 As processors become more powerful, power consumption and thermal

management become significant factors affecting overall performance. High-
performance systems generate substantial heat, requiring effective cooling solutions.
 Energy efficiency is an increasingly important metric, especially in mobile devices,
cloud computing, and supercomputers. Evaluating performance also includes
measuring how much power is consumed for each task performed, with systems like
Green500 ranking supercomputers based on their energy efficiency.

5.2 Energy-Delay Product (EDP)

 The Energy-Delay Product (EDP) is a metric that evaluates the trade-off between
energy consumption and execution time. It is calculated as:

 The goal is to minimize EDP, balancing energy efficiency with performance to

achieve optimal system operation in terms of both speed and power consumption.

6. Performance Evaluation Techniques

6.1 Profiling and Tracing

 Profiling involves monitoring system behavior during program execution, collecting

data such as CPU usage, memory access patterns, and disk I/O activity. This data
helps identify bottlenecks and inefficiencies.
 Tracing goes beyond profiling to capture detailed information about specific events,
such as function calls or context switches, helping developers understand performance
at a fine-grained level.

6.2 Simulation and Modeling

 Simulation tools allow performance evaluation of computer systems before actual

deployment. Simulators model various aspects of hardware or software and predict
how the system will perform under different configurations or workloads.
 Modeling involves mathematical representation of the system to predict performance
metrics based on factors like resource usage, parallelism, and workload distribution.

6.3 Analytical Performance Models

 Analytical models use mathematical formulas to estimate system performance based

on known parameters. These models are often used to predict performance without
needing to conduct extensive experiments or simulations. Patterson & Hennessey
describe the use of models like the Queuing Theory and Little’s Law to analyze
system throughput and latency.
Performance evaluation is a critical aspect of designing and optimizing computer systems.
By measuring metrics such as throughput, latency, execution time, speedup, and
efficiency, system designers can assess the effectiveness of their designs.

Superscalar Operation in Computer Architecture:

A superscalar architecture refers to a type of processor design that allows multiple

instructions to be executed in parallel during a single clock cycle. This is achieved by having
multiple execution units within the processor, which can handle different types of operations
simultaneously. Superscalar operation is a significant enhancement over scalar processors,
where only one instruction is processed at a time per clock cycle.

1. Key Characteristics of Superscalar Architecture

1.1 Multiple Instruction Pipelines

 A superscalar processor has multiple instruction pipelines that can execute

different instructions concurrently. Each pipeline is specialized for specific types of
operations, such as integer arithmetic, floating-point operations, memory access, etc.
 For example, a processor might have two pipelines: one for integer operations and
one for floating-point operations. This design allows the processor to execute two
instructions simultaneously, provided that the instructions are independent of each
other.

1.2 Instruction Fetch and Dispatch

 In a superscalar system, the instruction fetch unit fetches multiple instructions from
memory in parallel, while the instruction dispatch unit dynamically assigns these
instructions to available pipelines.
 Modern superscalar processors can fetch and decode several instructions per cycle,
making use of techniques such as out-of-order execution to maximize instruction
throughput.

1.3 Execution Units

 Superscalar processors have several execution units (also called functional units),
each of which performs a specific type of operation. Common execution units in
superscalar processors include:
o ALUs (Arithmetic Logic Units) for integer operations.
o FPUs (Floating Point Units) for floating-point calculations.
o Load/Store units for memory operations.
 The ability to execute multiple instructions in parallel depends on having multiple
functional units and scheduling them appropriately.

1.4 Instruction-Level Parallelism (ILP)

  Instruction-level parallelism (ILP) refers to the ability to execute multiple
instructions from a program simultaneously. Superscalar processors exploit ILP by
identifying independent instructions that can be processed in parallel without data
dependencies.
 Data hazards (read-after-write, write-after-read) and control hazards (branch
instructions) can limit ILP, so advanced scheduling and dynamic execution
mechanisms are required to maximize parallelism.

2. How Superscalar Processors Work

2.1 Instruction Fetching and Decoding

 Superscalar processors fetch and decode multiple instructions in parallel, usually

from a cache. The instruction fetch unit grabs several instructions from memory in
each clock cycle.
 The instruction decode unit identifies which instructions are independent and can be
executed concurrently.

2.2 Instruction Dispatch

 After decoding, the instructions are dispatched to available execution units based on
their type (e.g., integer operations go to the ALU, floating-point operations go to the
FPU).
 The processor ensures that dependent instructions are scheduled in the correct order,
while independent instructions can be processed simultaneously.

2.3 Out-of-Order Execution

 Superscalar processors often support out-of-order execution. This means that the
processor can execute instructions as soon as their operands are available, rather than
strictly following the program’s sequential order.
 Dynamic scheduling techniques, such as the score boarding technique or
Tomasulo’s algorithm, help in reordering instructions to avoid pipeline stalls and
improve parallel execution.

2.4 Multiple Pipelines

 The presence of multiple pipelines enables the processor to execute different types of
instructions (such as integer and floating-point) concurrently. This further increases
throughput by making efficient use of different execution units.

3. Benefits of Superscalar Architecture

3.1 Increased Throughput

 The primary advantage of superscalar processors is the increased throughput, as

they can process multiple instructions per clock cycle.
 This leads to higher instructions per cycle (IPC), meaning more work can be done in
a given amount of time compared to scalar processors.
3.2 Better Utilization of CPU Resources

 Superscalar processors can efficiently utilize the various execution units within the
CPU, improving resource utilization and reducing idle times for components like
ALUs and FPUs.

3.3 Parallelism without Multiple Cores

 Superscalar architecture allows for parallel execution of instructions on a single

processor without requiring multiple cores or processors. This makes superscalar
processors efficient for workloads that can be parallelized at the instruction level.

4. Challenges in Superscalar Design

4.1 Data Dependencies and Hazards

 One of the challenges in superscalar architectures is dealing with data dependencies

between instructions, which can prevent parallel execution. Common types of data
hazards include:
o Read-after-write (RAW) hazard: When an instruction depends on the result
of a previous instruction that has not yet completed.
o Write-after-read (WAR) hazard: When an instruction writes to a register
that is used by a previous instruction.
o Write-after-write (WAW) hazard: When two instructions write to the same
register.
 Techniques like data forwarding and register renaming to minimize the impact of
these hazards.

4.2 Branch Prediction

 Control hazards arise from branch instructions (conditional and unconditional

branches). In superscalar processors, multiple instructions may be fetched and
executed speculatively, but the correct path of execution must be determined quickly.
 Modern superscalar processors use branch prediction to guess the direction of
branches before they are resolved. Incorrect predictions can cause pipeline flushes,
which impact performance.

4.3 Instruction Dispatch Bottlenecks

 Even with multiple pipelines, the processor may encounter bottlenecks in instruction
dispatch. If the processor cannot quickly determine which functional unit should
handle each instruction, this can reduce the number of instructions that are executed in
parallel.
 Advanced techniques like dynamic instruction scheduling and register renaming
help alleviate this issue by optimizing the use of execution units.
4.4 Limited Parallelism

 Although superscalar processors can execute multiple instructions in parallel, the

degree of parallelism is still limited by factors such as instruction dependencies and
the program’s inherent parallelism.
 Amdahl’s Law plays a role here—if a program has a significant sequential portion,
adding more execution units will have diminishing returns.

5. Superscalar vs. Vector Processors

5.1 Vector Processors

 Vector processors can perform operations on entire vectors (arrays of data)

simultaneously, using a single instruction. While they are highly efficient for certain
tasks (like scientific computing), they are designed for specific workloads.
 In contrast, superscalar processors can handle a broader range of operations and are
more versatile, as they execute multiple independent scalar instructions
simultaneously.

5.2 Hybrid Approaches

 Some modern processors use a hybrid approach, incorporating both superscalar

execution and vector processing. For example, certain instructions can be executed in
parallel (superscalar), while others operate on vectors for efficiency (SIMD – Single
Instruction, Multiple Data).

6. Examples of Superscalar Processors

6.1 Intel’s Pentium

 The Intel Pentium processors are classic examples of superscalar architecture. The
Pentium Pro and later models featured multiple pipelines, allowing for execution of
several instructions per clock cycle.

6.2 ARM and AMD Processors

 ARM processors, used in many mobile and embedded systems, often feature
superscalar designs to enhance performance while keeping power consumption low.
 Similarly, AMD’s Ryzen processors use a superscalar architecture to deliver high
performance for both single-threaded and multi-threaded applications.

Superscalar operation is a critical technique in modern processor design, allowing for

parallel execution of multiple instructions in a single cycle to improve overall throughput
and system performance. While superscalar processors are highly effective in maximizing the
utilization of available resources, challenges such as data dependencies, branch prediction,
and instruction dispatch bottlenecks remain. By employing techniques like out-of-order
execution, branch prediction, and dynamic scheduling, superscalar processors achieve
high levels of instruction-level parallelism (ILP), making them essential in both general-
purpose and specialized computing environments.
Pipelining in CISC Processors:

Pipelining is a technique used in modern processors to improve instruction throughput and

overall system performance. In a pipelined processor, multiple instruction stages are executed
simultaneously, with each stage processing a different part of an instruction. While RISC
(Reduced Instruction Set Computing) processors are typically more efficient at exploiting
pipelining due to their simpler instructions, CISC (Complex Instruction Set Computing)
processors also employ pipelining techniques, though their more complex instructions present
unique challenges.

CISC processors, such as the x86 architecture, have a wide range of complex instructions,
and each instruction can vary greatly in length and execution time. These processors use
micro-operations (μ-ops) to break complex instructions into simpler steps, and pipelining is
used to optimize the execution of these instructions.

1. Basic Pipelining Overview

Pipelining in processors divides instruction execution into multiple stages, with each stage
performing a specific task. In a basic five-stage pipeline, these stages are typically:

1. Instruction Fetch (IF): The processor fetches the instruction from memory.
2. Instruction Decode (ID): The processor decodes the instruction and prepares the
necessary operands.
3. Execute (EX): The processor performs the operation specified by the instruction
(e.g., addition, subtraction, etc.).
4. Memory Access (MEM): If the instruction involves memory (e.g., load or store), the
memory is accessed.
5. Write Back (WB): The result of the instruction is written back to the register file or
memory.

These stages operate in parallel, so while one instruction is being decoded, another is being
executed, and a third may be in the memory access stage.

In CISC processors, the complexity of instructions makes pipelining more challenging

compared to RISC processors, which have simpler, uniform instruction lengths.

2. Challenges of Pipelining in CISC Processors

2.1 Variable Instruction Length

 One of the main challenges in CISC pipelining is that CISC instructions can vary
significantly in length. For example, an instruction in the x86 architecture may be
just one byte long or several bytes long, depending on the operation.
 This variability makes instruction fetching more complicated. In pipelining, the fetch
stage usually expects instructions to be of uniform length. In CISC processors, this
variability requires special handling mechanisms to correctly fetch and decode
instructions, ensuring that the correct instruction boundaries are identified.
2.2 Complex Decoding

 CISC instructions are often quite complex, meaning they can require multiple
decoding steps. An instruction like MOV might involve directly moving a register's
value, while a more complex instruction like LODS (which loads a string) can involve
different addressing modes and different operations.
 This complexity can cause delays during the decode stage of the pipeline.Multiple
levels of decoding might be necessary, which can create pipeline stall conditions.

2.3 Microcode and Micro-operations

 CISC processors often use microcode to implement complex instructions. A complex

instruction like MUL (multiply) might not be executed by a single machine cycle but
may instead be broken down into a series of simpler operations, known as micro-
operations (μ-ops).
 Pipelining micro-operations can introduce additional complexities, as these smaller
operations must be correctly ordered and synchronized to avoid pipeline stalls and
ensure correct execution.

2.4 Instruction Dependencies and Hazards

 Similar to other processor types, CISC processors face hazards in pipelining,

including:
o Data hazards: Occur when an instruction depends on the result of a previous
instruction.
o Control hazards: Arise from branch instructions that cause uncertainty about
which instruction to fetch next.
o Structural hazards: Happen when the hardware cannot support the
simultaneous execution of multiple instructions.
 In CISC architectures, especially with variable-length instructions and micro-
operations, these hazards become harder to manage, making the pipeline less
efficient.

3. Techniques to Overcome Challenges in CISC Pipelining

3.1 Instruction Pre-fetching and Buffering

 To address the problem of variable-length instructions, CISC processors often use

instruction pre-fetch buffers. These buffers store multiple bytes of an instruction to
ensure that the fetch stage can proceed without interruption.
 Branch prediction and Instruction caching can help reduce the stalls caused by
variable instruction lengths, as the processor can quickly access multiple instructions
that are likely to be needed next.

3.2 Decoding Techniques

 To mitigate delays in decoding, CISC processors use specialized instruction

decoders that can handle the complexities of variable-length instructions. Often, the
processor will decode the instruction incrementally, as more bits are fetched from
memory.
  Microcode allows complex instructions to be translated into simpler micro-
operations, which can then be pipelined more easily. However, to avoid delays,
modern CISC processors utilize multiple stages of microcode caching to speed up
instruction execution.

3.3 Dynamic Scheduling

 Dynamic scheduling helps manage the dependencies between instructions in the

pipeline. By using techniques like out-of-order execution and data forwarding,
CISC processors can reduce pipeline stalls caused by data hazards.

3.4 Pipelined Execution of Micro-operations

 Since CISC instructions are often broken into multiple micro-operations, pipelining
the micro-operations instead of the original instructions is a key optimization. This
allows independent operations to proceed in parallel without waiting for the
completion of the entire instruction.
 Out-of-order execution is frequently used here to allow micro-operations that don’t
depend on each other to proceed, reducing the time spent waiting for other operations
to complete.

4. Example: Intel’s x86 Architecture

The Intel x86 architecture is one of the most widely known examples of a CISC processor
employing pipelining. In early designs, the x86 processors used simple, non-pipelined
execution models. However, as technology advanced, Intel began to incorporate pipelined
execution into their processors with multiple stages of instruction processing.

In the Pentium processor, for instance, multiple instructions can be fetched and decoded in
parallel, and the instructions are divided into simpler micro-operations that can be pipelined
individually. The Pentium Pro and later models used deeper pipelines, achieving high
throughput despite the complexity of CISC instructions.

5. Benefits and Limitations of Pipelining in CISC Processors

5.1 Benefits

 Improved Throughput: Pipelining enables CISC processors to execute multiple

instructions in parallel, improving the overall throughput of the system.
 Efficient Use of Resources: By dividing instruction execution into multiple stages,
pipelining makes better use of available hardware resources, ensuring that different
parts of the processor (such as the ALU, memory access units, and registers) are
continuously utilized.
 Enhanced Clock Speeds: Pipelining allows for higher clock speeds, as the processor
can perform more work per clock cycle by executing different stages of multiple
instructions simultaneously.
5.2 Limitations

 Complexity in Handling Hazards: Due to the complexity of CISC instructions and

microcode, handling data and control hazards is more challenging in CISC
pipelining.
 Pipeline Stalls: Variable instruction lengths and complex decoding can lead to
pipeline stalls, reducing the potential performance benefits of pipelining.
 Increased Power Consumption: The more stages added to the pipeline and the more
complex the instruction decoding becomes, the greater the power consumption,
making energy efficiency harder to achieve.

Pipelining in CISC processors is an essential technique to improve the performance of

complex instruction architectures. While CISC instructions are more complicated than those
in RISC architectures, the use of micro-operations, dynamic scheduling, and advanced
decoding techniques allows pipelining to be implemented effectively. However, the
complexity of handling variable instruction lengths and managing hazards requires careful
design and optimization. CISC processors like Intel’s x86 and AMD processors illustrate
how pipelining, when combined with other techniques, can significantly improve instruction
throughput and overall performance despite the challenges.

Instruction-Level Parallelism (ILP)

Instruction-Level Parallelism (ILP) refers to the ability to execute multiple instructions

from a program simultaneously. ILP is a key concept in modern processor design that aims to
increase the throughput of a processor by exploiting parallelism at the level of individual
instructions, rather than at the level of whole tasks or threads. The more ILP a processor can
exploit, the more efficient it becomes in processing multiple instructions in parallel, leading
to improved overall performance.

1. Key Concepts in Instruction-Level Parallelism

1.1 Dependencies and Parallelism

 For ILP to be exploited, instructions must be independent of each other.

Dependencies between instructions can limit the degree of parallelism that can be
achieved.
o Data Dependency: This occurs when an instruction depends on the result of a
previous instruction. There are three primary types of data dependencies:
 Read-after-write (RAW) or True Dependency: Instruction A must
complete its write before Instruction B can read the data.
 Write-after-read (WAR) or Anti-dependency: Instruction A writes
to a register or memory location that Instruction B reads from.
 Write-after-write (WAW) or Output Dependency: Two instructions
write to the same register or memory location.
o Control Dependency: This is caused by branch instructions. When a branch
is encountered, the outcome is uncertain until the branch condition is
evaluated.
o Resource Dependency: Occurs when multiple instructions share the same
hardware resource (e.g., ALUs or registers), limiting how many instructions
can be executed in parallel.
1.2 ILP and Instruction Scheduling

 Instruction scheduling refers to the reordering of instructions to maximize parallel

execution. Out-of-order execution is a technique where instructions are executed as
soon as their operands are available, regardless of their original program order.
 Instructions can be executed in parallel, such as software pipelining and compiler-
based scheduling.

1.3 ILP and Pipelining

 Pipelining helps to exploit ILP by breaking down the execution of each instruction
into multiple stages. These stages can be overlapped, allowing multiple instructions to
be processed simultaneously in different stages.
 The combination of pipelining and ILP allows for the execution of more than one
instruction in parallel, increasing the overall throughput of the processor.

2. Techniques to Achieve High ILP

2.1 Out-of-Order Execution

 In out-of-order execution, instructions are executed as their operands become

available, rather than strictly following the original program order. This helps to
reduce pipeline stalls and increase ILP.
 Tomasulo’s Algorithm and Dynamic Scheduling are common methods used to
implement out-of-order execution. These techniques allow the processor to avoid
waiting for data dependencies and instead execute independent instructions as soon as
possible.

2.2 Register Renaming

 Register renaming is a technique used to eliminate false dependencies (WAR and

WAW hazards) by dynamically mapping register names to physical registers.
 This technique allows the processor to reuse registers more efficiently and avoid stalls
caused by data hazards, thus increasing ILP.

2.3 Speculative Execution

 Speculative execution involves executing instructions before it is certain whether

they will be needed. This is often used in branch prediction, where the processor
guesses the direction of a branch and continues executing instructions along that path.
If the guess is correct, the instructions are already executed by the time the branch is
resolved.
 Speculative execution increases ILP by keeping the pipeline full, even when the
outcome of a branch is unknown.

2.4 Branch Prediction

 Branch prediction is a technique used to minimize the performance impact of

branches in ILP. A branch predictor guesses the outcome of a branch instruction to
 allow the processor to continue executing instructions without waiting for the branch
to be resolved.
 Static and dynamic branch predictors: Static predictors use simple techniques like
always predicting the branch will be taken, while dynamic predictors use historical
execution data to make more accurate predictions.

2.5 VLIW (Very Long Instruction Word)

 VLIW is an architecture that explicitly encodes parallel instructions in a single long

instruction word. These instructions are issued and executed in parallel by the
processor.
 ILP is achieved by allowing the compiler to schedule multiple independent
instructions in the same VLIW instruction word, thus exploiting parallelism at the
instruction level.

2.6 SIMD (Single Instruction, Multiple Data)

 SIMD allows the same instruction to be applied to multiple data points in parallel.
This is particularly useful in applications like vector processing and multimedia,
where large datasets can be processed simultaneously.
 ILP is exploited in SIMD by performing multiple data operations in parallel, with
each operation requiring the same instruction.

3. Challenges in Exploiting ILP

3.1 Dependencies Limiting Parallelism

 Data dependencies and control dependencies often limit the amount of parallelism
that can be achieved. For instance, a RAW hazard (true dependency) prevents two
instructions from being executed in parallel if one depends on the result of the other.
 Despite advanced techniques like out-of-order execution, ILP is inherently limited
by these dependencies.

3.2 Branch Hazards

 Control hazards caused by branching instructions present a significant challenge in

ILP. If a branch is mispredicted, the processor may need to flush the pipeline, which
wastes cycles and reduces the effectiveness of ILP.
 To mitigate this, advanced branch prediction techniques are used, but even then, the
presence of branches can limit ILP.

3.3 Resource Contention

 Resource contention occurs when multiple instructions require the same hardware
resource (e.g., ALUs, registers, memory). This can limit parallelism, as only one
instruction can use a resource at a time.
 ILP can be improved by providing more resources (e.g., multiple ALUs) or by using
techniques like register renaming to avoid false dependencies.
3.4 Diminishing Returns

 While ILP can significantly improve performance, there are diminishing returns. Even
with advanced techniques like out-of-order execution and speculative execution, the
degree of parallelism is limited by the dependencies in the program and the
availability of resources.
 Amdahl’s Law highlights that improving ILP will not result in a proportional
speedup if there is a significant sequential portion in the program that cannot be
parallelized.

4. ILP in Modern Processors

4.1 Superscalar Architectures

 Superscalar processors exploit ILP by having multiple execution units that can
execute several instructions concurrently. Intel’s Pentium and AMD’s Ryzen
processors are examples of superscalar processors that use ILP to maximize
instruction throughput.
 These processors can dynamically schedule instructions to different execution units,
making use of ILP to perform several operations in parallel.

4.2 Multi-core Processors

 Modern multi-core processors improve overall system performance by executing

multiple threads in parallel, but they also rely heavily on ILP within each core. Each
core can exploit ILP within its own instruction pipeline to improve performance for
individual threads.

4.3 GPUs and ILP

 Graphics Processing Units (GPUs) are highly optimized for ILP, especially in
SIMD workloads. GPUs can execute thousands of threads in parallel, with each
thread performing the same operation on different data, making them well-suited for
workloads that require high levels of parallelism, such as graphics rendering and
machine learning.

Instruction-Level Parallelism (ILP) plays a crucial role in modern processor design. By

exploiting the parallelism inherent in the execution of individual instructions, processors can
achieve higher performance and efficiency. Techniques such as out-of-order execution,
register renaming, speculative execution, and branch prediction help maximize the
parallelism that can be extracted from programs. However, challenges like dependencies,
branch hazards, and resource contention limit the amount of ILP that can be exploited.
Modern processors, including superscalar processors and GPUs, have made significant
strides in overcoming these challenges to achieve impressive levels of instruction throughput
and parallelism.

Parallel Processing Challenges

Parallel processing is a computational model where multiple processors or cores work

simultaneously on different parts of a problem to solve it more quickly. It leverages the idea
of dividing a task into smaller sub-tasks that can be executed concurrently. While the
theoretical benefits of parallel processing are significant in terms of increased speed and
efficiency, there are numerous practical challenges involved in implementing parallel
processing effectively. These challenges stem from the inherent complexity of splitting tasks,
managing dependencies, and coordinating multiple processors.

1. Scalability Issues

1.1 Amdahl’s Law

 Amdahl’s Law defines the theoretical speedup of a system as a function of

parallelism. According to this law, the speedup of a program with a parallel portion is
limited by the size of the serial portion of the program.
o Formula:

o The law suggests that as the number of processors increases, the speedup
grows, but only up to a limit determined by the non-parallelizable portion.
This puts a fundamental limit on how much performance can be gained by
simply adding more processors.
o In practice, as processors are added, the overhead of coordinating them (e.g.,
managing memory, handling communication) can outweigh the benefits,
limiting scalability.

1.2 Diminishing Returns

 As the number of processors increases, the overhead required to manage and

synchronize the tasks also grows. This results in diminishing returns in terms of
performance improvement.
 For a parallel system to scale effectively, the overhead should not increase
significantly as more processors are added, and the parallelizable portion of the task
must remain large.

2. Synchronization and Coordination

2.1 Race Conditions

 Race conditions occur when multiple processes or threads attempt to access shared
resources (e.g., memory or I/O) simultaneously, and the final outcome depends on the
 order of execution. In parallel systems, unsynchronized access to shared resources
can result in unpredictable and incorrect results.
 Proper synchronization is required to prevent race conditions. This is typically done
using locks, mutexes, or semaphores, but these mechanisms can introduce delays
and reduce performance.

2.2 Deadlock

 A deadlock occurs when two or more processes are waiting for each other to release
resources, leading to a situation where none of the processes can proceed.
o For example, Process A holds Resource 1 and waits for Resource 2, while
Process B holds Resource 2 and waits for Resource 1.
 Deadlocks can significantly reduce the performance of parallel systems if they are not
properly managed. Techniques such as timeout mechanisms, deadlock detection,
and resource allocation graphs are used to prevent or resolve deadlocks.

2.3 Load Balancing

 Effective load balancing ensures that all processors in a parallel system are utilized
efficiently. If some processors are idle while others are overloaded, the overall
performance can be negatively impacted.
 The challenge is distributing the workload in such a way that the tasks are evenly
spread across the processors and the load is balanced throughout the execution. This
can be difficult, especially when tasks vary in complexity or size.
o Dynamic load balancing techniques, where tasks are redistributed during
execution, are employed to ensure optimal performance.

3. Communication Overhead

3.1 Inter-Processor Communication

 In parallel systems, especially multi-core or distributed systems, processors must

frequently exchange data. The time required for communication between processors is
known as communication overhead.
o In systems with shared memory, communication overhead occurs when data
needs to be transferred between different cores' local caches and main
memory.
o In distributed memory systems, processors must communicate over the
network, which introduces significant latency and bandwidth constraints.
 The challenge is ensuring that communication overhead does not dominate the
computation time. Techniques like message passing, shared memory, and remote
direct memory access (RDMA) are used to mitigate this issue.

3.2 Bandwidth Limitations

 Bandwidth limitations refer to the limited rate at which data can be transferred
between processors or between a processor and memory. This can be a bottleneck in
parallel systems, especially when large amounts of data need to be shared between
processors.
  Modern processors and architectures, such as multi-core processors and GPUs, use
high-bandwidth memory and fast interconnects (e.g., InfinityFabric, NVLink) to
improve data transfer rates.

4. Memory Access Bottlenecks

4.1 Memory Latency

 Memory latency refers to the delay between a processor requesting data from
memory and receiving it. In a parallel system, multiple processors may request data
from the same memory, leading to contention and increased latency.
 This issue can be mitigated by having multiple memory banks or employing non-
uniform memory access (NUMA), where processors are assigned to specific
memory regions to reduce contention.

4.2 Cache Coherence

 In multi-core systems, each core typically has its own local cache to store frequently
accessed data. However, when multiple cores modify the same memory locations, the
caches can become inconsistent, resulting in cache coherence problems.
 Solutions to this include the MESI protocol (Modified, Exclusive, Shared, Invalid),
which ensures that all caches in a multi-core system are kept consistent, but
maintaining cache coherence can introduce significant overhead and reduce
performance.

4.3 False Sharing

 False sharing occurs when different threads access different variables that happen to
reside on the same cache line. Even though the threads are not actually sharing data,
they may cause unnecessary cache invalidations, reducing performance.
 To avoid false sharing, data must be carefully aligned and placed in memory to ensure
that frequently accessed data does not reside on the same cache line.

5. Parallel Algorithm Design

5.1 Algorithm Parallelization

 Parallelizing an algorithm is not always straightforward, as some algorithms have

inherent sequential steps that cannot be parallelized. The challenge lies in breaking
down a task into smaller sub-tasks that can run concurrently with minimal
dependencies.
 Some algorithms, like divide-and-conquer algorithms, are naturally parallelizable,
whereas others, like dynamic programming, may not be easily parallelized due to
the need for sequential data access.

5.2 Communication Complexity

 In parallel algorithm design, communication complexity refers to the amount of

communication required between processors. Reducing the need for communication
between processors can significantly improve performance.
  Algorithms that require frequent inter-process communication or synchronization
(such as matrix multiplication in distributed systems) can suffer from scalability
issues due to high communication overhead.

6. Fault Tolerance and Reliability

6.1 Error Detection and Correction

 In large parallel systems, especially distributed ones, the possibility of hardware

failures increases. To maintain reliability, parallel systems must be designed with
mechanisms for error detection and fault tolerance.
 Redundancy (e.g., having backup components) and checkpointing (saving the state
of the system periodically) are commonly used to ensure that computation can
continue in case of failure.

6.2 Soft Errors

 Soft errors, caused by environmental factors like radiation, can lead to bit flips in
memory or processor states. In parallel systems, soft errors are a significant concern,
as they may lead to incorrect results if not detected and corrected.
 Error-correcting codes (ECC) and redundant execution are used to protect against
these types of errors in parallel systems.

7. Energy Efficiency

 Energy efficiency is a significant challenge in parallel processing, as the increasing

number of processors and the high communication demands of parallel systems can
lead to excessive energy consumption.
 Techniques such as dynamic voltage and frequency scaling (DVFS), power-aware
scheduling, and energy-efficient algorithms are used to mitigate the energy cost of
parallel processing.

While parallel processing has the potential to dramatically improve computational

performance, there are several significant challenges that must be addressed to fully exploit
this potential. These include issues with scalability, synchronization, communication
overhead, memory access bottlenecks, fault tolerance, and energy efficiency. Overcoming
these challenges requires advanced techniques in algorithm design, hardware architecture,
and software optimization, and continues to be a central focus in the development of
modern parallel computing systems. The effectiveness of parallel processing will ultimately
depend on how well these challenges are managed to allow for efficient, scalable, and reliable
computation.

Flynn’s Classification

Flynn's Classification is a fundamental taxonomy in parallel computing that categorizes

computer architectures based on the number of instruction streams and data streams they can
process simultaneously. This classification, introduced by Michael J. Flynn in 1966, helps in
understanding the structure and capabilities of various parallel computing systems. Flynn’s
model has been instrumental in guiding the development of processors and systems that aim
to exploit parallelism efficiently.

1. Single Instruction Single Data (SISD)

1.1 Definition

 SISD represents a traditional, sequential computing model where one processor

executes a single instruction stream, processing one data element at a time.
 In this model, there is no parallelism in terms of instruction execution. Each
instruction operates on a single piece of data, and tasks are completed sequentially.

1.2 Characteristics

 Processor: Single processor that handles both instruction and data sequentially.
 Memory: A single memory unit is used for both instructions and data.
 Example Systems: Early mainframe computers, basic microprocessors, and
personal computers.

1.3 Limitations

 SISD systems are constrained by the von Neumann bottleneck, where the processor
is limited by the speed of memory access, and sequential execution restricts
performance.
 It is not capable of taking advantage of modern parallel computing demands.

2. Single Instruction Multiple Data (SIMD)

2.1 Definition

 SIMD systems are capable of applying the same instruction to multiple data elements
at once. This allows multiple data elements to be processed in parallel using a single
instruction.
 SIMD is widely used in applications like vector processing, graphics processing,
and scientific computing, where the same operation needs to be applied to large
datasets.

2.2 Characteristics

 Processor: Single control unit issuing the same instruction to multiple processing
elements.
 Memory: Data is organized in such a way that the same instruction operates on
different pieces of data in parallel.
 Example Systems: Graphics Processing Units (GPUs), Vector processors, SIMD
extensions in CPUs (e.g., Intel AVX and SSE instructions).
2.3 Strengths

 SIMD systems can process large amounts of data in parallel with minimal overhead,
making them highly effective for data-parallel applications.
 They are particularly beneficial for media processing, 3D rendering, and machine
learning, where the same operation needs to be performed on many pieces of data
simultaneously.

2.4 Limitations

 SIMD is restricted to problems that exhibit data parallelism, where the same
operation can be performed on multiple data elements independently.
 SIMD cannot be used for tasks that require task parallelism (e.g., different
instructions for different data).

3. Multiple Instruction Single Data (MISD)

3.1 Definition

 MISD is a more theoretical and rare category where multiple instruction streams
operate on a single data stream. This system would execute multiple instructions
concurrently on the same data, but the data itself remains unchanged by the different
operations.
 It’s not commonly seen in practice and is more of a conceptual category.

3.2 Characteristics

 Processor: Multiple processors each execute different instructions on the same data.
 Memory: Single data stream, with each processor accessing the same data.
 Example Systems: While no commercially viable systems exist for MISD, it could
potentially be useful in fault tolerance systems where different computations are
applied to the same data to check for consistency and accuracy.

3.3 Strengths

 MISD could be useful for redundant computing or error detection in safety-critical

systems where the same data is processed by multiple independent instructions to
ensure reliability.

3.4 Limitations

 MISD is rarely implemented due to the lack of practical use cases.

 It’s inefficient for most applications because multiple instructions act on the same
data in an unpredictable or unnecessary manner.

4. Multiple Instruction Multiple Data (MIMD)

4.1 Definition
  MIMD systems are capable of executing multiple instruction streams concurrently on
multiple data streams. Each processor in an MIMD system can execute its own
instruction sequence on different data, which makes MIMD the most versatile and
widely used model in parallel computing.
 MIMD is suitable for a wide range of applications, from supercomputing to
distributed systems.

4.2 Characteristics

 Processor: Multiple processors, each executing its own instruction stream on

independent data.
 Memory: Can be shared memory (where all processors access a common memory)
or distributed memory (where each processor has its own memory).
 Example Systems:
o Multiprocessor systems (e.g., IBM Blue Gene, Cray supercomputers).
o Cluster computing and cloud computing systems.
o Multicore processors in modern desktop systems.

4.3 Strengths

 MIMD systems can handle both task parallelism and data parallelism, allowing
them to address a wide variety of complex computational problems.
 They can be scalable, supporting any number of processors, and can be optimized for
distributed processing in large clusters.

4.4 Limitations

 Coordination and synchronization of processors in MIMD systems can introduce

overhead, especially in systems with distributed memory, where processors must
communicate over a network.
 Handling shared memory efficiently requires careful cache management and
synchronization to avoid conflicts and ensure consistency.

5. Flynn’s Classification in Practice

 Flynn's classification continues to play a critical role in the design of modern multi-
core processors and distributed systems.
o SIMD is used extensively in GPUs and media processing units, where the
same instruction is applied to multiple data elements simultaneously.
o MIMD is the foundation for most modern supercomputers and cloud
computing infrastructures, enabling the parallel execution of independent
tasks across many processors or nodes.
 MIMD architectures, especially those used in multi-core processors, distributed
computing, and cluster-based systems, are the most flexible and widely adopted
systems in contemporary computing.

Flynn’s Classification provides a framework for understanding the basic architectures that
enable parallel processing. From the simple and sequential SISD to the highly flexible
MIMD, each category of architecture serves different types of computational needs. Modern
parallel computing continues to evolve within this framework, with SIMD and MIMD
architectures being central to the development of high-performance computing systems, such
as GPUs, multi-core processors, and supercomputers. Understanding Flynn’s categories
helps in selecting the right parallel processing model for a given application, ensuring optimal
performance based on the nature of the task and available hardware.

Hardware Multithreading

Hardware multithreading refers to a technique used in computer processors to improve the

efficiency of the processor by enabling it to execute multiple threads (streams of instructions)
simultaneously on different processing units or functional units within a single processor
core. This approach is designed to keep the processor's resources fully utilized, minimizing
idle time and increasing throughput.

The concept of hardware multithreading is particularly useful in scenarios where processor

execution units are underutilized due to high latency operations like memory access or cache
misses. By overlapping the execution of multiple threads, the processor can make better use
of its resources.

1. Types of Hardware Multithreading

There are several types of hardware multithreading techniques, each with its unique approach
to managing multiple threads within a processor. The most common types include Fine-
Grained Multithreading, Coarse-Grained Multithreading, and Simultaneous
Multithreading (SMT).

1.1 Fine-Grained Multithreading

 Definition: Fine-grained multithreading involves switching between threads on every

clock cycle or after each instruction. When one thread encounters a stall (e.g., waiting
for memory access or a cache miss), the processor switches to another thread to
continue execution.
 How it Works: A fine-grained multithreaded processor cycles through a series of
threads, executing one instruction from each thread in rapid succession. This
minimizes processor idle time because, even if one thread is stalled, others can
continue executing.
 Characteristics:
o Thread switching happens very frequently, potentially on every clock cycle.
o Processor utilization is maximized by switching threads quickly.
o Higher complexity in thread management due to frequent context switches.
 Example: Early versions of multithreaded processors, like Tera MTA and some
graphics processors (GPUs), used fine-grained multithreading for efficient execution
of parallel tasks.

1.2 Coarse-Grained Multithreading

 Definition: Coarse-grained multithreading involves switching between threads less

frequently, typically only when one thread encounters a long delay (e.g., a cache miss
or memory stall). This approach groups multiple instructions together and switches to
another thread only when the current thread is blocked for a significant period of time.
  How it Works: A processor with coarse-grained multithreading runs one thread for a
relatively long time until a stall or long latency occurs. Once that thread stalls, the
processor switches to another thread, which will run for a longer period before being
switched out again.
 Characteristics:
o Thread switching occurs only when there is a significant delay (e.g., waiting
for memory).
o Lower overhead for thread management compared to fine-grained
multithreading.
o Reduced context switching overhead.
 Example: Sun's Niagara processor (also known as UltraSPARC T1) uses coarse-
grained multithreading, where it runs multiple threads but switches between them
only when a thread encounters a long stall.

1.3 Simultaneous Multithreading (SMT)

 Definition: Simultaneous multithreading (SMT) is an advanced technique where a

processor can execute multiple threads in parallel within the same clock cycle. SMT
enables multiple threads to make use of the available processor resources, such as
functional units, registers, and caches, concurrently.
 How it Works: Unlike fine-grained multithreading, which alternates between threads,
SMT runs multiple threads simultaneously in a single processor core. This involves
complex hardware design to allow parallel execution of instructions from different
threads on the same core, using multiple execution pipelines and registers.
 Characteristics:
o Multiple threads execute simultaneously on the same core, improving
throughput.
o Efficient resource utilization: Different threads can use different functional
units of the processor (e.g., integer, floating-point).
o Increased complexity due to the need to manage multiple instruction streams
and maintain pipeline coherence.
o Example: Intel’s Hyper-Threading Technology (HTT) is a well-known
implementation of SMT, allowing a single core to execute two threads
concurrently.

2. Advantages of Hardware Multithreading

2.1 Increased Throughput

 Hardware multithreading increases the throughput of a processor by allowing it to

process multiple instructions or threads at the same time. This results in higher overall
execution rates and better performance, especially in workloads with many
independent threads.

2.2 Better Resource Utilization

 In traditional single-threaded execution, the processor may experience idle cycles due
to waiting on memory or I/O operations. Hardware multithreading minimizes these
idle cycles by executing other threads during these stalls, thus keeping the processor
busy and making full use of its resources.
2.3 Reduced Latency

 By switching between threads that are not stalled, hardware multithreading can reduce
the impact of latency caused by memory accesses, cache misses, or other long-latency
operations. This ensures that the processor can continue to perform useful work even
when one thread is waiting for data.

2.4 Scalability

 Multithreading can scale with the number of threads and available hardware
resources. This is especially beneficial in systems with many processors or cores, as
each processor or core can handle multiple threads concurrently.

3. Disadvantages of Hardware Multithreading

3.1 Increased Hardware Complexity

 Implementing hardware multithreading, especially SMT, requires significant

hardware resources, including multiple pipelines, registers, and complex scheduling
mechanisms to manage multiple threads efficiently. This increases the complexity of
the processor design and can lead to higher costs.

3.2 Diminishing Returns

 The benefits of hardware multithreading may diminish as the number of threads

increases. Once the processor reaches a certain point of thread parallelism, the
additional threads may not contribute significantly to performance because of
resource contention or the overhead of managing the threads.

3.3 Context Switching Overhead

 In fine-grained multithreading, frequent context switching between threads can

introduce overhead. Each switch involves saving and loading register states, which
can reduce the overall performance if not managed efficiently.

3.4 Resource Contention

 When multiple threads are executed on the same core or processor, they must share
the available resources (e.g., execution units, memory, cache). This can lead to
resource contention, where the threads compete for limited resources, potentially
reducing the performance benefits of multithreading.

4. Hardware Multithreading in Modern Processors

4.1 Multi-Core Processors

 Modern multi-core processors, such as Intel’s Core i7 or AMD Ryzen processors,

often use a combination of multithreading and multiple cores to increase overall
performance. Each core may support SMT to handle multiple threads, allowing for
even more parallelism and better performance in multi-threaded applications.
4.2 GPUs and Parallel Computing

 Graphics Processing Units (GPUs), which are designed for high-throughput parallel
computing, use a form of multithreading that allows many threads to run in parallel on
different processing units within the GPU. CUDA cores in NVIDIA GPUs can
execute thousands of threads simultaneously, making GPUs well-suited for parallel
workloads like deep learning, scientific simulations, and video rendering.

4.3 Supercomputing

 Supercomputers and high-performance computing (HPC) systems often use

hardware multithreading across many cores, along with specialized hardware like
GPUs, to achieve exceptional performance in simulations, weather modeling, and
other data-intensive applications.

5. Future of Hardware Multithreading

 Continued Evolution of SMT: As processor architectures evolve, Simultaneous

Multithreading (SMT) is expected to become more sophisticated, supporting even
more threads per core while improving resource management and reducing
contention.
 Hybrid Systems: Future processors may combine multithreading with other
technologies such as machine learning-based optimizations and heterogeneous
processing (e.g., integrating CPUs and GPUs on the same chip) to further enhance
performance.
 Quantum and Neuromorphic Computing: Emerging fields like quantum
computing and neuromorphic computing may introduce new forms of
multithreading, where threads may be more dynamic and context-dependent based on
quantum states or biological-inspired architectures.

Hardware multithreading is a key technique for enhancing processor performance by

exploiting parallelism at the hardware level. By enabling a processor to handle multiple
threads concurrently, multithreading increases throughput, improves resource utilization, and
reduces idle times. However, the complexity of managing multiple threads and the potential
for resource contention are key challenges that must be addressed. As processors continue to
evolve with more cores and better thread management techniques, hardware multithreading
will continue to be a cornerstone of high-performance computing systems.

Multicore Processors: GPU, Multiprocessor Network Topologies

Multicore Processors: GPU

In the context of modern computing, multicore processors and Graphics Processing Units
(GPUs) are crucial components that significantly contribute to the efficiency and
performance of a wide range of applications, from scientific computing to graphics rendering
and machine learning. While multicore processors are designed for general-purpose
computing, GPUs are specialized hardware designed for handling highly parallel tasks.
1. Multicore Processors

1.1 Definition

 A multicore processor is a single computing component with two or more

independent processing units (cores), each of which can execute instructions
independently but shares the same physical resources such as memory, cache, and
interconnects.
 Multicore systems enable parallel execution of tasks, improving system throughput
and efficiency.

1.2 Core Concepts

 Cores: Each core can execute instructions concurrently, which means a multicore
processor can handle multiple tasks or threads simultaneously. This is essential for
handling complex, multithreaded applications that require large amounts of
computational power.
 Shared Resources: In a multicore processor, multiple cores often share resources like
cache (L1, L2, and L3), memory bus, and I/O controllers. Efficient management of
shared resources is key to optimizing performance in multicore systems.
 Parallelism: Multicore processors can perform both task parallelism (multiple tasks
on separate cores) and data parallelism (dividing a large task into smaller chunks for
multiple cores).

1.3 Advantages of Multicore Processors

 Higher Performance: Multicore processors allow systems to run more complex

applications faster by distributing workloads across multiple cores.
 Energy Efficiency: By using multiple cores, processors can execute tasks in parallel,
leading to faster completion times and lower power consumption compared to scaling
clock speeds on a single core.
 Multithreading: Multicore systems are well-suited for multithreaded applications,
such as video editing, scientific simulations, and web servers, where different
threads can run on different cores.

1.4 Challenges

 Software Optimization: To take full advantage of multicore systems, software must

be parallelized, which can be a complex process. Not all algorithms or programs can
be easily parallelized.
 Cache Coherency: Since multiple cores share memory, maintaining data consistency
across cores (i.e., ensuring that one core's data doesn't conflict with another's) requires
sophisticated cache coherency protocols.
2. Graphics Processing Unit (GPU)

2.1 Definition

 A GPU is a specialized processor designed to handle parallel computation for

graphics rendering. While originally designed for graphics and visual processing
tasks, GPUs have become an essential tool for a broad range of high-performance
computing applications, including scientific computing and artificial intelligence.
 Modern GPUs are massively parallel processors with thousands of smaller cores that
execute computations in parallel, making them ideal for handling large amounts of
data in parallel tasks.

2.2 GPU Architecture

 Processing Cores: Unlike CPU cores, which are designed to handle a few threads
with high clock speeds and complex instructions, GPU cores are simpler and
designed for parallelism, making them suitable for applications like image
processing, matrix multiplications, and simulations.
 SIMD Model: Most modern GPUs operate on the SIMD (Single Instruction, Multiple
Data) model, where a single instruction is applied to multiple pieces of data
simultaneously. This is highly effective for data-intensive operations such as vector
and matrix operations.
 Memory Hierarchy: GPUs have a distinct memory hierarchy optimized for high-
throughput data access:
o Global Memory: Large but slower memory shared across all cores.
o Shared Memory: Faster, smaller memory used by threads within a block.
o Registers: The fastest form of memory, used for thread-local data.

2.3 Advantages of GPUs

 High Parallelism: With thousands of cores capable of processing data in parallel,

GPUs excel at tasks that involve large amounts of data, such as image rendering,
machine learning, and data mining.
 Throughput-Oriented: Unlike CPUs, which are designed for low-latency, sequential
tasks, GPUs are designed to maximize throughput. This makes them highly suitable
for batch processing, vector operations, and applications requiring massive
parallelism.
 Flexible Use: While originally designed for graphics rendering, modern GPUs can be
used for general-purpose computing (GPGPU) through programming models like
CUDA (Compute Unified Device Architecture) and OpenCL (Open Computing
Language).

2.4 GPU Use Cases

 Graphics and Gaming: GPUs are best known for their use in rendering graphics for
video games, movies, and interactive applications. They handle the complex
mathematical computations required for tasks such as texture mapping, lighting
calculations, and 3D rendering.
  Machine Learning: Deep learning and other machine learning techniques benefit
greatly from GPUs, as they can perform the massive matrix and vector operations
required for training neural networks in parallel.
 Scientific Computing: Simulations in physics, chemistry, and biology can take
advantage of the GPU’s parallel processing capabilities to speed up computational
models.

3. Integration of Multicore Processors and GPUs

In modern computing systems, multicore processors and GPUs are often used together to
exploit both types of parallelism—task parallelism from multicore CPUs and data
parallelism from GPUs. This combined architecture is increasingly common in high-
performance systems such as servers, supercomputers, and workstations.

3.1 Heterogeneous Computing

 Heterogeneous systems integrate CPUs (multicore processors) and GPUs to execute

different types of tasks based on the strengths of each. The CPU typically handles
tasks that require complex decision-making or involve serial execution, while the
GPU handles data-intensive, parallelizable tasks.
 GPGPU Programming: Technologies like CUDA (for NVIDIA GPUs) and
OpenCL (for AMD and other GPUs) allow developers to write programs that utilize
both the CPU and GPU. These tools enable parallel algorithms to be written for
GPUs, allowing CPUs to delegate tasks to GPUs for processing and thus speed up the
overall workload.

3.2 Communication between CPU and GPU

 Effective communication between the CPU and GPU is critical for achieving high
performance in heterogeneous systems. Direct Memory Access (DMA), PCI
Express (PCIe), and shared memory allow for high-speed data transfer between the
CPU and GPU.
 CPU handles control and high-level logic, while the GPU performs the data-heavy
parallel computations, such as matrix multiplication, image processing, or
simulation tasks.

3.3 Example of Hybrid Use

 Deep Learning: In a deep learning model, the CPU may be responsible for
controlling the flow of data, managing input/output operations, and executing
sequential tasks, while the GPU performs the massive matrix calculations required for
training large neural networks.
4. Future Trends and Advancements

4.1 Scaling of Multicore and GPU Architectures

 Multicore processors are scaling toward higher core counts, with processors like the
AMD Ryzen 9 and Intel Xeon offering up to 64 cores, designed to handle highly
parallel workloads in servers, workstations, and high-performance computers.
 GPUs are becoming more powerful, with companies like NVIDIA and AMD pushing
the boundaries of parallel processing. For instance, NVIDIA’s A100 Tensor Core
GPU for deep learning provides over 54 teraflops of processing power, highlighting
the growing importance of GPUs in AI workloads.

4.2 Integration of AI in Processors

 AI Acceleration: Future processors and GPUs are likely to include dedicated AI

cores that accelerate machine learning workloads. Tensor cores in GPUs are already
designed to accelerate AI and deep learning tasks, and future processors may integrate
more specialized AI hardware for faster computation.

The combination of multicore processors and GPUs represents the future of high-
performance computing. Multicore CPUs offer general-purpose processing power and handle
complex, serial workloads, while GPUs excel at parallelizing data-intensive tasks. Together,
they enable faster processing in a variety of fields, including machine learning, scientific
simulations, graphics rendering, and big data processing. As technology evolves, we can
expect even more powerful and efficient systems, integrating these components for enhanced
performance in a wide range of applications.

Multiprocessor Network Topologies

In the context of multicore processors and multiprocessor systems, the multiprocessor

network topology refers to the arrangement or interconnection of processors in a system.
The choice of network topology affects the communication efficiency, scalability, and overall
performance of the multiprocessor system. Understanding how processors are interconnected
is crucial for optimizing performance in high-performance computing environments, such as
servers, supercomputers, and parallel computing systems.

A well-designed topology ensures that data can be transferred efficiently between processors
and memory, minimizing latency and maximizing throughput.

1. Importance of Network Topology

The network topology in a multiprocessor system determines:

 Communication Pathways: How processors communicate with each other and with
memory units.
 Bandwidth and Latency: The speed and efficiency of data transfer between
processors and memory, affecting the overall system performance.
 Scalability: The ability to add more processors without significantly degrading
performance.
  Fault Tolerance: The system’s ability to continue functioning even if one or more
components fail.

Multiprocessor systems are typically classified into two main types based on their
interconnection structure:

1. Shared Memory Systems: Multiple processors access a common memory space.

2. Distributed Memory Systems: Each processor has its own local memory, and
communication between processors occurs over a network.

Both topologies have their specific advantages and challenges, with the design of the
interconnection network being key to efficient system operation.

2. Types of Multiprocessor Network Topologies

2.1 Bus-Based Topology

 Definition: In a bus-based topology, all processors and memory units are connected
to a single communication bus. The bus serves as the shared medium for data transfer
between processors and memory.
 Characteristics:
o Simple Design: The design is simple and cost-effective for small-scale
multiprocessor systems.
o Single Communication Path: All processors share the same bus, meaning
that only one processor can transmit data at a time.
o Scalability Issues: As more processors are added, the bus becomes a
bottleneck, reducing performance due to congestion.
 Advantages:
o Cost-effective and easy to implement for small systems.
o Easy to add new processors.
 Disadvantages:
o Limited scalability due to the shared bus.
o High contention for bus access leads to performance degradation as the system
scales up.
 Example: Bus-based topologies are typically used in small-scale systems like multi-
core desktop processors or embedded systems.

2.2 Ring Topology

 Definition: In a ring topology, processors are connected in a circular manner, with

each processor having a direct link to its two adjacent processors. Data circulates
around the ring in a set direction, and processors pass messages to one another in
sequence.
 Characteristics:
o Simple to Implement: Ring topologies are simple and inexpensive to
implement.
o Limited Communication Latency: The time to send data between two
processors is proportional to the number of processors in the system.
o Data Propagation: Data must travel through the ring until it reaches its
destination, which can increase latency as the system grows.
  Advantages:
o Simple design and easy to extend by adding processors.
o Ensures that all processors are interconnected in an orderly manner.
 Disadvantages:
o Latency: The distance between two processors in a large ring can increase
latency.
o Single Point of Failure: If one processor or link fails, it can break the entire
communication chain.
 Example: Token ring networks and some older network-on-chip (NoC) designs
used ring topologies.

2.3 Mesh Topology

 Definition: In a mesh topology, each processor is connected to multiple neighboring

processors, either in a two-dimensional or three-dimensional grid pattern. There are
two main types of mesh topologies: 2D Mesh and 3D Mesh.
 Characteristics:
o Multiple Communication Paths: Each processor has multiple direct
communication paths, which reduces congestion and improves fault tolerance.
o Scalable: Mesh topologies are more scalable than bus or ring topologies, as
adding more processors requires minimal changes to the existing system.
o Increased Complexity: While more scalable, mesh topologies are more
complex to design and manage, as multiple communication paths are required.
 Advantages:
o Higher Fault Tolerance: Since each processor is connected to multiple
others, failure of one processor or link does not necessarily disrupt
communication.
o Scalability: Easy to scale by adding rows and columns to the grid.
o Low Latency: Multiple paths reduce communication delays.
 Disadvantages:
o More Hardware: Requires more interconnects and wiring, which increases
hardware complexity and cost.
o Congestion: While latency is reduced, heavy traffic can cause congestion at
certain nodes in the network.
 Example: Modern multiprocessor servers and supercomputers often use mesh
topologies, especially in high-performance computing (HPC) environments.

2.4 Hypercube Topology

 Definition: In a hypercube topology, processors are arranged in a multi-dimensional

cube structure, where each processor is connected to other processors via edges of the
cube. A k-dimensional hypercube means that each processor is connected to k other
processors.
 Characteristics:
o Highly Parallel: Hypercube topology supports multiple parallel
communication paths, making it ideal for highly parallel computing tasks.
o Small Diameter: The maximum number of hops between any two processors
is relatively low compared to other topologies, ensuring efficient
communication even as the number of processors increases.
 o Exponential Scalability: The number of processors grows exponentially as
the dimensionality increases (i.e., doubling the dimension of the cube doubles
the number of processors).
 Advantages:
o Efficient Communication: The structure ensures that data can travel between
any two processors in just log2(N) hops, where N is the total number of
processors.
o Highly Scalable: The hypercube structure allows for efficient scaling as more
processors are added.
 Disadvantages:
o Complex Hardware: Building a hypercube network requires more complex
hardware for interconnections, which may make it cost-prohibitive for certain
applications.
o Routing Complexity: The routing algorithms for hypercubes can be more
complicated than for simpler topologies.
 Example: Hypercube topologies are commonly used in distributed systems, parallel
computing, and supercomputers.

2.5 Fat Tree Topology

 Definition: A fat tree is a type of network topology often used in data centers and
cloud computing environments. It uses a hierarchical, tree-like structure where the
inner nodes (routers) have more bandwidth than the outer nodes, ensuring that the
bottleneck does not occur in the network’s core.
 Characteristics:
o High Bandwidth: The architecture ensures that there is sufficient bandwidth
at the core of the network to support large-scale data transfers without
congestion.
o Scalable: Fat tree topologies can be easily scaled by adding more layers or
branching.
o Redundant Paths: It provides multiple paths between any two processors,
improving fault tolerance and reliability.
 Advantages:
o Fault Tolerant: Redundant paths ensure that the system remains operational
even if some paths fail.
o Balanced Traffic: The topology balances traffic across the network, reducing
bottlenecks.
 Disadvantages:
o Complex Routing: Fat tree topologies require more complex routing
algorithms, and the management of such networks can be more intricate.
o Higher Cost: The design complexity and need for more interconnection
hardware increase the cost.
 Example: Fat tree topologies are widely used in data center networks and cloud
computing environments.
1. Tree Topology

1.1 Definition

 Tree topology is a hierarchical structure where processors are arranged in a tree-like

configuration. It can be thought of as a star topology where each node (processor) is
connected to a central node (root), which in turn connects to other nodes, forming a
branching structure.

1.2 Characteristics

 Hierarchical Communication: In a tree topology, processors communicate with each

other through the hierarchical structure. Communication from one processor to
another may involve several intermediate nodes.
 Efficient for Broadcast: It is especially suitable for broadcasting messages to all
processors, as the message can be sent down the tree to all branches.

1.3 Advantages

 Scalable: New processors can be added to the tree without affecting the existing
network too much.
 Simple Design: The tree structure is relatively simple to design and manage.

1.4 Disadvantages

 Single Point of Failure: A failure at the root or any higher-level node can disrupt
communication across the entire system.
 Uneven Communication Delay: The distance between nodes can vary depending on
where they are in the tree, which can result in uneven communication latency.

2. Mesh-of-Trees Topology

2.1 Definition

 Mesh-of-trees is a combination of both mesh and tree topologies. It arranges

processors in a tree structure where each processor is also interconnected to form a
mesh with other processors in the system. The key feature is that processors can
communicate with multiple paths, increasing redundancy and fault tolerance.

2.2 Characteristics

 Redundancy: The mesh aspect of the topology ensures that processors can
communicate along multiple routes, avoiding potential bottlenecks in the network.
 Hierarchical and Parallel: The topology combines hierarchical communication
(from the tree) and parallel communication (from the mesh), making it flexible for
various types of workloads.
2.3 Advantages

 Fault Tolerance: Multiple communication paths ensure that the failure of a processor
or link will not disrupt the system completely.
 Balanced Traffic: Traffic is distributed across the mesh and tree structure, reducing
congestion at any single point.

2.4 Disadvantages

 Complex Design: The hybrid nature of the topology makes it more complex to
implement and manage compared to simpler topologies like bus or star topologies.

3. Complete Graph (Fully Connected) Topology

3.1 Definition

 In a complete graph topology, each processor is directly connected to every other

processor in the system. This creates a fully interconnected network where there are
no intermediate nodes between processors.

3.2 Characteristics

 No Communication Latency: Since all processors are directly connected to each

other, communication between processors is immediate, leading to very low
communication latency.
 Maximum Redundancy: Every processor has a direct link to every other processor,
providing the highest level of redundancy.

3.3 Advantages

 Low Latency: Data can be sent directly from one processor to another without
passing through intermediate processors.
 Fault Tolerance: The system can tolerate failures in individual processors or links, as
alternative paths are always available.

3.4 Disadvantages

 High Cost and Complexity: A complete graph requires an extremely large number of
interconnections, which is impractical for large systems due to the high cost and
hardware complexity.
 Scalability Issues: As the number of processors increases, the number of connections
grows quadratically, making this topology unscalable for large systems.

4. Star Topology

4.1 Definition

 In a star topology, all processors are connected to a central processor (or a

switch/router in some cases). The central node acts as the hub, facilitating
communication between the other nodes in the network.
4.2 Characteristics

 Centralized Control: The central node manages the communication between

processors. If the central node fails, the entire system loses connectivity.
 Simple and Easy to Implement: The design is simple, with each processor only
needing a single connection to the central hub.

4.3 Advantages

 Simple Design: The network is easy to set up and manage.

 Scalable: New processors can be added to the system by connecting them to the
central hub.

4.4 Disadvantages

 Single Point of Failure: The failure of the central processor or switch causes the
entire system to fail.
 Potential Bottleneck: All communication passes through the central node, which can
become a performance bottleneck as the number of processors increases.

5. Columnar or Torus Topology

5.1 Definition

 A columnar or torus topology is a variation of the mesh topology, where the network
is structured as a grid with the first and last rows (or columns) connected. This creates
a wraparound effect, making it a continuous loop.

5.2 Characteristics

 2D/3D Grid: In the simplest form, processors are arranged in a 2D grid, and
communication paths "wrap" around the edges, ensuring that every processor has a
direct connection to its neighbors.

5.3 Advantages

 Reduced Latency: The wraparound feature reduces the overall distance for
communication between processors, improving latency compared to standard mesh
topologies.
 Scalability: Like the mesh topology, torus topologies are scalable and can support a
large number of processors without congestion.

5.4 Disadvantages

 Complex Routing: The routing algorithms become more complicated because of the
wraparound connections, especially as the number of processors increases.
 Network Management: Managing a torus network can be more complex due to its
topology, requiring more sophisticated routing protocols.
6. Clusters and Clustered Interconnects

6.1 Definition

 In a clustered network, processors are grouped into smaller sets (clusters), and each
cluster is connected to a central interconnection network. This type of topology is
often used in distributed systems and data centers.

6.2 Characteristics

 Grouping Processors: Each cluster can operate relatively independently, with a

central network providing communication between clusters.
 Scalability: Clusters allow systems to scale by adding more clusters without
significantly affecting performance.

6.3 Advantages

 Modular Design: Clusters allow for modular expansion, making it easier to scale the
system.
 Fault Tolerance: If one cluster fails, the rest of the system can continue functioning.

6.4 Disadvantages

 Inter-Cluster Latency: Communication between clusters can be slower than within

clusters, as it involves the central interconnect.
 Complexity in Design: Managing and maintaining multiple clusters can introduce
complexity.

The choice of multiprocessor network topology is a critical factor that influences the
performance, scalability, and fault tolerance of a multiprocessor system. Different
topologies offer varying levels of efficiency in terms of communication speed, data transfer
capacity, and system reliability. For example, bus-based topologies are simple but not
scalable, while mesh and hypercube topologies offer higher scalability and lower latency at
the cost of increased complexity. As systems continue to grow in size and demand, the role of
network topology becomes increasingly significant in the design and performance of
multicore and multiprocessor systems, especially in supercomputing, cloud computing, and
parallel processing applications.