Computer Architecture:

Pipelining

Dr. Ashok Kumar Turuk

1
 Pipelining
 Pipelining?

 Pipeline is an implementation technique whereby instructions

are overlapped in execution

 It takes the advantage of parallelism that exists among the

actions needed to execute an instruction.

 Each steps in a pipeline completes a part of an instruction

 Different steps are completing different parts of different

instructions in parallel
2
 Pipelining
 Each steps is called a pipe stage or a pipe segment

 Time required between moving an instruction one step

down the pipeline is a processor cycle

 Length of a processor cycle is determined by the time

required for the slowest pipe stage

 Goal of pipeline designer is to balance the length of each

pipeline stage

3
Pipelining

4
 RISC Instruction Set
 Key properties of RISC architecture

 All operations on data apply to data in register

 Load and Store operations affects the memory

 Few instructions format, with all instructions typically being

one size
 The above property simplify the implementation of
pipeline

11
 RISC architecture
 Most RISC architecture have three classes of instructions
 ALU Instruction
 Load and Store Instruction
 Branches and Jumps: Branches are conditional transfer of
control. Two ways of specifying the branch condition in RISC
architecture:
 With a set of condition bits (condition code)
 Comparison between a pair of registers or between a register and
zero
 We consider only comparison for equality between two register
 In all RISC architecture, the branch address is obtained by
adding a sign-extended offset to the current PC

12
 A "Typical" RISC
 32-bit fixed format instruction (3 formats)
 Memory access only via load/store instructions
 32 64-bit GPR (R0 contains zero) and 32 64-bit FPR
 3-address, register-register arithmetic instruction; registers in
same place
 Single address mode for load/store:
 base + displacement
 no indirection
 Simple branch conditions
 Delayed branch (change the instruction sequence near a
branch instruction to avoid control penalty)

SPARC, MIPS, HP PA-Risc, DEC Alpha, IBM PowerPC,

13
 Example: MIPS (Note register location)
Register-Register
31 26 25 21 20 16 15 11 10 6 5 0

Op Rs1 Rs2 Rd Opx

Register-Immediate
31 26 25 21 20 16 15 0

Op Rs1 Rd immediate

Branch
31 26 25 21 20 16 15 0

Op Rs1 Rs2/Opx immediate

Jump / Call
31 26 25 0

Op target

14
 Implementation of RISC Instructions
 Instruction Fetch Cycle (IF)
 IR <- Mem[PC]; IR holds the instruction
 NPC <- PC + 4

 Instruction decode/register fetch cycle (ID)

 Decode the instruction and read the registers corresponding to
register source specifiers.
 Do the equality test on the registers as they are read for a
possible branch.
 Sign-extend the offset field
 Compute the possible branch target address
 Decoding is done in parallel with reading registers (why?)

15
 Implementation of RISC Instructions
 Execution/Effective address cycle (EX)
 The ALU operates on the operands and perform one of 3
functions:
 Memory reference: ALU adds the base register and the offset to
form the effective address.
 R-R ALU instruction: ALU perform the operation specified by
the ALU opcode on the values read from register files.
 R-Immediate ALU instruction: ALU performs the operation
specified by the ALU opcode on the first value read from the
register file and the sign-extended immediate
 Why can we combine the effective address and
execution cycles into a single clock cycle?

16
 Implementation of RISC Instructions
 Memory Access
 If load, memory does a read using the effective address.
 If store, then the memory writes the data from the register using the effective
address.
 Write-back cycle(WB)
 Register-Register ALU instruction: Regs[IR16…20] <- ALU output
 Register-Immediate ALU inst: Regs[IR16…20] <- ALU output
 Load instruction: Regs[IR11…15] <- LMD

 Branches:
 2 cycles
 Store:
 4 cycles
 Rest of Instructions:
 5 cycles
17
 Implementation of RISC Instructions
 What will be CPI for the above implementation,
assuming a branch frequency of 12% and a store
frequency of 10% ?

18
 Five-Stage Pipeline for RISC Processor
Instruction Clock number
number
1 2 3 4 5 6 7 8 9

Instruction i IF ID EX MEM WB

Instruction i + 1 IF ID EX MEM WB

Instruction i + 2 IF ID EX MEM WB

Instruction i + 3 IF ID EX MEM WB

Instruction i + 4 IF ID EX MEM

19
Figure C.2 The pipeline can be thought of as a series of data paths shifted in time. This shows the overlap among the parts of
the data path, with clock cycle 5 (CC 5) showing the steady-state situation. Because the register file is used as a source in the ID
stage and as a destination in the WB stage, it appears twice. We show that it is read in one part of the stage and written in another by
using a solid line, on the right or left, respectively, and a dashed line on the other side. The abbreviation IM is used for instruction
memory, DM for data memory, and CC for clock cycle.

Copyright © 2011, Elsevier Inc. All rights Reserved. 20

 Why does it work?
 Use separate I and D caches
 Register file can be read/written in 0.5 cycles
 PC: incremented in IF
 if branch taken, PC is not until ID stage
 Cannot keep any state in IR need to move it to
another register every cycle
 These registers IF/ID, ID/EX, EX/MEM, MEM/WB subsume
the temp ones
 EX., destination Reg in a LD

21
Figure C.3 A pipeline showing the pipeline registers between successive pipeline stages. Notice that the registers prevent
interference between two different instructions in adjacent stages in the pipeline. The registers also play the critical role of carrying
data for a given instruction from one stage to the other. The edge-triggered property of registers—that is, that the values change
instantaneously on a clock edge—is critical. Otherwise, the data from one instruction could interfere with the execution of another!

Copyright © 2011, Elsevier Inc. All rights Reserved. 22

 Performance Issues in Pipeline
 Increases CPU instruction throughput but it does not
reduces the execution time of individual instruction
 Pipeline overhead increases the execution time of each
instruction.
 A program runs faster and has lower total execution
time, even though no single instruction runs faster!

23
 Pipeline Limitations
 Limitation due to pipeline latency
 Execution time of each instruction does not decrease puts limits on the
practical depth of pipeline
 Imbalances among the pipe stages
 Reduces the performance since the clock can run faster than
the slowest pipeline stages
 Pipeline overhead
 Arises from the combination of pipeline register delay and
clock skew
 Pipeline register add setup time plus propagation delay to the
clock cycle
 Clock skew, is the maximum delay between when the clock
arrives at any two register

24
 Pipeline Limitations

 Once the clock cycle is as small as the sum of the clock

skew and latch overhead, no further pipelining is useful
since there is no time left in the cycle for useful work.

25
 Examples
 Unpipelined Processor:
 10 ns cycle time
 4 cycles for ALU operations (40%), branch (20%)
 5 cycles for memory operations (40%)
 Pipelining Processor:
 adds 1 ns of overhead to the clock (due to clock skew and setup)
 How much speedup in the instruction execution rate will we gain
from the pipeline? Ignore any latency impact.
 Unpipelined:
 average instruction execution time = clock * avg CPI
 10 * ((40%+20%)*4+40%*5) = 44 ns
 Pipelined = 11ns
 Speedup = 44/11 = 4
26
 Pipeline Hazards

 Hazards are situations that prevent the next instruction in

the instruction stream from executing during its
designated clock cycle

 Hazards reduces the performance from the ideal speedup

gained by pipelining

27
 Pipeline Hazards
 Three types of hazards
 Structural
 Data
 Control

28
 Structural Hazards

 Arise from resource conflicts when the hardware

cannot support all possible combinations of
instructions simultaneously in overlapped execution

29
 Data Hazards

•Arise when an instruction depends on the results of a previous

instruction in way that is exposed by the overlapping of
instructions in the pipeline

•Ex. ADD R1, R2, R3

ADD R4, R5, R1

30
 Control Hazards

 Arises from the pipelining of branches and other instructions

that changes the PC

• EX. BEQ R1, label

ADD R7, R6, R7

 Hazards in pipeline can make it necessary to stall the pipeline

Stall degrades the pipeline performance from the ideal performance

31
Performance of Pipelines with Stalls

32
Performance of Pipelines with Stalls

33
Performance of Pipelines with Stalls

34
Performance of Pipelines with Stalls

35
Performance of Pipelines with Stalls

36
Performance of Pipelines with Stalls

37
 Structural Hazards
 Some combination of instructions cannot be
accommodated because of resource conflicts
 Usually because some functional unit is not fully
pipelined. Two instructions using it cannot proceed at the
rate of one/cycle.
 Some resource has not been replicated enough
 Ex. 1 register file write port, combined I, D memory
 Result: Pipeline stall one of the instructions until the
required unit is available.

• Pipeline bubble

38
Figure C.4 A processor with only one memory port will generate a conflict whenever a memory reference occurs. In this
example the load instruction uses the memory for a data access at the same time instruction 3 wants to fetch an instruction from
memory.

Copyright © 2011, Elsevier Inc. All rights Reserved. 39

Instruction 1 2 3 4 5 6 7 8 9 10

Load Instruction IF ID EX MEM WB

Instruction i + 1 IF ID EX MEM WB

Instruction i + 2 IF ID EX ME WB
M
Instruction i + 3 Stall IF ID EX MEM WB

Instruction i + 4 IF ID EX MEM WB

Instruction i + 5 IF ID EX MEM

Instruction i + 6 IF ID EX

40
 Structural Hazards
 Example: 40 % of instructions are data reference.
Ideal CPI of unpipelined processor, ignoring the
structural hazards is 1. Assume that the processor with
structural hazard has a clock rate that is 1.05 times
higher than the clock rate of the processor without
hazard. Disregarding any performance losses, is the
pipeline with or without the structural hazards faster,
and by how
 Average instruction timeideal = 1*clock cycle timeideal
 Average instruction timehazard = CPI * Clock cycle time /
= (1+0.4*1)*clock cycle timeideal /1.05
= 1.3*(clock cycle time)

41
 Structural Hazards
 Why allow structural hazards?
 Reduce cost
 If structural hazards are rare, it may not be
worth the cost to avoid

42
 Data Hazards

Data hazards occurs when the pipeline changes

the order of read/write accesses to operands so
that the order differs from the order seen by
sequentially executing instructions on an
unpipelined processor

43
 Data Hazards

DADD R1, R2, R3

DSUB R4, R1,R5
AND R6, R1,R7
OR R8, R1,R9
XOR R10, R1,R11

44
Figure C.6 The use of the result of the DADD instruction in the next three instructions causes a hazard, since the register is
not written until after those instructions read it.

Copyright © 2011, Elsevier Inc. All rights Reserved.

45
 Minimizing Data Hazard Stalls by Forwarding

 Forwarding is also know as bypassing and short-circuiting

 Forwarding works as follows:

 ALU result from both the EX/MEM and MEM/WB

pipeline register is always fed back to the ALU inputs

 If forwarding hardware detects that the previous ALU

operation has written the register corresponding to a
source for the current ALU operation, control logic
selects the forwarded result as the ALU input rather
than the value read from the register file.

46
Figure C.7 A set of instructions that depends on the DADD result uses forwarding paths to avoid the data hazard. The inputs
for the DSUB and AND instructions forward from the pipeline registers to the first ALU input. The OR receives its result by
forwarding through the register file, which is easily accomplished by reading the registers in the second half of the cycle and writing
in the first half, as the dashed lines on the registers indicate. Notice that the forwarded result can go to either ALU input; in fact,
both ALU inputs could use forwarded inputs from either the same pipeline register or from different pipeline registers. This would
occur, for example, if the AND instruction was AND R6,R1,R4.

Copyright © 2011, Elsevier Inc. All rights Reserved. 47

 Minimizing Data Hazard Stalls by Forwarding

 In Forwarding the result is passed directly to the functional

unit that requires it:

 A result is forwarded from the pipeline register

corresponding to the output of one unit to the input of
another, rather than just from result of a unit to the input
of the same unit.

DADD R1, R2, R3

LD R4, 0(R1)
SD R4, 12(R1)

48
Figure C.8 Forwarding of operand required by stores during MEM. The result of the load is forwarded from the memory
output to the memory input to be stored. In addition, the ALU output is forwarded to the ALU input for the address calculation of
both the load and the store (this is no different than forwarding to another ALU operation). If the store depended on an immediately
preceding ALU operation (not shown above), the result would need to be forwarded to prevent a stall.

Copyright © 2011, Elsevier Inc. All rights Reserved. 49

 Data hazards that require stall

LW R1, 0(R2)
SUB R4, R1, R5
AND R6, R1, R7
OR R8, R1, R9

50
Figure C.9 The load instruction can bypass its results to the AND and OR instructions, but not to the DSUB, since that
would mean forwarding the result in “negative time.”

Copyright © 2011, Elsevier Inc. All rights Reserved.

51
 How to handle these hazards
 Load instruction has a delay or latency that cannot be
eliminated by forwarding

 Need to add hardware, called pipeline interlock to

preserve the correct execution pattern

 A pipeline interlock detects a hazard and stall the

pipeline until the hazard is cleared
 The CPI of the stalled instruction increases by the
length of the stall.

52
Instruction 1 2 3 4 5 6 7 8

LD R1, 0(R2) IF ID EX MEM WB

DSUB R4, R1, R5 IF ID EX MEM WB

AND R6, R1, R7 IF ID EX MEM WB
OR R8, R1, R9 IF ID EX MEM WB

Instruction 1 2 3 4 5 6 7 8

LD R1, 0(R2) IF ID EX MEM WB

DSUB R4, R1, R5 IF ID Stall EX MEM WB

AND R6, R1, R7 IF Stall ID EX MEM WB
OR R8, R1, R9 Stall IF ID EX MEM

53
 Branch Hazards
 When a branch is executed, it may or may not the PC to
something other than its current value plus 4

 If taken, the PC is not changed (usually) until the end of

ID, after the completion of address calculation and
comparison.

 Simplest method of dealing with branches is to redo the

fetch of the instruction following a branch, once we
detect the branch during ID

54
Instruction 1 2 3 4 5 6 7 8

Branch Instruction IF ID EX MEM WB

Branch successor IF IF ID EX MEM WB

Branch successor IF ID EX MEM WB
+1
Branch successor IF ID EX MEM WB
+2

One stall cycle for every branch will yield a performance loss
of 10% to 30% depending on branch frequency

55
 Reducing Pipeline Branch Penalties
 Freeze or Flush the Pipeline
 Delete any instruction after the branch until branch destination
is known
 Branch penalty is fixed and cannot be reduced by software

Instruction 1 2 3 4 5 6 7 8

Branch Instruction IF ID EX MEM WB

Branch successor IF IF ID EX MEM WB

Branch successor IF ID EX MEM WB
+1
Branch successor IF ID EX MEM WB
+2

56
 Reducing Pipeline Branch Penalties
 Predicted-Not-Taken (Predicted-untaken)

 Execute successor instructions as if the branch were a

normal instruction
 Do not change the processor state until branch outcome is
known definitely
 If branch actually taken, turn the fetched instruction into a
no-op and restart the fetch at the target

57
Instruction 1 2 3 4 5 6 7 8

Untaken Branch IF ID EX MEM WB

Instruction
Instruction i + 1 IF ID EX MEM WB
Instruction i + 2 IF ID EX MEM WB
Instruction i + 3 IF ID EX MEM WB

Instruction 1 2 3 4 5 6 7 8

Taken Branch IF ID EX MEM WB

Instruction
Instruction i + 1 IF Idle Idle Idle Idle
Branch Target IF ID EX MEM WB
Branch Target + 1 IF ID EX MEM WB

58
 Reducing Pipeline Branch Penalties
 Predicted-Taken

 Treat every branch as taken

 Compute the target as soon as the branch is decoded.
 Start fetching and executing at the target

 In either predict-taken or predict-not-taken scheme, the

compiler can improve the performance by organizing the
code so that the most frequent path matches the hardware’s
choices

59
 Reducing Pipeline Branch Penalties
 Delayed Branch: Heavily used in RISC processor
 Define branch to take place AFTER a following instruction

branch instruction

sequential successor1
sequential successor2 Branch delay of length n
........
sequential successorn

branch target if taken

 The instruction in the branch delay slots following the

branch will be executed irrespective if the outcome of the
branch
60
Instruction 1 2 3 4 5 6 7 8

Untaken Branch IF ID EX MEM WB

Instruction
Br. Delay IF ID EX MEM WB
instruction (i + 1)
Instruction i + 2 IF ID EX MEM WB
Instruction i + 3 IF ID EX MEM WB

Instruction 1 2 3 4 5 6 7 8

Taken Branch IF ID EX MEM WB

Instruction
Br. Delay IF ID EX MEM WB
instruction (i + 1)
Branch Target IF ID EX MEM WB
Branch Target + 1 IF ID EX MEM WB

61
 Reducing Pipeline Branch Penalties
 Almost all processor with delayed branch have a single
instruction delay

 Job of compiler is to make the successor instruction

valid and useful

 Where to get instructions to fill branch delay slot?

Before branch instruction
From the target address: only valuable when branch taken
From fall through: only valuable when branch not taken

62
Figure C.14 Scheduling the branch delay slot. The top box in each pair shows the code before scheduling; the bottom box shows
the scheduled code. In (a), the delay slot is scheduled with an independent instruction from before the branch. This is the best choice.
Strategies (b) and (c) are used when (a) is not possible. In the code sequences for (b) and (c), the use of R1 in the branch condition
prevents the DADD instruction (whose destination is R1) from being moved after the branch. In (b), the branch delay slot is
scheduled from the target of the branch; usually the target instruction will need to be copied because it can be reached by another
path. Strategy (b) is preferred when the branch is taken with high probability, such as a loop branch. Finally, the branch may be
scheduled from the not-taken fall-through as in (c). To make this optimization legal for (b) or (c), it must be OK to execute the
moved instruction when the branch goes in the unexpected direction. By OK we mean that the work is wasted, but the program will
still execute correctly. This is the case, for example, in (c) if R7 were an unused temporary register when the branch goes in the
unexpected direction.

Copyright © 2011, Elsevier Inc. All rights Reserved. 63

 Reducing Pipeline Branch Penalties

Limitations on delayed branch scheduling

 Restrictions on the instruction that are scheduled into delay slots

 Ability to predict at compile time whether a branch is taken or not

To improve the ability of the compiler to fill branch delay

slots, most processors with conditional branches have
introduced a cancelling or nullifying branch.
 Instruction in the delay slot includes the direction that the branch
was predicted
 For correct prediction instruction in the delay slot is executed
 For incorrect prediction instruction in the delay slot is turned into
no-op.

64
Performance of Branch Schemes

65
 Reducing Branch Cost Through Prediction
 As pipeline gets deeper and the potential penalty of
branches increases, using delayed branches and similar
schemes become insufficient

 We need more aggressive means for predicting branches

 Static branch prediction
 Compile time
 Dynamic branch prediction
 Run time

66
 Static Branch Prediction
 Use profile information collected from earlier runs
 Key observation : Behavior of branches is often
bimodally distributed
 An individual branch is often highly biased towards taken or
untaken

 Effectiveness of any branch prediction scheme depends

both on the accuracy of the scheme and the frequency of
conditional branches

67
Figure C.17 Misprediction rate on SPEC92 for a profile-based predictor varies widely but is generally better for the floating-
point programs, which have an average misprediction rate of 9% with a standard deviation of 4%, than for the integer
programs, which have an average misprediction rate of 15% with a standard deviation of 5%. The actual performance depends
on both the prediction accuracy and the branch frequency, which vary from 3% to 24%.

Copyright © 2011, Elsevier Inc. All rights Reserved. 68

Dynamic Branch Prediction and Branch-Prediction Buffer
 Branch-Prediction Buffer or branch history table
 A small memory indexed by the lower portion of the address of the branch
instruction,
 The memory contains a bit that says whether the branch was recently taken
or not
 Simple, and useful only when the branch delay is longer than the time to
calculate the target address
 The prediction bit is inverted each time there is a wrong
prediction
 The prediction buffer is a cache where access is a hit
 Performance of prediction buffer depends on
 How often the prediction is for the branch of interest
 How accurate the prediction is when it matches

69
Dynamic Branch Prediction and Branch-Prediction Buffer
 1-bit prediction has performance shortcoming
 Even if a branch is almost always taken, likely to mispredict twice rather
than once, when it is not taken

70
 Dynamic Branch Prediction
 2 bit scheme a prediction must miss twice before it is
changed
 A branch-prediction buffer can be implemented as a
small special cache accessed with the instruction
addressed during the IF stage or as a pair of bits attached
to each block in the instruction cache and fetched with
the instruction.

71
Figure C.18 The states in a 2-bit prediction scheme. By using 2 bits rather than 1, a branch that strongly favors taken or not taken
—as many branches do—will be mispredicted less often than with a 1-bit predictor. The 2 bits are used to encode the four states in
the system. The 2-bit scheme is actually a specialization of a more general scheme that has an n-bit saturating counter for each entry
in the prediction buffer. With an n-bit counter, the counter can take on values between 0 and 2n – 1: When the counter is greater than
or equal to one-half of its maximum value (2n – 1), the branch is predicted as taken; otherwise, it is predicted as untaken. Studies of
n-bit predictors have shown that the 2-bit predictors do almost as well, thus most systems rely on 2-bit branch predictors rather than
the more general n-bit predictors.

Copyright © 2011, Elsevier Inc. All rights Reserved. 72

Figure C.19 Prediction accuracy of a 4096-entry 2-bit prediction buffer for the SPEC89 benchmarks. The misprediction rate for
the integer benchmarks (gcc, espresso, eqntott, and li) is substantially higher (average of 11%) than that for the floating-point
programs (average of 4%). Omitting the floating-point kernels (nasa7, matrix300, and tomcatv) still yields a higher accuracy for the
FP benchmarks than for the integer benchmarks. These data, as well as the rest of the data in this section, are taken from a branch-
prediction study done using the IBM Power architecture and optimized code for that system. See Pan, So, and Rameh [1992].
Although these data are for an older version of a subset of the SPEC benchmarks, the newer benchmarks are larger and would show
slightly worse behavior, especially for the integer benchmarks.

Copyright © 2011, Elsevier Inc. All rights Reserved. 73

Figure C.20 Prediction accuracy of a 4096-entry 2-bit prediction buffer versus an infinite buffer for the SPEC89
benchmarks. Although these data are for an older version of a subset of the SPEC benchmarks, the results would be comparable for
newer versions with perhaps as many as 8K entries needed to match an infinite 2-bit predictor.

Copyright © 2011, Elsevier Inc. All rights Reserved. 74

 Pipelined Implementation of MIPS
Every MIPS instruction can be implemented at most 5 CC.
They are
Instruction Fetch Cycle (IF)
IR ← Mem[PC];
NPC ← PC + 4;
 Instruction decode / register fetch cycle (ID)
A ← Regs[rs];
B ← Regs[rt];
Imm ← sign-extended immediate field of IR

75
 Pipelined Implementation of MIPS
 Execution / Effective address cycle (EX):
 ALU performs one of the four functions depending on the MIPS
instruction type on the operand prepared in the prior cycle
 Memory reference
 ALUOutput ← A + Imm
 Register-Register ALU instruction
 ALUOutput ← A func B
 Register-Immediate ALU instruction
 ALUOutput ← A op Imm
 Branch
 ALUOutput ← NPC + (Imm << 2) [shifted to create word offset]
 Cond ← (A = = 0)

76
 Pipelined Implementation of MIPS
 Memory Access/ Branch Completion Cycle (MEM)
 PC is updated for all instruction: PC ← NPC
 Memory reference
LMD ← Mem[ALUOutput] or
Mem[ALUOutput] ← B;
 Branch
If (cond) PC ← ALUOutput

77
 Pipelined Implementation of MIPS
 Write-back cycle (WB)
 Register-Register ALU instruction:
 Regs[rd] ← ALUOutput
 Register-immediate ALU instruction:
 Regs[rt] ← ALUOutput
 Load instruction:
 Regs[rt] ← LMD

78
Figure C.21 The implementation of the MIPS data path allows every instruction to be executed in 4 or 5 clock cycles.
Although the PC is shown in the portion of the data path that is used in instruction fetch and the registers are shown in the portion of
the data path that is used in instruction decode/register fetch, both of these functional units are read as well as written by an
instruction. Although we show these functional units in the cycle corresponding to where they are read, the PC is written during the
memory access clock cycle and the registers are written during the write-back clock cycle. In both cases, the writes in later pipe
stages are indicated by the multiplexer output (in memory access or write-back), which carries a value back to the PC or registers.
These backward-flowing signals introduce much of the complexity of pipelining, since they indicate the possibility of hazards.

Copyright © 2011, Elsevier Inc. All rights Reserved. 79

 Single vs Multicycle
 Designers would never use single-cycle implementation
for two reasons:

 A single-cycle implementation would be very inefficient for

most CPUs that have a reasonable variation among the amount
of work, and hence in the clock cycle time, needed for
different instructions

 A single-cycle implementation requires the duplication of

functional unit that could be shared in a multicycle
implementation

80
Figure C.22 The data path is pipelined by adding a set of registers, one between each pair of pipe stages. The registers serve to
convey values and control information from one stage to the next. We can also think of the PC as a pipeline register, which sits
before the IF stage of the pipeline, leading to one pipeline register for each pipe stage. Recall that the PC is an edge-triggered register
written at the end of the clock cycle; hence, there is no race condition in writing the PC. The selection multiplexer for the PC has
been moved so that the PC is written in exactly one stage (IF). If we didn’t move it, there would be a conflict when a branch
occurred, since two instructions would try to write different values into the PC. Most of the data paths flow from left to right, which
is from earlier in time to later. The paths flowing from right to left (which carry the register write-back information and PC
information on a branch) introduce complications into our pipeline.

Copyright © 2011, Elsevier Inc. All rights Reserved. 81

 Events in the IF, ID Stage of MIPS
 IF :
 IF/ID.IR  Mem[PC]
 IF/ID.NPC, PC  (if ((EX/MEM.opcode = = branch) & EX/MEM.cond)
{EX/MEM.ALUOutput} else {PC + 4})

 ID:
 ID/EX.A  Regs[IF/ID.IR[rs]]; ID/EX.B  Regs[IF/ID.IR[rt]]
 ID/EX.NPC  IF/ID.NPC, ID/EX.IR  IF/ID.IR
 ID/EX.Imm  sign-extended(IF/ID.IR[immediate field])

82
 Events in the EX Stage of MIPS
 ALU Instruction:
 EX/MEM.IR  ID/EX.IR
 EX/MEM.ALUOutput ID/EX.A func ID/EX.B OR
 EX/MEM.ALUOutput ID/EX.A op ID/EX.Imm
 Load or Store Instruction
 EX/MEM.IR  ID/EX.IR
 EX/MEM.ALUOutput ID/EX.A + ID/EX.Imm
 EX/MEM.B ID/EX.B
• Branch Instruction
 EX/MEM.IR  ID/EX.IR
 EX/MEM.ALUOutput  ID/EX.NPC + (ID/EX.Imm << 2)
 EX/MEM.Cond  (ID/EX.A = = 0)

83
 Events in the MEM Stage of MIPS
 ALU Instruction:
 MEM/WB.IR  EX/MEM.IR
 MEM/WB.ALUOutput  EX/MEM.ALUOutput

• Load or Store Instruction

 MEM/WB.IR  EX/MEM.IR
 MEM/WB.LMD  Mem[EX/MEM.ALUOutput] OR
 Mem[EX/MEM.ALUOutput]  EX/MEM.B

 Branch Instruction

84
 Events in the WB Stage of MIPS
• ALU Instruction:
 Regs[MEM/WB.IR[rd]]  MEM/WB.ALUOutput or
 Regs[MEM/WB.IR[rt]]  MEM/WB.ALUOutput

 Load Instruction
 Regs[MEM/WB.IR[rt]]  MEM/WB.LMD

• Branch Instruction

85
86
 Implementing Control for MIPS Pipeline
 Instruction Issue
 Process of letting an instruction move from ID into EX stage
 In MIPS integer pipeline, all data hazards can be
detected in ID stage.
 If data hazard exists, stall the instruction
 What forwarding logic needed can be determined during ID
stage and set appropriate control
 Detecting interlocks early in the pipeline reduces the hardware
complexity
 Alternatively, we can detect hazard or forwarding at the
beginning of clock cycle that uses an operand.

87
 Implementing Control for MIPS Pipeline
 Implementation of interlock in ID stage for RAW
hazards with source coming from a load instruction (load
interlock).

 The implementation of forwarding paths to the ALU

inputs can be done during EX

88
Hazard Detection H/W can see by comparing the
destination and source of adjacent instruction

 No Dependence
 LD R1, 45(R2)
 DADD R5, R6, R7
 DSUB R8, R6, R7
 OR R9, R6, R7
 Dependence Requiring Stall
 LD R1, 45(R2)
 DADD R5, R1, R7
 DSUB R8, R6, R7
 OR R9, R6, R7
89
Hazard Detection H/W can see by comparing the
destination and source of adjacent instruction
 Dependence overcome by forwarding
 LD R1, 45(R2)
 DADD R5, R6, R7
 DSUB R8, R1, R7
 OR R9, R6, R7
 Dependence with accesses in order
 LD R1, 45(R2)
 DADD R5, R6, R7
 DSUB R8, R6, R7
 OR R9, R1, R7

90
91
 Load Interlock

If there is a RAW hazard with the source instruction

being a load, the load instruction will be the EX stage
when an instruction that needs the load data will be in the
ID stage

92
Logic to detect the need for load interlocks during the
ID stage of an instruction requires three comparisons

Opcode Field of ID/EX Opcode Field of IF/ID Matching Operand Fields

(ID/EX.IR0..5) (IF/ID.IR0..5)

Load Register – Register ALU ID/EX.IR[rt] = =

IF/ID.IR[rs]
Load Register – Register ALU ID/EX.IR[rt] = =
IF/ID.IR[rt]
Load Load, Store, ALU ID/EX.IR[rt] = =
Immediate, or Branch IF/ID.IR[rs]

93
94
95
Figure C.27 Forwarding of results to the ALU requires the addition of three extra inputs on each ALU multiplexer and the
addition of three paths to the new inputs. The paths correspond to a bypass of: (1) the ALU output at the end of the EX, (2) the
ALU output at the end of the MEM stage, and (3) the memory output at the end of the MEM stage.

Copyright © 2011, Elsevier Inc. All rights Reserved. 96

Figure C.28 The stall from branch hazards can be reduced by moving the zero test and branch-target calculation into the ID
phase of the pipeline. Notice that we have made two important changes, each of which removes 1 cycle from the 3-cycle stall for
branches. The first change is to move both the branch-target address calculation and the branch condition decision to the ID cycle.
The second change is to write the PC of the instruction in the IF phase, using either the branch-target address computed during ID or
the incremented PC computed during IF. In comparison, Figure C.22 obtained the branch-target address from the EX/MEM register
and wrote the result during the MEM clock cycle. As mentioned in Figure C.22, the PC can be thought of as a pipeline register (e.g.,
as part of ID/IF), which is written with the address of the next instruction at the end of each IF cycle.

97
Copyright © 2011, Elsevier Inc. All rights Reserved.
98
What makes pipeline hard to
implement?

99
 Dealing with Exceptions
 Exceptional situations are harder to handle in a pipelined
CPU

 Overlapping of instructions makes it more difficult to know

whether an instruction can safely change the state of the CPU

 An instruction in the pipeline can raise exceptions that may

force the CPU to abort the instructions in the pipeline before
they complete

100
 Classification
 Resume vs terminate
 Terminate: Program execution always stops after the interrupt.
Easier to implement since no need to restart the execution
 Resume: Program execution continues after the interrupt

105
 Difficulty
 It is difficult in implementing interrupts occurring within
instructions where the instructions must be resumed
 How to implement such exceptions?
 Invoke another program that
 Save the state of the executing program
 Correct the cause of the execution program
 Restore the state of the program before the instruction
caused exception
 Retry the instruction
 Above steps must be invisible to executing program

107
 Difficulty

 If pipeline allows machine to save state, handle

exception, and restart without affecting the execution of
program: restartable machine or pipeline
 All machines today are restartable (at least for integer
pipeline)

108
 Stopping and Restarting Execution

 Most difficult exceptions have two properties:

 They occur within the instruction (middle of the instruction
execution corresponding to EX or MEM pipe stage)
 They must be restartable
 Example: VM page fault from a data fetch
 Occurs when one instruction enters the MEM stage
 Other instructions already in the pipeline
 OS must be invoked, save the state in the pipeline, move
page to memory
 The pipeline must be safely shut down and the state saved,
so that instruction can be restarted in the correct state

109
 Stopping and Restarting Execution
 Restarting is usually implemented by
 Saving the PC of the instruction at which to restart

 Restarted instruction is not a branch

 Then fetch the sequential successors and begin their execution in the
normal fashion

 Restarted instruction is a branch

 Then reevaluate the branch condition and begin fetching from either
the target or fall-through.

110
 Saving the Pipeline State Safely
 Pipeline control can take the following steps to save the
pipeline safely:

 Force a trap instruction into the pipeline on the next IF (the

entry point of an OS trap)

 Until the trap is taken, turn off all writes for the faulting
instruction and for all instructions that follow in the pipeline
 Above is done by placing all zeros into the pipeline
latches of all instructions in the pipeline, starting with the
instruction that caused the exception but not those that
preceded that instruction. This prevents any writes and,
therefore, changing any state

111
 Saving the Pipeline State Safely
 Pipeline control can take the following steps to save the
pipeline safely
 After the exception-handling routine in the OS receives
control, it immediately saves the PC of the faulting
instruction. This value will be used to return from the
exception later.

 Suppose we have delayed branches.

 We need to save and restore as many PC as the length of
the branch delay plus 1

112
Returning Machine From Exception
 Last instruction in the OS trap handler is a special
instruction that returns the PC of the offending instruction
 The next instruction to execute is the one causing the exception
 A pipeline is said to have precise exceptions, if the
pipeline can be stopped so that the instructions just before
the faulting instruction are completed and those after it can
be restarted from the scratch
 Programmers prefer precise exceptions: easy to handle in
integer pipeline

113
 Returning Machine From Exception
 High performance CPUs have introduced two modes of
operations
 One mode precise exceptions and the other does not

 Precise exception mode is slower since it allows less

overlapping among FP instructions

114
 Precise Exception for MIPS Integer Pipeline
 Exception that might occur in each MIPS pipe stage
 IF: Page fault on I-fetch, misaligned memory access, memory protection
violation
 ID: undefined/ illegal opcode
 EX: arithmetic exception
 MEM: Page fault on D-fetch, misaligned memory access, memory
protection violation
 WB: none

115
 Precise Exception for MIPS Integer Pipeline

 In a pipeline, multiple exceptions may occur in the

same CC because there are multiple instructions in
execution

 Example: Consider the following sequence of

instruction

LD IF ID EX MEM WB
DAAD IF ID EX MEM WB

116
 Precise Exception for MIPS Integer Pipeline

 Exceptions may occur at the same stage

 Exceptions may occur out of order: An instruction can

cause an exception before an earlier instruction causes
one.

LD IF ID EX MEM WB
DAAD IF ID EX MEM WB

117
 Precise Exception for MIPS Integer Pipeline
 Pipeline does not handle the exception when it occurs in time.
 This is because, it will lead to exceptions occurring out of the unpipelined
order.
 Hardware posts all exceptions caused by a given instruction in a
status vector associated with that instruction
 Once an exception indication is set in the exception status vector,
any control signal that may cause a data value to be written is
turned off
 WB stage, exception status vector is checked. Exception are
handled in the order in which they would occur in time on an
unpipelined processor.

118
 Handling Multicycle Operations
 Floating-point operations usually take several EX cycles
 FP operation completing in 1 or 2 CC required
 A slow clock or using enormous amounts of logic in FP units or
both
 FP instruction having the same pipeline as the integer
instruction
 EX cycle is repeated as many times as needed to complete the
operation
 Multiple FP functional units
 A stall will occur, if the instruction to be issued, cause
 Structural hazard for the FU it uses or
 Data hazard

119
 Handling Multicycle Operations
 Assume there are four separate functional units:
 Integer unit: handles Load, Store, Integer ALU operation and
Branch
 FP and Integer multiplier
 FP adder: Handles FP add, subtract, and conversion
 FP and integer divider

 Assume that execution stages of these FU are not pipelined

 Instruction is issued to the FU only when it is free
 If an instructions cannot proceed to EX, entire pipeline
behind that instruction will be stalled

120
Figure C.33 The MIPS pipeline with three additional unpipelined, floating-point, functional units. Because only one
instruction issues on every clock cycle, all instructions go through the standard pipeline for integer operations. The FP operations
simply loop when they reach the EX stage. After they have finished the EX stage, they proceed to MEM and WB to complete
execution.

Copyright © 2011, Elsevier Inc. All rights Reserved. 121

 How to Pipeline EX stage
 Latency: Number of intervening cycles between an instruction
that produces a result and one that uses it

 Initiation interval or repeat interval: Number of cycles that

must elapse between issuing two operations of a given type

FU Latency Initiation interval

Integer ALU 0 1
Integer/Float Loads 1 1
Float add 3 1
Float/int multiply 6 1
Float/int divide 24 25
 Allow 4 outstanding FP adds, 7 Float/integer multiply and 1
Float/int div
122
Figure C.35 A pipeline that supports multiple outstanding FP operations. The FP multiplier and adder are fully pipelined and
have a depth of seven and four stages, respectively. The FP divider is not pipelined, but requires 24 clock cycles to complete. The
latency in instructions between the issue of an FP operation and the use of the result of that operation without incurring a RAW stall
is determined by the number of cycles spent in the execution stages. For example, the fourth instruction after an FP add can use the
result of the FP add. For integer ALU operations, the depth of the execution pipeline is always one and the next instruction can use
the results.

Copyright © 2011, Elsevier Inc. All rights Reserved. 123

124
 Introduction
 Consider two instruction i and j, with instruction i
preceding j in the program order . The possible data
hazards are:

 Read after write (RAW):

 j tries to read a source before i writes to it
 Write after write (WAW)
 j tries to write an operand before it is written by i
 Write after read (WAR)
 j tries to write a destination before it is read by i

Copyright © 2012, Elsevier Inc. All rights reserved. 125

 Hazards and Forwarding in Longer
Latency Pipeline
 Different aspect to the hazard detection and forwarding for a
longer latency pipeline.
 Divide is not fully pipelined, structural hazards can occur
 Because instructions have varying running times, the number of register
writes required in a cycle can be longer than 1
 WAW hazard is possible since instruction no longer reach WB stage in
order. WAR is not possible, since register read always occur in ID
 Instruction can complete out of order than they are issued, causing
problems with exceptions
 Because of longer latency operations, stalls for RAW hazards will be
more frequent

126
127
128
 How to Prevent Multiple WB?
 Track the use of write port in the ID stage and stall an instruction
before it issues.
 A shift register can be used to indicate when an issued instruction will use
the register file
 The instruction in the ID stage is stalled for a cycle, if it needs to use the
register file at the same time as an instruction already issued

 Stall a conflicting instructions when it tries to enter the MEM or

WB stage.

131
 How to Prevent Multiple WB?
 Assuming that the pipeline does all hazard detection in ID, there
are three checks that must be performed before an instruction can
issue:

 Check for structural hazard: Wait until the required FU is not

busy and make sure the register write port is available when it
will be needed

132
 How to Prevent Multiple WB?
 Assuming that the pipeline does all hazard detection in ID, there
are three checks that must be performed before an instruction can
issue:

 Check for RAW data hazard: Wait until the source register are
not listed as pending destinations in a pipeline register that will
not be available when this instruction needs the result

133
 How to Prevent Multiple WB?
 Assuming that the pipeline does all hazard detection in ID, there
are three checks that must be performed before an instruction can
issue:

 Check for WAW hazards: Check if any previous instructions

has the same register destination as present one. If so, stall
until that instruction enters the MEM stage

134
 MIPS R4000 Pipeline
• Eight stage pipeline
• Memory access is decomposed to from extra pipeline
stage
• Deeper pipeline are also known as superpipelining

135
Figure C.41 The eight-stage pipeline structure of the R4000 uses pipelined instruction
and data caches. The pipe stages are labeled and their detailed function is described in the
text. The vertical dashed lines represent the stage boundaries as well as the location of
pipeline latches. The instruction is actually available at the end of IS, but the tag check is
done in RF, while the registers are fetched. Thus, we show the instruction memory as
operating through RF. The TC stage is needed for data memory access, since we cannot
write the data into the register until we know whether the cache access was a hit or not.

Copyright © 2011, Elsevier Inc. All rights Reserved.

136
 MIPS R4000 Pipeline
 Function of each pipe stage
 IF: First half of instruction fetch; PC selection actually happens
here, together with initiation of instruction each access

 IS: Second half of instruction fetch, complete cache access

 RF: Instruction decode and register fetch, hazard checking, and

instruction cache hit detection

 EX: Execution, which includes effective address calculation,

ALU operation, and branch-target computation and condition
evaluation

137
 MIPS R4000 Pipeline
 Function of each pipe stage
 EX: Execution, which includes effective address calculation,
ALU operation, and branch-target computation and condition
evaluation
 DF: Data fetch, first half of data access
 DS: Second half of data fetch, completion of data cache access
 TC: Tag check, to determine whether the data cache access hit
 WB: Write-back for loads and register-register operations

 Longer-latency pipeline increases both LD and BR delays

138
Figure C.42 The structure of the R4000 integer pipeline leads to a 2-cycle load
delay. A 2-cycle delay is possible because the data value is available at the end of DS
and can be bypassed. If the tag check in TC indicates a miss, the pipeline is backed up
a cycle, when the correct data are available.

Copyright © 2011, Elsevier Inc. All rights Reserved.

139
Instruction 1 2 3 4 5 6 7 8 9

LD R1, …… IF IS RF EX DF DS TC WB

DADD R2, R1, … IF IS RF Stall Stall EX DF DS

SUB R3, R1, ….. IF IS Stall Stall RF EX DF

OR R4, R1, …… IF Stall Stall IS RF EX

A load instruction followed by an immediate use results in a 2-

cycle stall.

140
Figure C.44 The basic branch delay is 3 cycles, since the condition evaluation
is performed during EX.

Copyright © 2011, Elsevier Inc. All rights Reserved.

141
Instruction 1 2 3 4 5 6 7 8 9

Branch Instruction IF IS RF EX DF DS TC WB

Delay Slot IF IS RF EX DF DS TC WB

Stall Stall Stall Stall Stall Stall Stall Stall

Stall Stall Stall Stall Stall Stall Stall

Branch Target IF IS RF EX DF

Basic branch delay is 3 cycle., since branch condition is

computed during EX. The MIPS architecture has a single cycle
delayed branch. The R4000 uses a predict-not-taken strategy for
the remaining 2 cycles of branch delay. Taken branches have a
1-cycle delay slot followed by 2 idle cycle
142
Instruction 1 2 3 4 5 6 7 8 9

Branch Instruction IF IS RF EX DF DS TC WB

Delay Slot IF IS RF EX DF DS TC WB

Branch Instruction + IF IS RF EX DF DS TC
2
Branch Instruction + IF IS RF EX DF DS
3

Untaken branches have a simply 1-cycle delay branches.

143
 MIPS R4000 Pipeline
 Deeper pipeline increases
 Pipeline stall
 Number of levels of forwarding for ALU operations

 In R4000 pipeline, there are four possible sources for an ALU

bypass: EX/DF, DF/DS, DS/TC, and RC/WB

144
 The Floating-Point Pipeline
 R4000 FP unit consists of three FU
 FP adder
 FP divider
 FP multiplier
 Adder is used in the final step of multiply or divide

 DP, FP operations can take from 2 cycle (for negative) up to 112

cycles (for square root)

 FP, FU can be thought of having 8 stages

 Stages are combined in different order to execute various FP operations

145
Stage FU Description

A FP adder Mantissa ADD stage

D FP divider Divide pipeline stages

E FP multiplier Exception test stage
M FP multiplier First stage of multiplier
N FP multiplier Second stage of multiplier
R FP adder Rounding stage
S FP adder Operand shift stage

U Unpack FP number

Eight stages used in the R4000 FP pipeline

146
FP Instructions Latency Initiation Pipe Stages
Interval
Add, subtract 4 3 U, S + A, A+R, R+S

Multiply 8 4 U, E+M, M, M, M, N, N+A, R

Divide 36 35 U, A, R, D28, D+A, D+R, D+A, D+R,A+R
Square root 112 111 U, E, (A+R)108, A, R
Negative 2 1 U, S
Absolute value 2 1 U, S
FP compare 3 2 U, A, R

The latencies and initiation intervals for the FP operations both

depends on the FP unit stages that a given operation must use

147
Operati Issue 0 1 2 3 4 5 6 7 8 9 10 11
on / Stall
Multiply Issue U E+M M M M N N+A R

Add Issue U S+A A+R R+S

Issue U S+A A+R R+S

Issue U S+A A+R R+S

Stall U S+A A+R R+S

Stall U S+A A+R R+S

Issue U S+A A+R R+S

Issue U S+A A+R R+S

Issue U S+A A+R R+S

A FP multiply issued at clock 0 is followed by a single FP add

between clocks 1 and 7
148
Operati Issue 0 1 2 3 4 5 6 7 8 9 10 11
on / Stall
Add Issue U S+A A+R R+S

Multiply Issue U E+M M M M N N+A R

Issue U E+M M M M N N+A R

A multiply issued after a add can always proceed without

stalling, since the shorter instruction clears the shared stages
before longer instruction reaches them

149