Lecture

Recurrence Relations and how to solve them!

9/16/2024 1
Last time….
• Sorting: InsertionSort and MergeSort
• What does it mean to work and be fast?
• Worst-Case Analysis
• Big-Oh Notation
• Analyzing running time of recursive algorithms
• By writing out a tree and adding up all the work done.

9/16/2024 2
Today
• Recurrence Relations!
• How do we calculate the runtime a recursive algorithm?
• The Master Method
• A useful theorem so we don’t have to answer this
question from scratch each time.
• The Substitution Method
• A different way to solve recurrence relations, more
general than the Master Method.

9/16/2024 3
 Running time of MergeSort
• Let T(n) be the running time of MergeSort on a
length n array.

• We know that T(n) = O(nlog(n)).

• We also know that T(n) satisfies:

𝑛 MERGESORT(A):
𝑇 𝑛 =2⋅𝑇 + 𝑂(𝑛) n = length(A)
2 if n ≤ 1:
return A
L = MERGESORT(A[:n/2])
R = MERGESORT(A[n/2:])
9/16/2024 return MERGE(L,R) 4
 Running time of MergeSort
• Let T(n) be the running time of MergeSort on a
length n array.

• We know that T(n) = O(nlog(n)).

• We also know that T(n) satisfies:

𝑛 MERGESORT(A):
𝑇 𝑛 ≤2⋅𝑇 + 11 ⋅ 𝑛 n = length(A)
2 if n ≤ 1:
return A
Last time we showed that the time to run MERGE L = MERGESORT(A[:n/2])
on a problem of size n is O(n). For concreteness, R = MERGESORT(A[n/2:])
let’s say that it’s at most 11n operations. return MERGE(L,R)
9/16/2024 5
 Recurrence Relations
𝑛
• 𝑇 𝑛 =2⋅𝑇 + 11 ⋅ 𝑛 is a recurrence relation.
2
• It gives us a formula for T(n) in terms of T(less than n)

• The challenge:
Given a recurrence relation for T(n), find a
closed form expression for T(n).
Note that
𝑛
• For example, T(n) = O(nlog(n)) 𝑇 𝑛 ≤2⋅𝑇
2
+ 11 ⋅ 𝑛
(with a ≤) is also a recurrence
relation! Does it matter for a
9/16/2024 conclusion like T(n) = O(n log(n))? 6
Technicalities I
Base Cases
• Formally, we should always have base cases with
recurrence relations.
𝑛
• 𝑇 𝑛 =2⋅𝑇 + 11 ⋅ 𝑛 with 𝑇 1 = 1
2
is not the same function as
𝑛
• 𝑇 𝑛 =2⋅𝑇 + 11 ⋅ 𝑛 with 𝑇 1 = 1000000000
2
• However, no matter what T is, T(1) is O(1), so sometimes
we’ll just omit it. Why is T(1) = O(1)?

9/16/2024 7
• You played around with these examples (when n is
a power of 2):
𝑛
1. 𝑇1 𝑛 = 𝑇1 2
+ 𝑛, 𝑇 1 =1
𝑛
2. 𝑇 𝑛 = 2 ⋅ 𝑇 2
+ 𝑛, 𝑇 1 =1
𝑛
3. 𝑇2 𝑛 = 4 ⋅ 𝑇2 2
+ 𝑛, 𝑇 1 =1

9/16/2024 8
One approach for all of these
Size n
• The “tree” approach
from last time. n/2 n/2

• Add up all the work n/4 n/4 n/4 n/4

done at all the sub-
problems.
…
n/2t n/2t n/2t n/2t n/2t n/2t

9/16/2024
(Size 1) 9
 Contribution
𝑛 at this layer:
• 𝑇1 𝑛 = 𝑇1 + 𝑛, 𝑇1 1 = 1.
2 n
Size n
• Adding up over all layers:
n/2 n/2
log(𝑛)
𝑛
෍ 𝑖 = 2𝑛 − 1 n/4 n/4
2
𝑖=0
…
• So 𝑇1 𝑛 = 𝑂 𝑛 . n/2t n/2t

…
1
9/16/2024 (Size 1) 10
 𝑛
• 𝑇2 𝑛 = 4𝑇2 + 𝑛, 𝑇2 1 = 1.
2 Contribution
• Adding up over all layers: at this layer:

log(𝑛) log 𝑛 Size n n

𝑛
෍ 4𝑖 ⋅ 𝑖 = 𝑛 ෍ 2𝑖
2 4x n/2 2n
𝑖=0 𝑖=0
= 𝑛(2𝑛 − 1)
16x n/4 4n
• So 𝑇2 𝑛 = 𝑂 𝑛2
…
4t x n/2t 2tn
…
n2 x n2
9/16/2024 (Size 1) 11
More examples
T(n) = time to solve a problem of size n.
• Needlessly recursive integer multiplication
• T(n) = 4 T(n/2) + O(n)
• T(n) = O( n2 )

• Karatsuba integer multiplication

• T(n) = 3 T(n/2) + O(n)
• T(n) = O( 𝑛log2 3 ≈ n1.6 )

• MergeSort
• T(n) = 2T(n/2) + O(n)
• T(n) = O( nlog(n) )
9/16/2024 What’s the pattern?!?!?!?! 12
The master theorem
A useful
• A formula for many formula it is.
recurrence relations. Know why it works
• You’ll come up with an you should.
example in next class when it
won’t work.

• Proof: “Generalized” tree

method.

Jedi master Yoda

9/16/2024 13
 We can also take n/b to
𝑛 𝑛
mean either 𝑏 or 𝑏 and
The master theorem the theorem is still true.

• Suppose that 𝑎 ≥ 1, 𝑏 > 1, and 𝑑 are constants

(independent of n).
𝑛
• Suppose 𝑇 𝑛 = 𝑎 ⋅ 𝑇 + 𝑂 𝑛𝑑 . Then
𝑏

O 𝑛𝑑 log 𝑛 if 𝑎 = 𝑏 𝑑
𝑇 𝑛 = O 𝑛𝑑 if 𝑎 < 𝑏 𝑑
O 𝑛log𝑏 𝑎
if 𝑎 > 𝑏 𝑑
Three parameters:
a : number of subproblems Many symbols
b : factor by which input size shrinks
d : need to do nd work to create all the
those are….
subproblems
9/16/2024
and combine their solutions. 14
Technicalities II
Integer division
• If n is odd, I can’t break it up into two problems of
size n/2.
𝑛 𝑛
𝑇 𝑛 =𝑇 +𝑇 + 𝑂(𝑛)
2 2
• However one can show that the Master theorem
works fine if you pretend that what you have is:

𝑛
𝑇 𝑛 =2⋅𝑇 + 𝑂(𝑛)
2
• From now on we’ll mostly ignore floors and ceilings
in recurrence relations.
9/16/2024 15
 𝑛
𝑇 𝑛 =𝑎⋅𝑇 + 𝑂 𝑛𝑑 .
Examples 𝑏

O 𝑛𝑑 log 𝑛 if 𝑎 = 𝑏𝑑
(details on board)
𝑇 𝑛 = O 𝑛𝑑 if 𝑎 < 𝑏𝑑
O 𝑛log𝑏 𝑎
if 𝑎 > 𝑏𝑑

✓
• Needlessly recursive integer mult.
a=4
• T(n) = 4 T(n/2) + O(n) b=2 a > bd
• T(n) = O( n2 ) d=1

✓
• Karatsuba integer multiplication
a=3
• T(n) = 3 T(n/2) + O(n) b=2 a > bd
• T(n) = O( nlog_2(3) ≈ n1.6 ) d=1

✓
• MergeSort
a=2
• T(n) = 2T(n/2) + O(n) b=2 a = bd
• T(n) = O( nlog(n) ) d=1

✓
• That other one
a=1
• T(n) = T(n/2) + O(n)
b=2 a < bd
• T(n) = O(n) d=1
9/16/2024 16
Understanding the Master Theorem

• Let 𝑎 ≥ 1, 𝑏 > 1, and 𝑑 be constants.

𝑛
• Suppose 𝑇 𝑛 = 𝑎 ⋅ 𝑇 + 𝑂 𝑛𝑑 . Then
𝑏

O 𝑛𝑑 log 𝑛 if 𝑎 = 𝑏 𝑑
𝑇 𝑛 = O 𝑛𝑑 if 𝑎 < 𝑏 𝑑
O 𝑛log𝑏 𝑎
if 𝑎 > 𝑏 𝑑

• What do these three cases mean?

9/16/2024 18
 The eternal struggle

Branching causes the number The problems lower in

of problems to explode! the tree are smaller!
The most work is at the The most work is at
bottom of the tree! the top of the tree!
9/16/2024 19
Consider our three warm-ups
𝑛
1. 𝑇 𝑛 = 𝑇 2
+𝑛
𝑛
2. 𝑇 𝑛 = 2 ⋅ 𝑇 2
+𝑛
𝑛
3. 𝑇 𝑛 = 4 ⋅ 𝑇 2
+𝑛

9/16/2024 20
First example: tall and skinny tree
𝑛
1. 𝑇 𝑛 = 𝑇 + 𝑛, 𝑎 < 𝑏𝑑 Size n
2

• The amount of work done at the n/2

top (the biggest problem) swamps
the amount of work done anywhere n/4
else.

• T(n) = O( work at top ) = O(n)

n/2t
WINNER

Most work at the

1
9/16/2024 top of the tree! 21
 Third example: bushy tree
WINNER
𝑛
3. 𝑇 𝑛 = 4 ⋅ 𝑇 + 𝑛, 𝑎 > 𝑏𝑑
2
Size n
Most work at
the bottom
n/2 n/2 of the tree!
n/2 n/2

• There are a HUGE number of leaves, and the total work is

dominated by the time to do work at these leaves.

• T(n) = O( work at bottom ) = O( 4depth of tree ) = O(n2)

1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1
11 1 1 1 1 1 1 1 1
1 1
9/16/2024 1 1 1 1 22
1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1
Second example: just right
𝑛
2. 𝑇 𝑛 = 2 ⋅ 𝑇 + 𝑛, 𝑎 = 𝑏𝑑
2 Size n

• The branching just balances n/2

n/2
out the amount of work.
• The same amount of work n/4 n/4 n/4 n/4

is done at every level.

• T(n) = (number of levels) * (work per level)
• = log(n) * O(n) = O(nlog(n))

TIE!
1 1 1 1 1 1 1 1 1 1
9/16/2024 23
What have we learned?
• The “Master Method” makes our lives easier.
• But it’s basically just codifying a calculation we
could do from scratch if we wanted to.

9/16/2024 24
The Substitution Method
• Another way to solve recurrence relations.
• More general than the master method.

9/16/2024 25
The Substitution Method
first example

• Let’s return to:

𝑛
𝑇 𝑛 =2⋅𝑇 + 𝑛, with 𝑇 1 = 1.
2
• (assuming n is a power of 2...)
• The Master Method says 𝑇 𝑛 = 𝑂 𝑛 log 𝑛 .
• We will prove this via the Substitution Method.

9/16/2024 26
 𝑛
𝑇 𝑛 =2⋅𝑇 + 𝑛, with 𝑇 1 = 1.
2

Step 1: Guess the answer You can guess the

𝑛 answer however
• 𝑇 𝑛 =2⋅𝑇 +𝑛 𝑛 you want: meta-
2 Expand 𝑇 2 reasoning, a little
𝑛 𝑛
• 𝑇 𝑛 =2⋅ 2⋅𝑇 + +𝑛 bird told you,
4 2 wishful thinking,
Simplify
𝑛
• 𝑇 𝑛 =4⋅𝑇 + 2𝑛 etc. One useful
4 𝑛 way is to try to
𝑛 𝑛 Expand 𝑇 “unroll” the
4
• 𝑇 𝑛 =4⋅ 2⋅𝑇 + + 2𝑛 recursion, like
8 4
𝑛 we’re doing here.
Simplify
• 𝑇 𝑛 =8⋅𝑇 + 3𝑛
8
•…
𝑛
Guessing the pattern: 𝑇 𝑛 = 2𝑗 ⋅ 𝑇 𝑗 + 𝑗 ⋅ 𝑛
2
Plug in 𝑗 = log 𝑛 , and get
9/16/2024
𝑇 𝑛 = 𝑛 ⋅ 𝑇 1 + log 𝑛 ⋅ 𝑛 = 𝑛 log 𝑛 + 1 27
Why two methods?
• Sometimes the Substitution Method works where
the Master Method does not.

9/16/2024 28
Next Time
• What happens if the sub-problems are different sizes?
• And when might that happen?

9/16/2024 29