Week 04: Recurrences CS478

Recurrences

Dr. Zahid Halim

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

What are Recurrence Relations

• Recurrence relations describe functions over the natural
numbers.
• Recurrences can take many forms. For example, a recursive
algorithm might divide subproblems into unequal sizes, such as a
2/3-to-1/3 split. If the divide and combine steps take linear
time, such an algorithm would give rise to the recurrence

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

• Subproblems are not necessarily constrained to being a

constant fraction of the original problem size.

• For example, a recursive version of linear search would

create just one subproblem containing only one element fewer
than the original problem.

• Each recursive call would take constant time plus the time for
the recursive calls it makes, yielding the recurrence

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Three methods for solving recurrences

• In the substitution method, we guess a bound and then use
mathematical induction to prove our guess correct.

• The recursion-tree method converts the recurrence into a tree

whose nodes represent the costs incurred at various levels of the recursion.
We use techniques for bounding summations to solve the recurrence.

• The master method provides bounds for recurrences of the form

• where a>1 b>1, and f(n) is a given function. Such recurrences arise frequently.
• Creates a subproblems, each of which is 1/b the size of the original problem, and in
which the divide and combine steps together take f(n) time

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Technicalities in recurrences
• In practice, we neglect certain technical details when we state
and solve recurrences. For example, if we call MERGE-SORT
on n elements when n is odd, we end up with subproblems of
size

• When we state and solve recurrences, we often omit floors,

ceilings, and boundary conditions.

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Recursive FIND_MAXIMUM_SUBARRAY
• We make the simplifying assumption that the original problem
size is a power of 2, so that all subproblem sizes are integers.

• We denote by T(n) the running time of FIND-MAXIMUM-

SUBARRAY on a subarray of n elements.

• For starters, line 1 takes constant time. The base case, when
n=1, is easy: line 2 takes constant time, and so on

• The recursive case occurs when n>1. Lines 1 and 3 take

constant time. Each of the subproblems solved in lines 4 and 5
is on a subarray of n=2 elements
Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi
Week 04: Recurrences CS478

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

• We spend T(n/2) time solving each of them. Because we have to solve two
subproblems—for the left subarray and for the right subarray

• The contribution to the running time from lines 4 and 5 comes to 2T(n/2).

• The call to FIND-MAX-CROSSING-SUBARRAY in line 6 takes O(n) time.

• Lines 7–11 take only O(1) time.

This recurrence has the solution T(n) = O (n lg n)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi
Week 04: Recurrences CS478

Motivation

• The running times of many algorithms (especially recursive

algorithms) are easily expressed using recurrence relations:

• Example: (Merge Sort)

• Problem: How do we compare functions expressed using recurrence

relations?
• We need to solve them, that is, derive closed expressions of the functions
expressed by recurrences using O-, Ω-, and Θ-notation.

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Introduction
• A recurrence is an equation or inequality that describes a
function in terms of its value on smaller inputs

T ( n)  (1)
2T ( n / 2)  ( n )
If n = 1
If n > 1
Θ(nlgn)

• Three methods for solving recurrences

– Substitution method
– Recursion-tree method
– Master method
• Notes – some technical details are neglected
– Assumption of integer arguments to functions: T(n)
– Boundary condition for small n  T(n) is constant for small n
– Floors, ceilings

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Substitution Method

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

The Substitution Method

The three steps of the substitution method:

1. Make a good guess

2. Verify the guess, assuming that it can be verified for some

base case n = n0

3. Choose an n0 for which the guess works and such that Step 2
for any n > n0 does not depend on the claim for some n' < n0

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Basic concept
• Useful when it is easy to guess the form of the answer

• Can be used to establish either upper or lower bound on a

recurrence

• Example:
T (n)  2T ( n / 2)  n

T(n) = O(n lg n) ?

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478
Determine an upper bound on (Cont.)
T (n)  2T ( n / 2)  n
– Induction : holds for n ?
T ( n / 2)  c n / 2 lg n / 2)
T (n)  2T ( n / 2)  n
 2(c n / 2 lg n / 2))  n
 cn lg n / 2  n
 cn(lg n  lg 2)  n
 cn lg n  cn  n
 cn lg n (holds as long as c 1)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Determine an upper bound

T (n)  2T ( n / 2)  n on
• Guess T(n) = O(n lg n)  prove T(n)  cn lg n for some c
– By mathematical induction
– Inductive base: prove the inequality holds for some small n
• n = 1?  T(1)  c * 1 * log 1 = 0  but T(1) = 1
• start from T(2) = 4 or T(3) = 5  choose any c  2
– OK because asymptotic notation requires us to prove T(n)  cn lg n for n  n0

• Trick: extend boundary conditions to make the inductive

assumption work for small n
– Induction assumption: assume the bound holds for
n / 2
T ( n / 2)  c n / 2 lg n / 2) T(2)=2T(1) + 2 = 4
cn lg n = c * 2 * lg 2 = 2c

T(3)=2T(1) + 3 = 5
cn Technology,
Ghulam Ishaq Khan Institute of Engineering Sciences and lg n = c * 3Topi
* lg 3
Week 04: Recurrences CS478

Making a good guess

• Need experience and creativity
• Use recursion trees to generate good guesses
• If a recurrence is similar to one you are familiar, then guessing
a similar solution is reasonable
– T(n) = O(n lg n)  Why?
– The n)  2T ( term
T (additional n / 2(17)
 17cannot
)  n substantially affect the solution to the
recurrence (when n is large)
• Prove loose upper and lower bounds and then reduce the
range of uncertainty

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Recursion-Tree Method

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Basic concept
• Recursion tree
– Each node represents the cost of a single subproblem somewhere in the set of
recursive function invocations
– Sum the costs within each level of the tree to obtain a set of per-level costs
– Sum all the per-level costs to determine the total cost of all levels of the recursion
• Useful when the recurrence describes the running time of a divide-and-
conquer algorithm
• Useful for generating a good guess, which is then verified by the
substitution method
– Sloppiness are allowed
• Example:
– Ignore the floors and write out the implicit coefficient c > 0 (sloppiness)
2
T (n)  3T ( n / 4)  (n )

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Recurrence tree for T(n)=3T(n/4) + cn2

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Recurrence tree for T(n)=3T(n/4) + cn2

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

T(n)=3T(n/4) + cn2
• The subproblem size for a node at depth i is n/4i
– When the subproblem size is 1  n/4i = 1i=log4n
– The tree has log4n+1 levels (0, 1, 2,.., log4n)
• The cost at each level of the tree (0, 1, 2,.., log4n-1)
– Number of nodes at depth i is 3i
– Each node at depth i has a cost of c(n/4i)2
– The total cost over all nodes at depth i is 3i c(n/4i)2=(3/16)icn2
• The cost at depth log4nlog n log 4 3
– Number of nodes is
3 4
 n
– Each contributing cost T(1)
– The total cost n log 4 3T (1)  (n log 4 3 )

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

T(n)=3T(n/4) + cn2 (Cont.)

3 2 3 2 2 3 log 4 n 1 2
2
T (n)  cn  cn  ( ) cn  ...  ( ) cn  (n log 4 3 )
16 16 16
log 4 n 1
3 i 2
  ( ) cn  (n log 4 3 )
i 0 16

3 i 2
  ( ) cn  (n log 4 3 )
i  0 16

1 log 4 3 16 2
 2
cn  (n )  cn  (n log 4 3 )
3 13
1
16
 O(n 2 )

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

A recursion tree for T(n)=T(n/3) + T(2n/3) + cn

2
( )x n  1
3
n  (3 / 2) x
x  log (3 / 2) n
Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi
Week 04: Recurrences CS478

The Master Method

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Basic concept
• Cookbook for solving
T (n)  aT (n / b)  f (n)
– Describe the running time of an algorithm that divides a problem of size
n into a subproblems, each of size n/b
– a  1 and b > 1 (constant), f(n): asymptotically positive
– Require memorization of three cases
– Floor and ceiling functions are omitted

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

How To Solve Recurrences

• Ad hoc method:
– expand several times
– guess the pattern
– can verify with proof by induction
• Master theorem
– general formula that works if recurrence has the form T(n) =
aT(n/b) + f(n)
• a is number of subproblems
• n/b is size of each subproblem
• f(n) is cost of non-recursive part

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Master Theorem
Consider a recurrence of the form
T(n) = a T(n/b) + f(n)
with a>=1, b>1, and f(n) eventually positive.
a)If f(n) = O(nlogb(a)-), then T(n)=(nlogb(a)).
b)If f(n) = (nlogb(a) ), then T(n)=(nlogb(a) log(n)).
c) If f(n) = (nlogb(a)+) and f(n) is regular, then
T(n)=(f(n))
[f(n) regular iff eventually af(n/b)<= cf(n) for some
constant c<1]

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Excuse me, what did it say???

Essentially, the Master theorem compares the function f(n)
with the function g(n)=nlogb(a).
Roughly, the theorem says:
a)If f(n) << g(n) then T(n)=(g(n)).
b)If f(n)  g(n) then T(n)=(g(n)log(n)).
c)If f(n) >> g(n) then T(n)=(f(n)).
Now go back and memorize the theorem!

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Solving the Recurrence Relation

Applying the Master Theorem to
T(n) = a T(n/b) + f(n)
with a=7, b=2, and f(n)=(n2).

Since f(n) = O(nlogb(a)-) = O(nlog2(7)-),

case a) applies and we get
T(n)= (nlogb(a)) = (nlog2(7)) = O(n2.81).

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Using the master method

• Example 1: T(n)=9T(n/3) + n (Case 1)
– a=9, b=3, f(n) = n
– n log b a  n log3 9  (n 2 ), f (n)  O(n log3 91 )
– T(n) = (n2)
• Example 2: T(n) = T(2n/3) + 1 (Case 2)
– a=1, b=3/2, f(n)=1
–
log b a log 3 / 2 1
n 
– T(n) = (lg n) n  1, f (n)  1  (1)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Using the master method (Cont.)

• Example 3: T(n) = 3T(n/4) + n lg n (Case 3)
– a=3, b=4, f(n) = n lg n
log 4 3
– n b  n 4  O(n
log a log 3 0 . 793 f ( n )   ( n ) ( 0.2)
)
– For sufficiently large n, af(n/b) = 3(n/4)lg(n/4)  (3/4)n lg n=cf(n) for c=3/4
– T(n) = (n lg n)
• Example 4: T(n) = 2T(n/2) + n lg n
– a=2, b=2, f(n) = n lg n
– n logb a  n log 2 2  n
–
– Falls into the
log b agap between case 2 and case 3
f ( n) / n  n lg n / n  lg n (Asymptotically less than n for any  )

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Motivation: Asymptotic Behavior of Recursive Algorithms

• When analyzing algorithms, recall that we only care about the

asymptotic behavior

• Recursive algorithms are no different

• Rather than solving exactly the recurrence relation associated

with the cost of an algorithm, it is sufficient to give an
asymptotic characterization

• The main tool for doing this is the master theorem

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Master Theorem
• Let T(n) be a monotonically increasing function that
satisfies
T(n) = a T(n/b) + f(n)
T(1) = c
where a  1, b  2, c>0. If f(n) is (nd) where d  0
then

if a < bd
T(n) = If a = bd
if a > bd

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Master Theorem: Pitfalls

• You cannot use the Master Theorem if
– T(n) is not monotone, e.g. T(n) = sin(x)
– f(n) is not a polynomial, e.g., T(n)=2T(n/2)+2n
– b cannot be expressed as a constant, e.g.

• Note that the Master Theorem does not solve the recurrence
equation

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Week 04: Recurrences CS478

Example: merge sort

• 1. Divide: Trivial.

• 2. Conquer: Recursively sort 2 subarrays.

• 3. Combine: Linear-time merge.

T(n) = 2 T(n/2) + O(n)

# of sub-problems size of sub-problems work dividing
& combining
Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi
Week 04: Recurrences CS478

Example: Binary search

• 1. Divide: Trivial.

• 2. Conquer: Recursively search 1 subarray.

• 3. Combine: Linear-time merge.

T(n) = 1T(n/2) + O(1)

# of sub-problems size of sub-problems work dividing
& combining
Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi