CS-622 : Advanced Analysis of

Algorithms

Solving Recurrence

Week #3-4
Saima Gul
 BACKGROUND

08/03/23 06:54 2
Logarithms

 The following notations will be used:

lg n  log 2 n (binary logarithm)

ln n  log e n (natural logarithm)
lg k n  lg n  (exponentiation)
k

lg lg n  lglg n  (composition)

08/03/23 06:54 3
Logarithms

 For real numbers a  0, b  0, c  0, n  0

a logb n  n logb a
b logb a  a logb b  a
log b (ac)  log b a  log b c
log c a
log b a 
log c b

08/03/23 06:54 4
Summations
n
n(n  1)

k 1
k
2
n 1
n
x 1

k 0
k
x 
x 1

08/03/23 06:54 5
 RECURRENCES

08/03/23 06:54 6
Recurrence

 A recurrence is an equation or inequality that describes a

function in terms of its value on smaller inputs.

 When an algorithm contains a recursive call to itself (e.g.

a divide and conquer algorithm), its running time can
often be described as a recurrence.

(1) if n  1
T ( n)  
2T (n / 2)  (n) otherwise
 We will see a number of methods to solve recurrences.

08/03/23 06:54 7
Solution of a Recurrence

 Solving a recurrence means, finding a recursion-free

expression for the function.
 For example, consider the recurrence:
T (n)  T (n  1)  1 where T (0)  3
 The function is described in terms of itself. However, the
same function could have been given as:
T (0)  3
T (1)  T (0)  1  3  1  4 T ( n)  n  3
T (2)  T (1)  1  4  1  5 T ( n )  ( n )
 Sometimes, we don’t even need an exact solution to find
asymptotic bounds on the solutions of recurrences

08/03/23 06:54 8
Technicalities
 While solving recurrences, we will usually neglect some
details.

 We usually ignore the fact that the arguments of the

running time functions should be integers.

 We will also ignore the time actually required for small

inputs, and assume that the algorithms require a
constant time to finish on small inputs.

 In fact we will assume that it takes constant time to

execute any algorithm with a constant input size.
08/03/23 06:54 9
Technicalities

 Floors and ceilings [Floors and ceilings can

easily be removed and don’t affect the
solution to the recurrence. They are better left
to a discrete math course.]
 Exact vs. asymptotic functions
 Boundary conditions In algorithm analysis,
we usually express both the recurrence and
its solution using asymptotic notation

08/03/23 06:54 10
Why we can assume constant time on small
inputs
 When functions are polynomially bounded, the initial
conditions (the value on small inputs) do not make a
difference for the solution in asymptotic notations.
T (n)  T (n  1)  1

T (0)  3 T (0)  5
T (1)  T (0)  1  3  1  4 T (1)  T (0)  1  5  1  6
T (2)  T (1)  1  4  1  5 T (2)  T (1)  1  6  1  7

T ( n)  n  3 T ( n)  n  5
T ( n )  ( n ) T ( n )  ( n )

08/03/23 06:54 11
Does matter when exponentially bounded!!!

 The initial conditions can make a difference, when the

function is not polynomially bounded.

T (n)  T (n / 2) 
2
 For example:
T (1)  2 T (1)  3 T (1)  1
T (2)  T (1)   2 2 T (2)  T (1)   32
2 2

?
T (4)  T (2)   2 T (4)  T (2)   3
2 4 2 4

T (n)  (1)
T (8)  T (4)   28 T (8)  T (4)   38
2 2

T ( n )  ( 2 n ) T (n)  (3n )

08/03/23 06:54 12
Paying attention to the technicalities

 Therefore, the integer arguments and the initial

conditions do not actually make a difference when we try
to find asymptotic solutions to the recurrences.

 However, it is important to know when these

technicalities do make a difference on the result.

 In general, go ahead and solve the recurrence. Then,

check if it makes a difference or not.

08/03/23 06:54 13
 SOLVING RECURRENCES

08/03/23 06:54 14
Method 1: The Substitution Method

 Two step approach:

1. Guess the solution
2. Verify the solution

 It is a very powerful method for showing upper and

lower bounds (O and Ω)

 Tight bounds can be shown by:

f (n)  O( g (n)) and f(n)  Ω( g (n))  f (n)  ( g (n))

08/03/23 06:54 15
Method 1: The Substitution Method

 The difficulty of the substitution method is to guess a

solution.
 In general, there is no way to find a good guess.
 It requires experience and intuition to come up with good
guesses.
 We usually try a loose bound first, and then try to
improve and tighten it gradually.
 The second step (verification step) requires to perform
an induction proof.

08/03/23 06:54 16
The Substitution Method: Exact solution

1. Guess: T (n) = n lg n + n. [Here, we have a recurrence

with an exact function, rather than asymptotic notation,
and the solution is also exact rather than asymptotic. We
will have to check boundary conditions and the base
case.]

2. Induction: Basis: n = 1 ⇒ n lg n + n = 1 = T (n) Inductive

step:

08/03/23 06:54 17
The Substitution Method: Exact solution

08/03/23 06:54 18
Example for the substitution method

 Let us try to find an asymptotic bound on the recurrence:

T (n)  4T (n / 2)  n
3
 Step 1 (Guessing): Claim that T ( n )  O ( n )

3
 Step 2 (Verification): Use induction to prove T (n)  O( n )

 Note that, in order to prove this, we need to show:

c, n0  0 such that n  n0 : T (n)  cn 3

08/03/23 06:54 19
Example for the substitution method

 Induction proof itself has two steps:

i. Induction base: ? ?
3
T (1)  c1  c we can always pick c big
enough to satisfy this
This is usually the case, and therefore we will skip the induction
base step in the inductive proofs of the substitution method
most of the time…

ii. Inductive step:

a. Assume that: T (k )  ck 3 for all k  n induction hypothesis
3
b. And show that: T ( n)  cn

08/03/23 06:54 20
 Example for the substitution method
T (n)  4T (n / 2)  n trick is to try to write it as:
induction “what we want” – “something nonnegative”
hypothesis 3
n c 3
 4c    n T ( n)  n  n
2 2
3 c 3 
n3 T (n)  cn   n  n   cn 3
 4c  n  2    
8  0 if c  2 and n 1

c 3
 n n
2 T (n)  cn3

we want this to look like cn 3

08/03/23 06:54 21
Example for the substitution method

 OK, we have shown that

T (n)  4T (n / 2)  n T ( n)  O ( n 3 )

 However, this may be a loose upper bound.

 Let us try to find a better upper bound.

2
 New claim is: T ( n )  O ( n )

08/03/23 06:54 22
 Example for the substitution method

 We directly jump to the proof of inductive step:

Assume that T (k )  ck 2 for all k  n, and show that T (n)  cn 2
T (n)  4T (n / 2)  n
induction
hypothesis 2 This is a usual mistake.
n
 4c    n Be careful, while doing this inductive proof,
2
we are trying to prove:
2
n T (n)  cn 2
 4c  n
4 not
 cn 2  n
T ( n)  O ( n 2 )
 O(n 2 )
08/03/23 06:54 23
Example for the substitution method

2
 So, we have derived: T (n)  cn  n

 But, we cannot put it into the form:

2
T (n)  cn
  " something  0"
answer
we want

 Note that, T (n)  O(n 2 ) also if T (n)  c1n 2  c2 n

 Idea is to subtract a lower order term…

08/03/23 06:54 24
Example for the substitution method

Assume that T (k )  c1k 2  c2 k for all k  n

and show that T (n)  c1n 2  c2 n
T (n)  4T (n / 2)  n
  n 2 n 
 4 c1    c2   n
 2 2 
 
 c1n 2  2c2 n  n
 c1n 2  c2 n  (c2 n  n)
      
answer we want  0 when c2 1

 c1n 2  c2 n

08/03/23 06:54 25
Showing lower bounds using the substitution
method
 Consider again: T (n)  4T (n / 2)  n

2
 Step1 (Guess and claim): T (n)  (n )

2
 Step2 (prove by induction): assume T ( k )  ck for k  n
T (n)  4T (n / 2)  n
2
n
 4c    n
2 T (n)  cn 2
2
 cn  n
answer something 0
we want

08/03/23 06:54 26
Showing tight bounds using the substitution
method
 Note that for
T (n)  4T (n / 2)  n
 We have shown that:
T (n)  O(n 2 ) and T (n)  (n 2 )
 Therefore:
T ( n )  ( n 2 )

 To find tight bounds for recurrences using the

substitution method, this is the way to follow.

08/03/23 06:54 27
Method 3: The iteration method

 This method for solving recurrences based on the idea to

iterate the recurrence a few times to come up with a
summation that describes the recurrence.

 It requires to know the rules and intuition for series.

 The mathematics used can get messy and hard.

 It is often used to generate good guesses for the

substitution method.

08/03/23 06:54 28
 Example for the iteration method
iteration level

i  1 T (n)  4T (n / 2)  n T (n / 2)  4T (n / 4)  n / 2
i  2 T (n)  n  4( 4T (n / 4)  n / 2)
 n  2n  16T (n / 4) T (n / 4)  4T (n / 8)  n / 4

i  3 T (n)  n  2n  16(4T (n / 8)  n / 4)
 n  2n  4n  64T (n / 8)
grows as grows as
grows as

2 i 1 4i 2i
Note that when i  lg n, T (n / 2i )  T (n / 2lg n )  T (n / n)  T (1)
Hence the recursion will bottom out when i  lg n
08/03/23 06:54 29
Example for the iteration method
So when i  lg n, we will have
T (n)  n  2n  4n  ...  2 lg n 1 n  4 lg n T (1)
a lg b  b lg a
lg n 1
T (n)  n  2 k  n lg 4T (1) t
x t 1
1
k 0 
k 0
k
x 
x 1
 2lg n  1  2
T (n)  n   n T (1)
 2 1 

T (n)  n(n  1)  n 2 c
Book keeping can be easier if we
T ( n )  ( n 2 ) use “recursion trees” to visualize
the iterations.

08/03/23 06:54 30
 Using Recursion Trees total cost
of the level
recursion level T (n)  4T (n / 2)  n
l  0: T (n) : n n

l  1: T (n / 2) : n / 2 T (n / 2) : n / 2 T (n / 2) : n / 2 T (n / 2) : n / 2 2n

l  2: T (n / 4) : n / 4 T (n / 4) : n / 4 T (n / 4) : n / 4 T (n / 4) : n / 4 4n

grows as 2l

l  lg n  1 : 2lg n 1 n

l  lg n : T (n / n) :T (1) T (n / n) :T (1) n 2T (1)

+
number of leaves  4l  4lg n  n 2

08/03/23 06:54 31
Method 4: The Master Method

 The master method provides an easy way to solve

recurrences that are in the following form:
T (n)  aT (n / b)  f (n)
where a  1, b  1, f (n) asymptotically positive
 It requires to remember 3 conditions.

 When a condition is satisfied, you can directly write down

the solution to the recurrence.

 The method is based on the “Master Theorem”

08/03/23 06:54 32
The Master Theorem

 Given T (n)  aT (n / b)  f (n)

such that a  1, b  1, f (n) asymptotically positive
Case 1
if f (n)  O(n lg b a 
) for some   0  T (n)  (n lgb a )
Case 2
if f (n)  (n lgb a )  T (n)  (n lgb a lg n)
Case 3
lg b a  
if f (n)  (n ) for some   0,
and if af (n / b)  cf (n)  T (n)  ( f (n))
for some c  1, and for large n
08/03/23 06:54 33
Example for using the master method

T (n)  9T (n / 3)  n
a  9  1, b  3  1, f (n)  n (asymptotically positive)
- We need n lgb a , let' s calculate it :
n lgb a  n lg3 9  n 2
- We need to compare n lgb a with f (n)
f (n)  n  O(n lgb a  )  O(n lg3 9 )  O (n 2 ) for any 0    1

T (n)  (n lgb a )  (n 2 )

08/03/23 06:54 34
Example for using the master method
T (n)  2T (n / 2)  (n) (merge sort)
a  2  1, b  2  1, f (n)  (n) (asymptotically positive)
- We need n lgb a , let' s calculate it :
n lg 2 2  n
- We need to compare n lgb a with f (n)
f (n)  (n)  ( n lgb a )
- Case 2 applies, therefore
T (n)  (n lgb a lg n)  (n lg n)

08/03/23 06:54 35
Example for using the master method
T (n)  4T (n / 2)  n 3
a  4, b  2, f (n)  n 3
- We need n lgb a , let' s calculate it : n lg 2 4  n 2
- We need to compare n lgb a with f (n)
f (n)  n 3  (n lgb a  )  (n 2 )
- Case 3 may apply, we need to check :
?
af (n / b)  cf (n)
?
3
4(n / 2)  cn 3 T ( n)  ( f (n))  (n 3 )
n 3 / 2  cn 3 , for 1 / 2  c  1

08/03/23 06:54 36