0% found this document useful (0 votes)
76 views37 pages

4 Recurrence and Master Theorem

The document discusses recurrence relations and methods for solving recurrences such as the substitution method, recursion tree method, and master theorem. It provides examples of using these methods to solve recurrences such as T(n)=2T(n/2)+n and T(n)=3T(n/4)+nlg(n), and discusses exceptions to applying the master theorem.

Uploaded by

Harshit Roy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
76 views37 pages

4 Recurrence and Master Theorem

The document discusses recurrence relations and methods for solving recurrences such as the substitution method, recursion tree method, and master theorem. It provides examples of using these methods to solve recurrences such as T(n)=2T(n/2)+n and T(n)=3T(n/4)+nlg(n), and discusses exceptions to applying the master theorem.

Uploaded by

Harshit Roy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 37

Design and Analysis of Algorithms

RecurrenceAndMasterTheorem
Lecture 5
References:
Introduction to Algorithms, Thomas H. Cormen, 2ed edition,

Instructor: Prashant mishra


BIAS,Bhimtal
Recurrence
 A recurrence is an equation or inequality that
describe a function in term of its value on smaller
inputs.
 For example the worst case running time T(n) of the
merge sort procedure can be described by the
recurrence

T(n)= {Θ(1) if n=1

{ aT(n/b) +D(n) +C(n) if n>1


Whose solution was claimed to be T(n)=Θ(n lg n)
Recurrences

• MERGE-SORT
– Contains details:
• T(n) = Θ(1) if n=1
T(n/2)+ T(n/2)+ Θ(n) if n>1
• Ignore details, T(n) = 2T(n/2)+ Θ(n).
– T(n) = Θ(1) if n=1
2T(n/2)+ Θ(n) if n>1

3
Methods for solving recurrence
Substitution method
Recursion method
The master method
The Substitution Method
• Two steps:
1. Guess the form of the solution.
• By experience, and creativity.
• By some heuristics.
– If a recurrence is similar to one you have seen before.
» T(n)=2T(n/2+17)+n, similar to T(n)=2T(n/2)+n, , guess O(nlg n).
– Prove loose upper and lower bounds on the recurrence and then reduce the
range of uncertainty.
» For T(n)=2T(n/2)+n, prove lower bound T(n)= Ω(n), and prove upper
bound T(n)= O(n2), then guess the tight bound is T(n)=O(nlg n).
• By recursion tree.
2. Use mathematical induction to find the constants and show
that the solution works.

5
Solve T(n)=2T(n/2)+n
• Guess the solution: T(n)=O(nlg n),
– i.e., T(n)≤ cnlg n for some c.
• Prove the solution by induction:
– Suppose this bound holds for n/2, i.e.,
• T(n/2)≤ cn/2 lg (n/2).
– T(n) ≤ 2(cn/2 lg (n/2))+n
• ≤ cn lg (n/2))+n
• = cn lg n - cn lg 2 +n
• = cn lg n - cn +n
• ≤ cn lg n (as long as c≥1)

Question: Is the above proof complete? Why?


6
Boundary (base) Condition
• In fact, T(n) =1 if n=1, i.e., T(1)=1.
• However, cnlg n =c×1×lg 1 = 0, which is odd with
T(1)=1.
• Take advantage of asymptotic notation: it is
required T(n)≤ cnlg n hold for n≥ n0 where n0 is a
constant of our choosing.
• Select n0 =2, thus, n=2 and n=3 as our induction
bases. It turns out any c ≥ 2 suffices for base cases
of n=2 and n=3 to hold.

7
Subtleties
• Guess is correct, but induction proof not work.
• Problem is that inductive assumption not strong
enough.
• Solution: revise the guess by subtracting a lower-order
term.
• Example: T(n)=T(n/2)+T(n/2)+1.
– Guess T(n)=O(n), i.e., T(n) ≤ cn for some c.
– However, T(n) ≤c n/2+c n/2+1 =cn+1, which does not
imply T(n) ≤ cn for any c.
– Attempting T(n)=O(n2) will work, but overkill.
– New guess T(n) ≤ cn – b will work as long as b ≥ 1.
– (Prove it in an exact way).
8
Avoiding Pitfall
• It is easy to guess T(n)=O(n) (i.e., T(n) ≤
cn) for T(n)=2T(n/2)+n.
• And wrongly prove:
– T(n) ≤ 2(c n/2)+n
• ≤ cn+n
• =O(n).  wrongly !!!!
• Problem is that it does not prove the exact
form of T(n) ≤ cn.
9
Find bound, ceiling, floor, lower term–
domain transformation
• Find the bound: T(n)=2T(n/2)+n (O(nlogn)
• How about T(n)=2T(n/2)+n ?
• How about T(n)=2T(n/2)+n ?
– T(n)≤2T(n/2+1)+n
– Domain transformation
• Set S(n)=T(n+a) and assume S(n)=2S(n/2)+n (so S(n)=O(nlogn))
• S(n)=2S(n/2)+n T(n+a)=2T(n/2+a)+n
• T(n)≤2T(n/2+1)+n T(n+a) ≤ 2T((n+a)/2+1)+n
• Thus, set n/2+a=(n+a)/2+1, get a=2.
• so T(n)=S(n-2)=O((n-2)log(n-2)) = O(nlogn).
• How about T(n)=2T(n/2+19)+n ?
– Set S(n)=T(n+a) and get a=38.
• As a result, ceiling, floor, and lower terms will not affect.
– Moreover, the master theorem also provides proof for this.

10
Changing Variables
• Suppose T(n)=2T(√n)+lg n.
• Rename m=lg n. so T(2m)=2T(2m/2)+m.
• Domain transformation:
– S(m)=T(2m), so S(m)=2S(m/2)+m.
– Which is similar to T(n)=2T(n/2)+n.
– So the solution is S(m)=O(m lg m).
– Changing back to T(n) from S(m), the solution is T(n)
=T(2m)=S(m)=O(m lg m)=O(lg n lg lg n).

11
The Recursion-tree Method
• Idea:
– Each node represents the cost of a single subproblem.
– Sum up the costs with each level to get level cost.
– Sum up all the level costs to get total cost.
• Particularly suitable for divide-and-conquer
recurrence.
• Best used to generate a good guess, tolerating
“sloppiness”.
• If trying carefully to draw the recursion-tree and
compute cost, then used as direct proof.
12
Recursion Tree for T(n)=3T(n/4)+Θ(n2)
T(n) cn2 cn2
T(n/4) T(n/4) T(n/4) c(n/4)2 c(n/4)2 c(n/4)2

T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16)


(a) (b) (c)
cn2 cn2

c(n/4)2 c(n/4)2 (3/16)cn2


c(n/4)2

log 4n (3/16)2cn2
c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2

T(1)T(1)T(1) T(1)T(1)T(1) Θ(nlog 43)


3log4n= nlog 43 Total O(n2)
(d) 13
Solution to T(n)=3T(n/4)+Θ(n2)
• The height is log 4n,
• #leaf nodes = 3log 4n= nlog 43. Leaf node cost: T(1).
• Total cost T(n)=cn2+(3/16) cn2+(3/16)2 cn2+
⋅⋅⋅ +(3/16)log 4n-1 cn2+ Θ(nlog 43)
=(1+3/16+(3/16)2+ ⋅⋅⋅ +(3/16)log 4n-1) cn2 + Θ(nlog 43)
<(1+3/16+(3/16)2+ ⋅⋅⋅ +(3/16)m+ ⋅⋅⋅) cn2 + Θ(nlog 43)
=(1/(1-3/16)) cn2 + Θ(nlog 43)
=16/13cn2 + Θ(nlog 43)
=O(n2).

14
Prove the above Guess
• T(n)=3T(n/4)+Θ(n2) =O(n2).
• Show T(n) ≤dn2 for some d.
• T(n) ≤3(d (n/4)2) +cn2
≤3(d (n/4)2) +cn2
=3/16(dn2) +cn2
≤ dn2, as long as d≥(16/13)c.

15
One more example
• T(n)=T(n/3)+ T(2n/3)+O(n).
• Construct its recursive tree (Figure 4.2,
page 71).
• T(n)=O(cnlg3/2n) = O(nlg n).
• Prove T(n) ≤ dnlg n.

16
Recursion Tree of T(n)=T(n/3)+ T(2n/3)+O(n)

17
Master Method/Theorem
• Theorem 4.1 (page 73)
– for T(n) = aT(n/b)+f(n), n/b may be n/b or n/b.
– where a ≥ 1, b>1 are positive integers, f(n) be a non-
negative function.
1. If f(n)=O(nlogba-ε) for some ε>0, then T(n)= Θ(nlogba).
2. If f(n)= Θ(nlogba), then T(n)= Θ(nlogba lg n).
3. If f(n)=Ω(nlogba+ε) for some ε>0, and if af(n/b) ≤cf(n)
for some c<1 and all sufficiently large n, then T(n)=
Θ(f(n)).

18
Implications of Master Theorem
• Comparison between f(n) and nlogba (<,=,>)
• Must be asymptotically smaller (or larger) by a
polynomial, i.e., nε for some ε>0.
• In case 3, the “regularity” must be satisfied, i.e.,
af(n/b) ≤cf(n) for some c<1 .
• There are gaps
– between 1 and 2: f(n) is smaller than nlogba, but not
polynomially smaller.
– between 2 and 3: f(n) is larger than nlogba, but not
polynomially larger.
– in case 3, if the “regularity” fails to hold.
19
Application of Master Theorem
• T(n) = 9T(n/3)+n;
– a=9,b=3, f(n) =n
– nlogba = nlog39 = Θ (n2)
– f(n)=O(nlog39-ε) for ε=1
– By case 1, T(n) =Θ (n2).
• T(n) = T(2n/3)+1
– a=1,b=3/2, f(n) =1
– nlogba = nlog3/21 = Θ (n0) = Θ (1)
– By case 2, T(n)= Θ(lg n).
20
Application of Master Theorem
• T(n) = 3T(n/4)+nlg n;
– a=3,b=4, f(n) =nlg n
– nlogba = nlog43 = Θ (n0.793)
– f(n)= Ω(nlog43+ε) for ε≈0.2
– Moreover, for large n, the “regularity” holds for
c=3/4.
• af(n/b) =3(n/4)lg (n/4) ≤ (3/4)nlg n = cf(n)
– By case 3, T(n) =Θ (f(n))=Θ (nlg n).
21
Exception to Master Theorem
• T(n) = 2T(n/2)+nlg n;
– a=2,b=2, f(n) =nlg n
– nlogba = nlog22 = Θ (n)
– f(n) is asymptotically larger than nlogba , but not
polynomially larger because
– f(n)/nlogba = lg n, which is asymptotically less
than nε for any ε>0.
– Therefore,this is a gap between 2 and 3.
22
Where Are the Gaps
f(n), case 3, at least polynomially larger
nε Gap between case 3 and 2
c1
nlogba f(n), case 2: within constant distances
c2
nε Gap between case 1 and 2
f(n), case 1, at least polynomially smaller

Note: 1. for case 3, the regularity also must hold.


2. if f(n) is lg n smaller, then fall in gap in 1 and 2
3. if f(n) is lg n larger, then fall in gap in 3 and 2
4. if f(n)=Θ(nlogbalgkn), then T(n)=Θ(nlogbalgk+1n). (as exercise)
23
Proof of Master Theorem
• The proof for the exact powers, n=bk for k≥1.
• Lemma 4.2
– for T(n) = Θ(1) if n=1
– aT(n/b)+f(n) if n=bk for k≥1
– where a ≥ 1, b>1, f(n) be a nonnegative function,
– Then logbn-1
– T(n) = Θ(nlogba)+ ajf(n/bj)
j=0

• Proof:
– By iterating the recurrence
– By recursion tree (See figure 4.3)

24
Recursion tree for T(n)=aT(n/b)+f(n)

25
Proof of Master Theorem (cont.)
• Lemma 4.3:
– Let a ≥ 1, b>1, f(n) be a nonnegative function defined on
exact power of b, then
logbn-1
– g(n)=  ajf(n/bj) can be bounded for exact power of b as:
j=0
1. If f(n)=O(nlogba-ε) for some ε>0, then g(n)= O(nlogba).
2. If f(n)= Θ(nlogba), then g(n)= Θ(nlogba lg n).
3. If f(n)= Ω(nlogba+ε) for some ε>0 and if af(n/b) ≤cf(n) for
some c<1 and all sufficiently large n ≥b, then g(n)= Θ(f(n)).

26
Proof of Lemma 4.3
• For case 1: f(n)=O(nlogba-ε) implies f(n/bj)=O((n /bj)logba-ε), so
logbn-1 logbn-1
• g(n)=  ajf(n/bj) =O(  a j(n /bj)logba-ε )
j=0 j=0
logbn-1 logbn-1
• = O(nlogba-ε  a j/(blogba-ε)j ) = O(nlogba-ε
 a j/(aj(b-ε)j))
j=0 j=0
logbn-1
• = O(nlogba-ε  (bε)j ) = O(nlogba-ε (((bε ) logbn-1)/(bε-1) )
j=0
• = O(nlogba-ε (((blogbn)ε -1)/(bε-1)))=O(nlogba n-ε (nε -1)/(bε-1))
• = O(nlogba )
27
Proof of Lemma 4.3(cont.)
• For case 2: f(n)= Θ(nlogba) implies f(n/bj)= Θ((n /bj)logba), so

logbn-1 logbn-1
• g(n)=  ajf(n/bj) = Θ(  aj(n /bj)logba)
j=0 j=0
logbn-1 logbn-1
• = Θ(nlogba  aj/(blogba)j ) = Θ(nlogba  1)
j=0 j=0

• = Θ(nlogba logbn) = Θ(nlogbalg n)

28
Proof of Lemma 4.3(cont.)
• For case 3:
– Since g(n) contains f(n), g(n) = Ω(f(n))
– Since af(n/b) ≤cf(n), ajf(n/bj) ≤cjf(n) , why???

logbn-1 j logbn-1 j ∞
– g(n)=  a f(n/bj) ≤  c f(n) ≤ f(n)  cj
j=0 j=0 j=0

– =f(n)(1/(1-c)) =O(f(n))
– Thus, g(n)=Θ(f(n))
29
Proof of Master Theorem (cont.)
• Lemma 4.4:
– for T(n) = Θ(1) if n=1
– aT(n/b)+f(n) if n=bk for k≥1
– where a ≥ 1, b>1, f(n) be a nonnegative function,
1. If f(n)=O(nlogba-ε) for some ε>0, then T(n)= Θ(nlogba).
2. If f(n)= Θ(nlogba), then T(n)= Θ(nlogba lg n).
3. If f(n)=Ω(nlogba+ε) for some ε>0, and if af(n/b) ≤cf(n)
for some c<1 and all sufficiently large n, then T(n)=
Θ(f(n)).

30
Proof of Lemma 4.4 (cont.)
• Combine Lemma 4.2 and 4.3,
– For case 1:
• T(n)= Θ(nlogba)+O(nlogba)=Θ(nlogba).
– For case 2:
• T(n)= Θ(nlogba)+Θ(nlogba lg n)=Θ(nlogba lg n).
– For case 3:
• T(n)= Θ(nlogba)+Θ(f(n))=Θ(f(n)) because f(n)=
Ω(nlogba+ε).

31
Floors and Ceilings
• T(n)=aT(n/b)+f(n) and T(n)=aT(n/b)+f(n)
• Want to prove both equal to T(n)=aT(n/b)+f(n)
• Two results:
– Master theorem applied to all integers n.
– Floors and ceilings do not change the result.
• (Note: we proved this by domain transformation too).
• Since n/b≤n/b, and n/b≥ n/b, upper bound for
floors and lower bound for ceiling is held.
• So prove upper bound for ceilings (similar for lower
bound for floors).

32
Upper bound of proof for T(n)=aT(n/b)+f(n)
• consider sequence n, n/b,  n/b/b,   n/b /b/b,

• Let us define nj as follows:
• nj = n if j=0
• = nj-1/b if j>0
• The sequence will be n0, n1, …, nlogbn
• Draw recursion tree:

33
Recursion tree of T(n)=aT(n/b)+f(n)

34
The proof of upper bound for ceiling
logbn -1
– T(n) = Θ(nlogba)+  ajf(nj)
j=0

– Thus similar to Lemma 4.3 and 4.4, the upper


bound is proven.

35
The simple format of master theorem

• T(n)=aT(n/b)+cnk, with a, b, c, k are


positive constants, and a≥1 and b≥2,
O(nlogba), if a>bk.
• T(n) = O(nklogn), if a=bk.
O(nk), if a<bk.

36
Summary
Recurrences and their bounds
– Substitution
– Recursion tree
– Master theorem.
– Proof of subtleties
– Recurrences that Master theorem does not
apply to.

37

You might also like