NEW SOUTH WALES

Algorithms:
COMP3121/3821/9101/9801

Aleks Ignjatovic

School of Computer Science and Engineering

University of New South Wales

TOPIC 1: RECURRENCES

COMP3121/3821/9101/9801 1 / 20
Asymptotic notation

Big Oh notation: f (n) = O(g(n)) is an abbreviation for:

There exist positive constants c and n0 such that

0 f (n) c g(n) for all n n0 .

In this case we say that g(n) is an asymptotic upper bound for

f (n).

f (n) = O(g(n)) means that f (n) does not grow substantially faster
than g(n) because a multiple of g(n) eventually dominates f (n).

Clearly, multiplying constants c of interest will be larger than 1,

thus enlarging g(n).

COMP3121/3821/9101/9801 2 / 20
Asymptotic notation
Omega notation: f (n) = (g(n)) is an abbreviation for:

There exists positive constants c and n0 such that

0 c g(n) f (n) for all n n0 .

In this case we say that g(n) is an asymptotic lower bound for

f (n).

f (n) = (g(n)) essentially says that f (n) grows at least as fast as

g(n), because f (n) eventually dominates a multiple of g(n).

Clearly, multiplying constants c of interest will be smaller than 1,

thus shrinking g(n) by a constant factor.
Theta notation: f (n) = (g(n)) iff and only if
f (n) = O(g(n)) and f (n) = (g(n)); thus, f (n) and g(n) have the
same asymptotic growth rate.
COMP3121/3821/9101/9801 3 / 20
Recurrences

Recurrences are important to us because they arise in estimations

of time complexity of divide-and-conquer algorithms.

Merge-Sort(A, p, r) *sorting A[p..r]*

1 if p < r
2 then q b p+r
2 c
3 Merge-Sort(A, p, q)
4 Merge-Sort(A, q + 1, r)
5 Merge(A, p, q, r)

Since Merge(A, p, q, r) runs in linear time, the runtime T (n) of

Merge-Sort(A, p, r) satisfies
n
T (n) = 2T + cn
2
COMP3121/3821/9101/9801 4 / 20
Recurrences

Let a 1 be an integer and b > 1 a real number;

Assume that a divide-and-conquer algorithm:
reduces a problem of size n to a many problems of smaller size n/b;
the overhead cost of splitting up/combining the solutions for size
n/b into a solution for size n is if f (n),
then the time complexity of such algorithm satisfies
n
T (n) = a T + f (n)
b
Note: we should be writing
l n m
T (n) = a T + f (n)
b
but it can be shown that assuming that n is a power of b is OK,
and that the estimate produced is still valid for all n.

COMP3121/3821/9101/9801 5 / 20
 n
T (n) = a T + f (n)
b

size of instance = n

a many size of instances = n/b
instances
size of instances = n/b2
. depth of
. recursion:
. log !
. .
. . . .
. .
. . .
. .
. .

size of instances = 1

COMP3121/3821/9101/9801 6 / 20
Some recurrences can be solved explicitly, but this tends to be
tricky.

Fortunately, to estimate efficiency of an algorithm we do not need

the exact solution of a recurrence

We only need to find:

1 the growth rate of the solution i.e., its asymptotic behaviour;
2 the sizes of the constants involved (more about that later)

This is what the Master Theorem provides (when it is

applicable).

COMP3121/3821/9101/9801 7 / 20
Master Theorem:
Let:
a 1 and b > 1 be integers;
f (n) > 0 be a monotonically increasing function;
T (n) be the solution of the recurrence T (n) = a T (n/b) + f (n);
Then:
1 If f (n) = O(nlogb a ) for some > 0, then T (n) = (nlogb a );
2 If f (n) = (nlogb a ), then T (n) = (nlogb a log2 n);
3 If f (n) = (nlogb a+ ) for some > 0, and for some c < 1,

a f (n/b) c f (n)

then T (n) = (f (n));

4 If none of these conditions hold, the Master Theorem is NOT
applicable (in the form presented).

COMP3121/3821/9101/9801 8 / 20
Master Theorem - a remark
Note that for any b > 1,
logb n = logb 2 log2 n;
Since b > 1 is constant (does not depend on n), we have for
c = logb 2 > 0
logb n = c log2 n;
1
log2 n = logb n;
c
Thus,
logb n = (log2 n)
and also
log2 n = (logb n).
So whenever we have f = (g(n) log n) we do not have to specify
what base the log is - all bases produce equivalent asymptotic
estimates.
COMP3121/3821/9101/9801 9 / 20
Master Theorem - Examples
Let T (n) = 4 T (n/2) + n;

then nlogb a = nlog2 4 = n2 ;

thus f (n) = n = O(n2 ) for any < 1.

Condition of case 1 is satisfied; thus, T (n) = (n2 ).

Let T (n) = 2T (n/2) + c n;

then nlogb a = nlog2 2 = n1 = n;

thus f (n) = c n = (n) = (nlog2 2 ).

Thus, condition of case 2 is satisfied; and so,

T (n) = (nlog2 2 log n) = (n log n).
COMP3121/3821/9101/9801 10 / 20
Master Theorem - Examples
Let T (n) = 3 T (n/4) + n;
then nlogb a = nlog4 3 < n0.8 ;
thus f (n) = n = (n0.8+ ) for any < 0.2.
Also, af (n/b) = 3f (n/4) = 3/4 n < c n for c = .8 < 1.

Thus, Case 3 applies, and T (n) = (f (n)) = (n).

Let T (n) = 2 T (n/2) + n log2 n;

then nlogb a = nlog2 2 = n1 = n.
Thus, f (n) = n log2 n = (n).
However, f (n) = n log2 n 6= (n1+ ), no matter how small > 0.
This is because for every > 0, and every c > 0, no matter how
small, log2 n < c n for all sufficiently large n.
Homework: Prove this.
Hint: Use de LHopitals Rule to show that log n/n 0.

Thus, in this case the Master Theorem does not apply!

COMP3121/3821/9101/9801 11 / 20
Master Theorem - Proof:
Since  
n
T (n) = a T + f (n) (1)
b
implies (by applying it to n/b in place of n)
     
n n n
T = aT +f (2)
b b2 b

and (by applying (1) to n/b2 in place of n)

     
n n n
T = aT +f (3)
b2 b3 b2

and so on . . ., we get
(1)
z }|
  {     
n n n
T (n) = a T +f (n) = a a T 2
+f + f (n)
b b b
| {z } | {z }
(2) (2)
      
2 n n

2 n n n


=a T 2
+a f b + f (n) = a aT +f + af b + f (n)
b b3 b2
| {z } | {z }
(3) (3)
 
3 n  
+a2 f bn2 + a f n


=a T 3 b + f (n) = . . .
b
| {z }

COMP3121/3821/9101/9801 12 / 20
Master Theorem Proof:
Continuing in this way logb n 1 many times we get ...
n n n
T (n) = a3 T 3 +a2 f 2 + a f + f (n) =
| {zb } b b
= ...
 n   n 
= ablogb nc T blog nc + ablogb nc1 f blog nc1 + . . .
b b   n  b n
b

3 n 2
+ a f 3 + a f 2 + af + f (n)
b b b
blogb nc1
logb n
 n  X n
a T + ai f
blogb n i=0
bi

We now use alogb n = nlogb a :

blogb nc1 n
X
T (n) nlogb a T (1) + ai f (4)
i=0
bi
Note that so far we did not use any assumptions on f (n) . . .
COMP3121/3821/9101/9801 13 / 20
Master Theorem Proof:

Case 1: f (m) = O(mlogb a )

blogb nc1 n blogb nc1  n logb a
X X
ai f = ai O
i=0
bi i=0
bi

blogb nc1 blogb nc1 
ai
X  n logb a X 
i logb a
= O a = O n
i=0
bi i=0
(bi )logb a

blogb nc1  blogb nc1 
logb a
X a i
logb a
X a i
= O n = O n
blogb a
i=0 i=0
blogb a b

blogb nc1  i
 blogb nc1
log a
X a b log a
X i
= O n b = O n b (b )
i=0
a i=0
! m
blogb nc
logb a (b ) 1 X q m+1 1
=O n ; we are using qm =
b 1

i=0
q1

COMP3121/3821/9101/9801 14 / 20
Master Theorem Proof:
Case 1 - continued:
blogb nc1
!
blogb nc
X n
logb a (b ) 1
ai f i = O n
i=0
b b 1
 
bblogb nc 1
logb a
= O n
b 1
n 1
 
= O nlogb a
b 1
 log a
n b nlogb a

=O
b 1
 
= O nlogb a
blogb nc1 n
X
Since we had: T (n) nlogb a T (1) + ai f we get:
i=0
bi
 
T (n) nlogb a T (1) + O nlogb a
 
= nlogb a

COMP3121/3821/9101/9801 15 / 20
Master Theorem Proof:
Case 2: f (m) = (mlogb a )
blogb nc1 n blogb nc1  n logb a
X X
ai f = ai
i=0
bi i=0
bi

blogb nc1  n logb a
X i
= a
i=0
bi

blogb nc1  i

X a
= nlogb a
i=0
(bi )logb a

blogb nc1 
X a i
= nlogb a
i=0
blogb a

blogb nc1
X
= nlogb a 1
i=0
 
= nlogb a blogb nc

COMP3121/3821/9101/9801 16 / 20
Master Theorem Proof:
Case 2 (continued):

Thus,

blogb nc1 n    

X
ai f = nlogb a logb n = nlogb a log2 n
i=0
bi

because logb n = log2 n logb 2 = (log2 n). Since we had (1):

blogb nc1 n
X
T (n) nlogb a T (1) + ai f
i=0
bi

we get:
 
T (n) nlogb a T (1) + nlogb a log2 n
 
= nlogb a log2 n

COMP3121/3821/9101/9801 17 / 20
Master Theorem Proof:
Case 3: f (m) = (mlogb a+ ) and a f (n/b) c f (n) for some 0 < c < 1.
c
We get by substitution: f (n/b) f (n)
a
c
f (n/b2 ) f (n/b)
a
c
f (n/b3 ) f (n/b2 )
a
...
c
f (n/bi ) f (n/bi1 )
a
By chaining these inequalities we get
c c c c2
f (n/b2 ) f (n/b) f (n) = 2 f (n)
a | {z } a |a {z } a

c c c2 c3
f (n/b3 ) f (n/b2 ) 2 f (n) = 3 f (n)
a | {z } a |a {z } a

...
c c ci1 ci
f (n/bi ) f (n/bi1 ) i1 f (n) = i f (n)
a | {z } a |a {z } a

COMP3121/3821/9101/9801 18 / 20
Master Theorem Proof:
Case 3 (continued):
ci
We got f (n/bi ) f (n)
ai
Thus,
blogb nc1 blogb nc1
X n X ci X f (n)
ai f ai f (n) < f (n) ci =
i=0
bi i=0
ai i=0
1 c

Since we had (1):

blogb nc1 n
X
T (n) nlogb a T (1) + ai f
i=0
bi
and since f (n) = (nlogb a+ ) we get:
T (n) < nlogb a T (1) + O (f (n)) = O (f (n))
but we also have

T (n) = aT (n/b) + f (n) > f (n)

thus,
T (n) = (f (n))
COMP3121/3821/9101/9801 19 / 20
Master Theorem Proof: Homework

Exercise 1: Show that condition

f (n) = (nlogb a+ )

follows from the condition

a f (n/b) c f (n) for some 0 < c < 1.

Exercise 2: Estimate T (n) for

T (n) = 2 T (n/2) + n log n

Note: we have seen that the Master Theorem does NOT apply, but the technique
used in its proof still works! Just unwind the recurrence and sum up the logarithmic
overheads.

COMP3121/3821/9101/9801 20 / 20