Solving

Recurrences

Lecture 13
CS2110 – Summer 2009
 In the past…

Complexity:
 Big-O definition
 Non-recursive programs

2
 Today
 Complexity of recursive programs
 Introduce recurrences
 Methods to solve recurrences
 Substitution
 Recursion tree
 Master

3
 Analysis of Merge-Sort
public static Comparable[] mergeSort(Comparable[] A, int low, int high) {
if (low < high) { //at least 2 elements? cost = c
int mid = (low + high)/2; cost = d
Comparable[] A1 = mergeSort(A, low, mid); cost = T(n/2) + e
Comparable[] A2 = mergeSort(A, mid+1, high); cost = T(n/2) + f
return merge(A1,A2); cost = gn + h
}
Comparable[] temp = new Comparable[1]; cost = i (base case)
temp[0] = A[low];
return temp;
}

Recurrence:
T(n) = c + d + e + f + 2T(n/2) + gn + h  recurrence
T(1) = i  base case

 Recurrences are important when designing divide & conquer

algorithms
 How do we solve this recurrence? 4
 Solving Recurrences

 Unfortunately, solving recurrences is like solving differential

equations
 No general technique works for all recurrences

 Some techniques:
 Substitution:
 Make a guess, then prove the guess correct by induction
 Can sometimes change variables to get a simpler recurrence
 Recursion-tree:
 Build a recursion tree and use it to determine solution
 Can use the Master Method
 A “cookbook” scheme that handles many common recurrences

5
 Substitution Method
1. Guess the form of the solution
2. Use induction to show that the solution works.

6
 Ex1: Analysis of Merge-Sort
Recurrence:
T(n) = c + d + e + f + 2T(n/2) + gn + h
T(1) = i

First, simplify the recurrence:

T(n) ≤ 2T(n/2) + kn
T(1) = i

7
 Analysis of Merge-Sort

 Proof: strong induction on n

 Recurrence for MergeSort  Show that
T(2)  2i + 2k
 T(n) ≤ 2T(n/2) + kn T(n)  2T(n/2) + kn
 T(2) = 2i + 2k imply
T(n)  pn log n (where p ≥ i + k)

 Solution is T(n) = O(n log n)  Basis

T(2) = 2(i + k)  2p = p 2 log 2

 Induction step
T(n)  2T(n/2) + kn
 2(pn/2 log n/2) + kn (IH)
= pn (log n – 1) + kn
= pn log n + (k – p)n
 pn log n 8
 Ex2: The Fibonacci Function
 Mathematical definition:
fib(0) = 0
fib(1) = 1
fib(n) = fib(n  1) + fib(n  2), n ≥ 2

 Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, …

static int fib(int n) { Fibonacci (Leonardo

if (n == 0) return 0; Pisano) 11701240?
else if (n == 1) return 1;
else return fib(n-1) + fib(n-2); Statue in Pisa, Italy
} Giovanni Paganucci
1863

9
 Recursive Execution
static int fib(int n) {
if (n == 0) return 0;
else if (n == 1) return 1;
else return fib(n-1) + fib(n-2);
}

Execution of fib(4): fib(4)

fib(3) fib(2)

fib(2) fib(1) fib(1) fib(0)

fib(1) fib(0)
10
 The Fibonacci Recurrence
static int fib(int n) {
if (n == 0) return 0;
else if (n == 1) return 1;
else return fib(n-1) + fib(n-2);
}

T(0) = c
T(1) = c
T(n) = T(n – 1) + T(n – 2) + c

 Solution is exponential in n
 But not quite O(2n)...
11
 The Golden Ratio

 = (a+b)/b = b/a

2 =  + 1

1+5
=
2
ratio of sum of sides
(a+b) to longer side (b) = 1.618...
a =
ratio of longer side (b) to
b
shorter side (a)
12
 Fibonacci Recurrence is O(n)

• want to show T(n)  cn

• have 2 =  + 1
• multiplying by cn, cn+2 = cn+1 + cn

• Basis:
• T(0) = c = c0
• T(1) = c  c1

• Induction step:
• T(n+2) = T(n+1) + T(n)  cn+1 + cn = cn+2
13
Detour:
Can We Do Better?
if (n <= 1) return n;
int parent = 0;
int current = 1;
for (int i = 2; i ≤ n; i++) {
int next = current + parent;
parent = current;
current = next;
}
return (current);

 Number of times loop is executed? Less than n

 Number of basic steps per loop? Constant
 Complexity of iterative algorithm = O(n)
 Much, much, much, much, much, better than O(n)!

14
Detour:
...But We Can Do Even Better!
 Let fn denote the nth Fibonacci number
 f0 = 0
 f1 = 1
 fn+2 = fn+1 + fn, n  0

0 1 fn fn+1 0 1 n f0 fn
 Note that = , thus =
1 1 fn+1 fn+2 1 1 f1 fn+1

 Can compute the nth power of a matrix by repeated squaring in

O(log n) time

 Gives complexity O(log n)

 Just a little cleverness got us from exponential to logarithmic!

15
Detour:
Recall Problem-Size Examples
 Suppose we have a computing device that can
execute 1000 operations per second; how large a
problem can we solve?

1 second 1 minute 1 hour 1 century

12
log n 21000 260,000 23,600,000 2 3.2 10
n 1000 60,000 3,600,000 3.2  1012
n log n 140 4893 200,000 8.7  1010
n2 31 244 1897 1,776,446
3n2 18 144 1096 1,025,631
n3 10 39 153 1,318
2n 9 15 21 41
16
 Hints on Guessing
 Use recursion trees (next)
 Use similar, known recurrences
 E.g., T(n) = 2T(n/2 + 17) + n => O(n log n)
 Loose bounds for first guess, then adjust gradually
 E.g., T(n) = 2T(n/2) + n
 First lower bound: O(n)
 First upper bound: O(n2)

17
 Hints on Proving
 Prove the exact form
 E.g., T(n) = 2T(n/2) + n, guess T(n) = O(n)
 T(n) ≤ 2(cn/2) + n
≤ cn + n
= O(n)  WRONG
 Change variables
 E.g., T(n) = 2T(n.5) + log n
 Let m = log n
 => T(2m) = 2T(2m/2) + m
 Let S(m) = T(2m)
 => S(m) = 2S(m/2) + m
 => S(m) = O(m log m)
 => T(n) = T(2m) = S(m) = O(m log m) = O(log n * log log n)
18
 Recursion Tree Method
 Node = cost of a single subproblem
 Sum nodes in same level to get per-level costs
 Sum all per-level costs to get total costs

19
 Ex: Merge-Sort
 T(n) = 2T(n/2) + cn
cn cn

c(n/2) c(n/2) cn

c(n/4) c(n/4) c(n/4) c(n/4) cn

c c c c c c c c c c cn

Total: (log n)*cn

 Master Method
 To solve recurrences of the form
T(n) = aT(n/b) + f(n),
with constants a ≥ 1, b > 1, compare f(n) with nlogba

 if f(n) grows more rapidly, and if a*f(n/b) ≤ c*f(n) for some c ≤ 1, n

sufficiently large
 Solution is T(n) = O(f(n))

 if nlogba grows more rapidly

 Solution is T(n) = O(nlogba)

 if both grow at same rate

 Solution is T(n) = O(f(n) log n)

21
 Recurrence Examples

 T(n) = T(n – 1) + 1  T(n) = O(n) Linear Search

 T(n) = T(n – 1) + n  T(n) = O(n2) QuickSort worst-case

 T(n) = T(n/2) + 1  T(n) = O(log n) Binary Search

 T(n) = T(n/2) + n  T(n) = O(n)

 T(n) = 2 T(n/2) + n  T(n) = O(n log n) MergeSort

 T(n) = 2 T(n – 1)  T(n) = O(2n)

 T(n) = 4 T(n/2) + n  T(n) = O(n2)

22
 Moral: Complexity Matters!

 But you are acquiring the best tools to deal with it!

23