A N A LY T I C C O M B I N AT O R I C S

PART ONE

2. Recurrences

http://aofa.cs.princeton.edu
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

2. Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2a.Recur.Values
What is a recurrence?

Def. A recurrence is an equation that recursively defines a sequence.

Familiar example 1: Fibonacci numbers

recurrence

= − + − ≥ = =
MUST specify
for all N with
sequence
initial conditions
0, 1, 1, 2, 3, 5, 8, 13, 21, ...

Q. Simple formula for sequence (function of N )?

3
What is a recurrence?

Recurrences directly model costs in programs.

Familiar example 2: Quicksort (see lecture 1)

recurrence program
public class Quick
= + + ( + ) {
private static int partition(Comparable[] a, int lo, int hi)
{
int i = lo, j = hi+1;
while (true)
≥ = {
while (less(a[++i], a[lo])) if (i == hi) break;
while (less(a[lo], a[--j])) if (j == lo) break;
if (i >= j) break;
sequence exch(a, i, j);
}
exch(a, lo, j);
0, 2, 5, 8 2/3, 12 5/6, 17 2/5,... return j;
}

private static void sort(Comparable[] a, int lo, int hi)

{
if (hi <= lo) return;
int j = partition(a, lo, hi);
sort(a, lo, j-1);
sort(a, j+1, hi);
}
}
4
Common-sense rule for solving any recurrence

Use your computer to compute values. = − + − ≥ = =

public static void F(int N)

{
Use a recursive program? if (N == 0) return 0; ✘
if (N == 1) return 1;
return F(N-1) + F(N-2);
} F(50)

F(49) F(48)

NO, NO, NO: Takes exponential time! F(48) F(47) F(47) F(46)

F(47) F(46) F(46) F(45) F(46) F(45) F(45) F(44)

Top of recursion tree for naïve Fibonacci function

long[] F = new long[51];

F[0] = 0; F[1] = 1;
Instead, save all values in an array.
if (N == 1) return 1; ✓
for (int i = 2; i <= 50; i++)
F[i] = F[i-1] + F[i-2];

5
Common-sense starting point for solving any recurrence

Use your computer to compute initial values.

First step: Download "standard model" from Algorithms, 4th edition booksite.

StdIn Standard Input

StdOut Standard Output

StdDraw Standard Drawings

StdRandom Random Numbers

... (Several other libraries)

http://algs4.cs.princeton.edu
6
Common-sense starting point for solving any recurrence

Use your computer to compute initial values (modern approach).

Ex. 1: Fibonacci = − + − = = Sequence.java
public interface Sequence
Fib.java public class Fib implements Sequence {
{ public double eval(int N);
private final double[] F;
}
public Fib(int maxN)
{ Compute all values % java Fib 15
F = new double[maxN+1]; in the constructor 0.0
F[0] = 0; F[1]= 1; 1.0
for (int N = 2; N <= maxN; N++) 1.0
F[N] = F[N-1] + F[N-2];
2.0
}
3.0
public double eval(int N) 5.0
{ return F[N]; } 8.0
13.0
public static void main(String[] args) 21.0
{ 34.0
int maxN = Integer.parseInt(args[0]); 55.0
Fib F = new Fib(maxN); 89.0
for (int i = 0; i < maxN; i++) 144.0
StdOut.println(F.eval(i)); 233.0
} 377.0
}
7
Common-sense starting point for solving any recurrence

Ex. 2: Quicksort =( + ) +

QuickSeq.java
public class QuickSeq implements Sequence
{
private final double[] c; % java QuickSeq 15
0.000000
public QuickSeq(int maxN) 2.000000
{ 5.000000
c = new double[maxN+1]; 8.666667
c[0] = 0; 12.833333
for (int N = 1; N <= maxN; N++) 17.400000
c[N] = (N+1)*c[N-1]/N + 2; 22.300000
} 27.485714
32.921429
public double eval(int N) 38.579365
{ return c[N]; } 44.437302
50.477056
public static void main(String[] args) 56.683478
{ 63.043745
// Similar to Fib.java. 69.546870
}
}

8
Common-sense starting point for solving any recurrence

Use your computer to plot initial values. QuickSeq.java

public class QuickSeq implements Sequence
{
// Implementation as above.
Values.java
public class Values public static void main(String[] args)
{ {
public static void show(Sequence f, int maxN) int maxN = Integer.parseInt(args[0]);
{ QuickSeq q = new QuickSeq(maxN);
double max = 0; Values.show(q, maxN);
for (int N = 0; N < maxN; N++) }
}
if (f.eval(N)>max) max = f.eval(N);
for (int N = 0; N < maxN; N++)
{
double x = 1.0*N/maxN;
double y = 1.0*f.eval(N)/max;
StdDraw.filledCircle(x, y, .002);
}
StdDraw.show();
}
}

% java QuickSeq 1000

9
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

2. Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2a.Recur.Values
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

2. Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2b.Recur.Telescope
Telescoping a (linear first-order) recurrence
Linear first-order recurrences telescope to a sum.

Example 1.
= + =

Apply equation for n−1 = +( )+

Do it again = +( )+( )+

Continue, leaving a sum = +

( + )
Evaluate sum = Check.
( + ) ( )
= +

Challenge: Need to be able to evaluate the sum.

12
Elementary discrete sums

geometric series =
<

( )
arithmetic series = =
<

+
binomial (upper) =
+
see Knuth volume 1
for many more

binomial theorem =( + )

Harmonic numbers =

+
Vandermonde convolution =

13
 Telescoping a (linear first-order) recurrence (continued)

When coefficients are not 1, multiply/divide by a summation factor.

Example 2.

= + =

Divide by 2n = +

Telescope to a sum = =

=
Check.
= ( ) +
Challenge: How do we find the summation factor?
14
 Telescoping a (linear first-order) recurrence (continued)

Q. What’s the summation factor for = + ... ?

A. Divide by ...
Example 3.

= + + > =
summation factor:
+
... = +

Divide by n+1 = +
+ +
2

Telescope = = +
+

= ( + )( + )

Challenge: Still need to be able to evaluate sums.

15
In-class exercise 1.

Verify the solution for Example 3.

Check initial values

= + + > = = ( + )( + )

= + = = ( )=

= + = = ( )=

= ( )
= + = /
= ( / + / + / )
= /

Proof

+ ( )+ = ( + )( )+

= ( + )( + )

16
In-class exercise 2.

Solve this recurrence:

=( ) + > =

Hard way:

summation factor: ··· =

( )

Easy way: = =
=

WHY?

17
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2b.Recur.Telescope
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2c.Recur.Types
Types of recurrences

linear an = nan 1 1
first
order
nonlinear an = 1/(1 + an 1)

linear an = an 1 + 2an 2

p
second order nonlinear an = an 1 an 2+ an 2

variable
coefficients an = nan 1 + (n 1)an 2 +1

higher order an = f (an 1 , an 2 , . . . , a n t )

full history an = n + an 1 + an 2 ... + a1

divide-and-conquer an = abn/2c + adn/2e + n

20
Nonlinear first-order recurrences

! "
Example. (Newton’s method ) = + [Typical in scientific computing]
−
−

SqrtTwo.java
public class SqrtTwo implements Sequence
{
private final double[] c;
% java SqrtTwo 10
1.0
public SqrtTwo(int maxN)
{
1.5
c = new double[maxN+1]; 1.4166666666666665
c[0] = 1; quadratic convergence:
number of significant 1.4142156862745097
for (int N = 1; N <= maxN; N++)
c[N] = (c[N-1] + 2/c[N-1])/2; digits doubles for 1.4142135623746899
} each iteration 1.414213562373095
public double eval(int N) 1.414213562373095
{ return c[N]; } 1.414213562373095
public static void main(String[] args) 1.414213562373095
{ 1.414213562373095
int maxN = Integer.parseInt(args[0]);
SqrtTwo test = new SqrtTwo(maxN);
for (int i = 0; i < maxN; i++)
StdOut.println(test.eval(i));
}
}
21
Higher-order linear recurrences

[ Stay tuned for systematic solution using generating functions (next lecture) ]

Example 4.
= = =

Postulate that an = xn =
Divide by xn−2 + =
Factor ( )( )=

Form of solution must be = +

Use initial conditions to = = + Note dependence

solve for coefficients on initial conditions
= = +

Solution is c0 = 1 and c1 = −1 =

22
Higher-order linear recurrences

[ Stay tuned for systematic solution using generating functions (next lecture) ]

Example 5. Fibonacci numbers

= + = =

Postulate that an = xn = +
Divide by xn−2 =
+
ˆ) = =
Factor ( )(
ˆ=
Form of solution must be = + ˆ

Use initial conditions to = = + Note dependence

solve for coefficients on initial conditions
= = +ˆ
ˆ
Solution =

23
Higher-order linear recurrences (continued)

Procedure amounts to an algorithm.

Multiple roots? Add nαn terms (see text)

Need to compute roots? Use symbolic math package.

sage: realpoly.<z> = PolynomialRing(CC)

sage: factor(z^2-z-1)
(z - 1.61803398874989) * (z + 0.618033988749895)

Complex roots? Stay tuned for systematic solution using GFs (next lecture)
24
Divide-and-conquer recurrences

Divide and conquer is an effective technique in algorithm design.

Recursive programs map directly to recurrences.

Classic examples:
• Binary search
• Mergesort
• Batcher network
• Karatsuba multiplication
• Strassen matrix multiplication

25
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2c.Recur.Types
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2d.Recur.Mergesort
Warmup: binary search
Everyone’s first divide-and-conquer algorithm

// Precondition: array a[] is sorted.

public static int rank(int key, int[] a)
{
int lo = 0;
int hi = a.length - 1;
while (lo <= hi)
{
// Key is in a[lo..hi] or not present.
int mid = lo + (hi - lo) / 2;
if (key < a[mid]) hi = mid - 1;
else if (key > a[mid]) lo = mid + 1;
else return mid;
}
return -1;
}

Number of compares in the worst case

= / + > =

28
Analysis of binary search (easy case)

= / + > =

Exact solution for N = 2n .

= + > =
a0 +
+1

Telescope to a sum = =

+1

=
Check. = ( / )+

29
Analysis of binary search (general case)

Easy by correspondence with binary numbers

Define BN to be the number of bits in the binary representation of N. Example.

• B1 = 1. 1101011 110101 1

• Removing the rightmost bit of N gives N/2 . 107 53

Therefore = / + > =
N N/2
same recurrence as for binary search

Theorem. BN = lg N +1 +
= + <
Proof. Immediate by definition of . < + = =

N 1 2 3 4 5 6 7 8 9
binary 1 10 11 100 101 110 111 1000 1001
lg N 0 1.0 1.58... 2.0 2.32... 2.58... 2.80... 3 3.16...
lg N 0 1 1 2 2 2 2 3 3
lg N +1 1 2 2 3 3 3 3 4 4
30
Mergesort
Everyone’s second divide-and-conquer algorithm

public class Merge

{
...
private static void
sort(Comparable[] a, Comparable[] aux, int lo, int hi)
{
if (hi <= lo) return;
int mid = lo + (hi - lo) / 2;
sort(a, aux, lo, mid);
sort(a, aux, mid + 1, hi);
merge(a, aux, lo, mid, hi);
}
...
}

For simplicity, assume merge implementation uses N compares

Number of compares for sort: = / + / + > = 0

31
Analysis of mergesort (easy case)

Number of compares for sort: = / + / + > = 0

Already solved for N = 2n

Solution: =

32
Analysis of mergesort (general case)

Number of compares for sort: = / + / + > = 0

Solution: =

Q. For quicksort, the number of compares is ( )

Is the number of compares for mergesort + for some constant α?

A. NO !

33
Coefficient of the linear term for mergesort

public class MergeLinearTerm implements Sequence

{
private final double[] c;

public MergeLinear(int maxN)

{
c = new double[maxN+1]; % java MergeLinearTerm 512
c[0] = 0;
for (int N = 1; N <= maxN; N++)
c[N] = N + c[N/2] + c[N-(N/2)];
for (int N = 1; N <= maxN; N++)
c[N] -= N*Math.log(N)/Math.log(2)) + N;
}

public double eval(int N)

{ return c[N]; }

public static void main(String[] args)

{
int maxN = Integer.parseInt(args[0]);
MergeLinearTerm M = new MergeLinearTerm(maxN);
Values.show(M, maxN);
}
}

34
Analysis of mergesort (general case)

Number of compares for sort: = / + / + > = 0

/ = ( + )/
Same formula for N+1. + = ( + )/ + ( + )/ + +
= / + / + + + 1 0 1 1 1
2 1 1 1 2
3 1 2 2 2
Subtract. + = / + / + 4 2 2 2 3
5 2 3 3 3
6 3 3 3 4
Define DN = CN+1 - CN. = / + 7 3 4 4 4
different 8 4 4 4 5
initial 9 4 5 5 5
Solve as for binary search. = lg + value

/ + = ( + )/
Telescope.
= + ( lg + )
<

Theorem. CN = N −1 + number of bits in binary representation of numbers < N

35
Combinatorial correspondence
SN = number of bits in the binary rep. of all numbers < N

/2 /2 1
1 1 1 1
10 10 10 10
11 11 11 11
100 100 100 100
101 101 101 101
110 110 110 110
111 111 111 111
1000 = 1000 + 1000 + 1000
1001 1001 1001 1001
1010 1010 1010 1010
1011 1011 1011 1011
1100 1100 1100 1100
1101 1101 1101 1101
1110 1110 1110 1110

= /2 + /2 + 1

Same recurrence as mergesort (except for −1): = + 1

36
Number of bits in all numbers < N (alternate view)
842 1
bits are in an subtract off red 0s
N by lgN+1 box column by column
0000
0001
= ( + 1) 2
0010
0
0011
0100 +1
= 2 + +1
0101
0110
0111
1000
1001
= + 1
1010 +1
= 2 +2
1011
1100
1101
1110 Theorem. Number of compares for mergesort is 2 +1
+2

+1 37
Analysis of mergesort (summary)

Number of compares for sort: = / + / + > = 0

Solution: =

+1
Theorem. Number of compares for mergesort is 2 +2

Alternate formulation (Knuth). = lg + ( )

Notation: lg = lg {lg }

( )
{lg }
+
{lg }

=
( ) {lg } {lg }
✓
38
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2d.Recur.Mergesort
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2e.Recur.Master
Divide-and-conquer algorithms

Suppose that an algorithm attacks a problem of size N by

• Dividing into α parts of size about N/β.
• Solving recursively.
• Combining solutions with extra cost Θ(N γ(log N )δ)
only valid when
N is a power of 2

Example 1 (mergesort): α = 2, β = 2, γ = 1, δ = 0 = / +

Example 2 (Batcher network): α = 2, β = 2, γ = 1, δ = 1 = / + lg

Example 3 (Karatsuba multiplication): α = 3, β = 2, γ = 1, δ = 0 = / +

Example 4 (Strassen matrix multiply): α = 7, β = 2, γ = 1, δ = 0 = / +

41
“Master Theorem” for divide-and-conquer algorithms
Suppose that an algorithm attacks a problem of size n by
dividing into α parts of size about n/β with extra cost Θ(nγ(log n)δ)

Theorem. The solution to the recurrence

= / + ( ) + / + ( ) + ... + / + ( ) + ( (log ) )
α terms >
is given by
= ( (log ) ) < log
+
= ( (log ) ) = log
log
= ( ) > log AN INTRODUCTION

ANALYSIS
TO THE

ALGORITHMS
OF

S E C O N D E D I T I O N

< ROBERT SEDGEWICK

Example: α = 3
PHILIPPE FLAJOLET

β=2

β=3

β=4
42
Typical “Master Theorem” applications
Master Theorem
Suppose that an algorithm attacks a problem of size N by
= ( (log ) < log
• Dividing into α parts of size about N/β.
+
= ( (log ) = log
• Solving recursively.
log
• Combining solutions with extra cost Θ(N γ(log N )δ)
= ( ) > log

Asymptotic growth rate

Example 1 (mergesort): α = 2, β = 2, γ = 1, δ = 0 Θ(N log N )

Example 2 (Batcher network): α = 2, β = 2, γ = 1, δ = 1 Θ(N (log N )2 )

Example 3 (Karatsuba multiplication): α = 3, β = 2, γ = 1, δ = 0 Θ(N log2 3 ) = Θ(N 1.585... )

Example 4 (Strassen matrix multiply): α = 7, β = 2, γ = 1, δ = 0 Θ(N log2 7 ) = Θ(N 2.807... )

43
Versions of the “Master Theorem”

Suppose that an algorithm attacks a problem of size N by

• Dividing into α parts of size about N/β.
• Solving recursively.
• Combining solutions with extra cost Θ(N γ(log N )δ)

1. Precise results are available for certain applications AN INTRODUCTION

ANALYSIS
TO THE

in the analysis of algorithms. ALGORITHMS

OF

S E C O N D E D I T I O N

ROBERT SEDGEWICK
PHILIPPE FLAJOLET

2. General results are available for proofs

in the theory of algorithms.

3. A full solution using analytic combinatorics

was derived in 2011 by Szpankowski and Drmota.

see “A Master Theorem for Divide-and-Conquer Recurrences”

by Szpankowski and Drmota (SODA 2011).
44
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

Recurrences
• Computing values
• Telescoping
• Types of recurrences
OF
• Mergesort
• Master Theorem
http://aofa.cs.princeton.edu

2e.Recur.Master
Exercise 2.17

Percentage of three nodes at the bottom level of a 2-3 tree?

Typical 2-3 tree built from random keys

AN INTRODUCTION

ANALYSIS
TO THE

ALGORITHMS
OF

S E C O N D E D I T I O N

ROBERT SEDGEWICK
PHILIPPE FLAJOLET

46
Exercise 2.69

Details of divide-by-three and conquer?

AN INTRODUCTION

ANALYSIS
TO THE

ALGORITHMS
OF

S E C O N D E D I T I O N

ROBERT SEDGEWICK
PHILIPPE FLAJOLET

47
Assignments for next lecture

1. Read pages 41-86 in text.

AN INTRODUCTION

ANALYSIS
TO THE

ALGORITHMS
OF

S E C O N D E D I T I O N

2. Write up solution to Ex. 2.17.

ROBERT SEDGEWICK
PHILIPPE FLAJOLET

3. Set up StdDraw from Algs booksite

4. Do Exercise 2.69.

48
 A N A LY T I C C O M B I N AT O R I C S
PART ONE

2. Recurrences

http://aofa.cs.princeton.edu