Chapter 5

Reductions, Recursion and Divide

and Conquer

CS 473: Fundamental Algorithms, Spring 2011

February 1, 2011

5.1 Reductions and Recursion

5.1.0.1 Reduction
Reducing problem A to problem B:
(A) Algorithm for A uses algorithm for B as a black box

5.1.0.2 Distinct Elements Problem

Problem Given an array A of n integers, are there any duplicates in A?

Naive algorithm:
for i = 1 to n − 1 do
for j = i + 1 to n do
if (A[i] = A[j])
return YES
return NO

Running time: O(n2 )

5.1.0.3 Reduction to Sorting

Sort A
for i = 1 to n − 1 do
if (A[i] = A[i + 1]) then
return YES
return NO

1
Running time: O(n) plus time to sort an array of n numbers
Important point: algorithm uses sorting as a black box

5.1.0.4 Two sides of Reductions

Suppose problem A reduces to problem B
(A) Positive direction: Algorithm for B implies an algorithm for A
(B) Negative direction: Suppose there is no “efficient” algorithm for A then it implies no
efficient algorithm for B (technical condition for reduction time necessary for this)
Example: Distinct Elements reduces to Sorting in O(n) time
(A) An O(n log n) time algorithm for Sorting implies an O(n log n) time algorithm for Dis-
tinct Elements problem.
(B) If there is no o(n log n) time algorithm for Distinct Elements problem then there is no
o(n log n) time algorithm for Sorting.

5.2 Recursion
5.2.0.5 Recursion
Reduction: reduce one problem to another

Recursion: a special case of reduction

(A) reduce problem to a smaller instance of itself
(B) self-reduction
(A) Problem instance of size n is reduced to one or more instances of size n − 1 or less.
(B) For termination, problem instances of small size are solved by some other method as
base cases

5.2.0.6 Recursion
(A) Recursion is a very powerful and fundamental technique
(B) Basis for several other methods
(A) Divide and conquer
(B) Dynamic programming
(C) Enumeration and branch and bound etc
(D) Some classes of greedy algorithms
(C) Makes proof of correctness easy (via induction)
(D) Recurrences arise in analysis

5.2.0.7 Selection Sort

Sort a given array A[1..n] of integers.

2
 shall have been thus transferred from the needle on which at the creation God placed them to
one of the other needles, tower, temple, and Brahmins alike will crumble into dust, and with a
thunderclap the world will vanish.

Of course, being good computer scientists, we read this story and immediately substitute n for the
hardwired constant sixty-four.4 How can we move a tower of n disks from one needle to another,
using a third needles as an occasional placeholder, never placing any disk on top of a smaller disk?

Algorithms The Tower of Hanoi puzzle Lecture 1: Recursion

The trick to solving

STOP!! this
That’s it! puzzle
We’re is We’ve
done! to think recursively.
successfully Instead
reduced of trying
the n-disk Towertoofsolve the
Hanoi entireto
problem puzzle
all at once, let’s concentrate
two instances of the (n − on moving
1)-disk Towerjust the largest
of Hanoi problem,disk.
whichWewecan’t move ithand
can gleefully at the
off beginning,
to the
becauseRecursion Fairy (or,
all the other to carry
disks the original
are covering it;story further,
we have totomove
the junior
thosemonks
n − 1atdisks
the temple).
to the third needle
before we can move the nth disk. And then after we move the nth disk, we have to move those
n − 1 disks back on top of it. So now all we have to figure out is how to. . .
recursion
3
This English translation is from W. W. Rouse Ball and H. S. M. Coxeter’s book Mathematical Recreations and Essays.
4
Recognizing that the underlying mathematical abstraction would be unchanged, we may also freely use ‘cookies’ and
‘dowels’ instead of ‘discs’ and ‘needles’. Ha ha. . . underlying!

recursion
2
The Tower of Hanoi algorithm; ignore everything but the bottom disk

Our algorithm does make one subtle but important assumption: there is a largest disk. In other
words, our recursive algorithm works for any n ≥ 1, but it breaks down when n = 0. We must
handle that base case directly. Fortunately, the monks at Benares, being good Buddhists, are quite
adept at moving zero disks from one needle to another.

Recursive version of Selection sort.

SelectSort(A[1..n]): The base case for the Tower of Hanoi algorithm; there is no bottom disk

if n = 1
While it’s tempting return
to think about how all those smaller disks get moved—in other words,
what happens
Find when the recursion
smallest numberis unfolded—it’s
in A. notLet necessary.
A[i] In befact, for more complicated
smallest number
problems, opening up the recursion is a distraction. Our only task is to reduce the problem to one
Swap A[1] and A[i]
or more simpler instances, or to solve the problem directly if such a reduction is impossible. Our
SelectSort(A[2..n])
algorithm is trivially correct when n = 0. For any n ≥ 1, the Recursion Fairy correctly moves (or
more formally, the inductive hypothesis implies that our algorithm correctly moves) the top n − 1
disks, so our algorithm is clearly correct.
Here’s the recursive Hanoi algorithm in more typical pseudocode.

T (n): time for SelectSort on anHANOI

n element array.
(n, src, dst, tmp):
if n > 0
H ANOI(n, src, tmp, dst)
move disk n from src to dst
T (n) = T (n − 1) + n for n > 1 and T (1) = 1 for n = 1 H ANOI(n, tmp, dst, src)

Let T (n) denote the number of moves required to transfer n disks—the running time of our
algorithm. Our vacuous base case implies that T (0) = 0, and the more general recursive algorithm
2
T (n) = Θ(n ). implies that T (n) = 2T (n − 1) + 1 for any n ≥ 1. The annihilator method lets us quickly derive a
closed form solution T (n) = 2n − 1 . In particular, moving a tower of 64 disks requires 264 − 1 =
18,446,744,073,709,551,615 individual moves. Thus, even at the impressive rate of one move per
second, the monks at Benares will be at work for approximately 585 billion years before, with a
thunderclap, the world will vanish.
5.2.0.8 Tower The
of Hanoi
Hanoialgorithm has two very simple non-recursive formulations, for those of us who do
not have an army of assistants to take care of smaller piles. Let’s label the needles 0, 1, and 2,

Move stack of n disks from peg 0 to peg 2, one

3 disk at a time.
Rule: cannot put a larger disk on a smaller disk.
Question: what is a strategy and how many moves does it take?

3
5.2.0.9 Tower of Hanoi via Recursion

5.2.0.10 Recursive Algorithm

Hanoi(n, src, dest, tmp):

If (n > 0) then
Hanoi(n − 1, src, tmp, dest)
Move disk n from src to dest
Hanoi(n − 1, tmp, dest, src)

T (n): time to move n disks via recursive strategy

T (n) = 2T (n − 1) + 1 n>1 and T (1) = 1

5.2.0.11 Analysis

T (n) = 2T (n − 1) + 1
= 22 T (n − 2) + 2 + 1
= ...
= 2i T (n − i) + 2i−1 + 2i−2 + . . . + 1
= ...
= 2n−1 T (1) + 2n−2 + . . . + 1
= 2n−1 + 2n−2 + . . . + 1
= (2n − 1)/(2 − 1) = 2n − 1

5.2.0.12 Non-Recursive Algorithms for Tower of Hanoi

Pegs numbered 0, 1, 2

Non-recursive Algorithm 1:
(A) Always move smallest disk forward if n is even, backward if n is odd.
(B) Never move the same disk twice in a row.
(C) Done when no legal move.
Non-recursive Algorithm 2:
(A) Let ρ(n) be the smallest integer k such that n/2k is not an integer. Example: ρ(40) = 4,
ρ(18) = 2.
(B) In step i move disk ρ(i) forward if n − i is even and backward if n − i is odd.
Moves are exactly same as those of recursive algorithm. Prove by induction.

4
5.3 Divide and Conquer
5.3.0.13 Divide and Conquer Paradigm
Divide and Conquer is a common and useful type of recursion
Approach
(A) Break problem instance into smaller instances - divide step
(B) Recursively solve problem on smaller instances
(C) Combine solutions to smaller instances to obtain a solution to the original instance -
conquer step
Question: Why is this not plain recursion?
(A) In divide and conquer, each smaller instance is typically at least a constant factor smaller
than the original instance which leads to efficient running times.
(B) There are many examples of this particular type of recursion that it deserves its own
treatment.

5.4 Merge Sort

5.4.1 Merge Sort
5.4.1.1 Sorting
Input Given an array of n elements

Goal Rearrange them in ascending order

5.4.2 Merge Sort [von Neumann]

5.4.2.1 MergeSort
1. Input: Array A[1 . . . n]
ALGORI T H M S

2. Divide into subarrays A[1 . . . m] and A[m + 1 . . . n], where m = bn/2c

ALGOR I T H M S

3. Recursively MergeSort A[1 . . . m] and A[m + 1 . . . n]

AGLOR H I M ST

4. Merge the sorted arrays

AGH I LM ORST

5
5.4.2.2 Merging Sorted Arrays
(A) Use a new array C to store the merged array
(B) Scan A and B from left-to-right, storing elements in C in order

AGLOR H I M ST
AGH I LM ORST

(C) Merge two arrays using only constantly more extra space (in-place merge sort): doable
but complicated and typically impractical

5.4.3 Analysis
5.4.3.1 Running Time
T (n): time for merge sort to sort an n element array

T (n) = T (bn/2c) + T (dn/2e) + cn

What do we want as a solution to the recurrence?
Almost always only an asymptotically tight bound. That is we want to know f (n) such
that T (n) = Θ(f (n)).
(A) T (n) = O(f (n)) - upper bound
(B) T (n) = Ω(f (n)) - lower bound

5.4.4 Solving Recurrences

5.4.4.1 Solving Recurrences: Some Techniques
(A) Know some basic math: geometric series, logarithms, exponentials, elementary calculus
(B) Expand the recurrence and spot a pattern and use simple math
(C) Recursion tree method — imagine the computation as a tree
(D) Guess and verify — useful for proving upper and lower bounds even if not tight bounds
Albert Einstein: “Everything should be made as simple as possible, but not simpler.”
Know where to be loose in analysis and where to be tight. Comes with practice, practice,
practice!

5.4.4.2 Recursion Trees

1. Unroll the recurrence. T (n) = 2T (n/2) + cn

2. Identify a pattern. At the ith level total work is cn

3. Sum over all levels. The number of levels is log n. So total is cn log n = O(n log n)

6
 n cn

n/2 cn/2 n/2 cn/2

n/4 cn/4 n/4 cn/4 n/4 cn/4 n/4 cn/4

5.4.5 MergeSort Analysis

5.4.5.1 When n is not a power of 2

(A) When n is not a power of 2, the running time of mergesort is expressed as

T (n) = T (bn/2c) + T (dn/2e) + cn

(B) n1 = 2k−1 < n ≤ 2k = n2 (n1 , n2 powers of 2)

(C) T (n1 ) < T (n) ≤ T (n2 ) (Why?)
(D) T (n) = Θ(n log n) since n/2 ≤ n1 < n ≤ n2 ≤ 2n.

5.4.5.2 Recursion Trees

T (n) = T (bn/2c) + T (dn/2e) + cn

Observation: For any number x, bx/2c + dx/2e = x.

5.4.5.3 Mergesort Analysis

When n is not a power of 2: Guess and Verify MergeSort: n is not a power of

2
If n is power of 2 we saw that T (n) = Θ(n log n).
Can guess that T (n) = Θ(n log n) for all n.
Verify? proof by induction!

Induction Hypothesis: T (n) ≤ 2cn log n for all n ≥ 1

Base Case: n = 1. T (1) = 0 since no need to do any work and 2cn log n = 0 for n = 1.
Induction Step Assume T (k) ≤ 2ck log k for all k < n and prove it for k = n.

7
5.4.5.4 Induction Step
We have
T (n) = T (bn/2c) + T (dn/2e) + cn
≤ 2cbn/2c logbn/2c + 2cdn/2e logdn/2e + cn (by induction)
≤ 2cbn/2c logdn/2e + 2cdn/2e logdn/2e + cn
≤ 2c(bn/2c + dn/2e) logdn/2e + cn
≤ 2cn logdn/2e + cn
≤ 2cn log(2n/3) + cn (since dn/2e ≤ 2n/3 for all n ≥ 2)
≤ 2cn log n + cn(1 − 2 log 3/2)
≤ 2cn log n + cn(log 2 − log 9/4)
≤ 2cn log n

5.4.5.5 Guess and Verify

The math worked out like magic!
Why was 2cn log n chosen instead of say 4cn log n?

Typically we don’t know upfront what constant to choose. Instead we assume that
T (n) ≤ αcn log n for some constant α that will be fixed later. All we need to prove that
there is some sufficiently large constant α that will make the algebra go through.

We need to choose α such that α log 3/2 > 1.

Typically you do the algebra with α and then show at the end that α can be chosen to
be sufficiently large constant.

5.4.5.6 Guess and Verify: When is a guess incorrect?

Suppose we guessed that the soln to the mergesort recurrent is T (n) = O(n). We try to
prove by induction that T (n) ≤ αcn for some constant α.

Induction Step: attempt

T (n) = T (bn/2c) + T (dn/2e) + cn
≤ αcbn/2c + αcdn/2e + cn
≤ αcn + cn
≤ (α + 1)cn
But we want to show that T (n) ≤ αcn! So guess does not work for any constant α. Suggests
that our guess is incorrect.

8
5.4.5.7 Selection Sort vs Merge Sort
(A) Selection Sort spends O(n) work to reduce problem from n to n − 1 leading to O(n2 )
running time.
(B) Merge Sort spends O(n) time after reducing problem to two instances of size n/2 each.
Running time is O(n log n)
Question: Merge Sort splits into 2 (roughly) equal sized arrays. Can we do better by
splitting into more than 2 arrays? Say k arrays of size n/k each?

5.5 Quick Sort

5.5.0.8 Quick Sort

Quick Sort[Hoare]

1. Pick a pivot element from array

2. Split array into 3 subarrays: those smaller than pivot, those larger than pivot, and the
pivot itself. Linear scan of array does it. Time is O(n)

3. Recursively sort the subarrays, and concatenate them.

Example:
(A) array: 16, 12, 14, 20, 5, 3, 18, 19, 1
(B) pivot: 16
(C) split into 12, 14, 5, 3, 1 and 20, 19, 18 and recursively sort
(D) put them together with pivot in middle

5.5.0.9 Time Analysis

(A) Let k be the rank of the chosen pivot. Then, T (n) = T (k − 1) + T (n − k) + O(n)
(B) If k = dn/2e then T (n) = T (dn/2e − 1) + T (bn/2c) + O(n) ≤ 2T (n/2) + O(n). Then,
T (n) = O(n log n).
(A) Theoretically, median can be found in linear time.
(C) Typically, pivot is the first or last element of array. Then,

T (n) = max (T (k − 1) + T (n − k) + O(n))

1≤k≤n

In the worst case T (n) = T (n − 1) + O(n), which means T (n) = O(n2 ). Happens if
array is already sorted and pivot is always first element.

9
5.6 Fast Multiplication

5.7 The Problem

5.7.0.10 Multiplying Numbers

Problem Given two n-digit numbers x and y, compute their product.

Grade School Multiplication

Compute “partial product” by multiplying each digit of y with x and adding the partial
products.
3141
×2718
25128
3141
21987
6282
8537238

5.8 Algorithmic Solution

5.8.1 Grade School Multiplication
5.8.1.1 Time Analysis of Grade School Multiplication
(A) Each partial product: Θ(n)
(B) Number of partial products: Θ(n)
(C) Addition of partial products: Θ(n2 )
(D) Total time: Θ(n2 )

5.8.1.2 A Trick of Gauss

Carl Fridrich Gauss: 1777–1855 “Prince of Mathematicians”

Observation: Multiply two complex numbers: (a + bi) and (c + di)

(a + bi)(c + di) = ac − bd + (ad + bc)i

How many multiplications do we need?

Only 3! If we do extra additions and subtractions.
Compute ac, bd, (a + b)(c + d). Then (ad + bc) = (a + b)(c + d) − ac − bd

10
5.8.2 Divide and Conquer Solution
5.8.2.1 Divide and Conquer
Assume n is a power of 2 for simplicity and numbers are in decimal.

(A) x = xn−1 xn−2 . . . x0 and y = yn−1 yn−2 . . . y0

(B) x = 10n/2 xL + xR where xL = xn−1 . . . xn/2 and xR = xn/2−1 . . . x0
(C) y = 10n/2 yL + yR where yL = yn−1 . . . yn/2 and yR = yn/2−1 . . . y0
Therefore
xy = (10n/2 xL + xR )(10n/2 yL + yR ) = 10n xL yL + 10n/2 (xL yR + xR yL ) + xR yR

5.8.2.2 Example

1234 × 5678 = (100 × 12 + 34) × (100 × 56 + 78)

= 10000 × 12 × 56
+100 × (12 × 78 + 34 × 56)
+34 × 78

5.8.2.3 Time Analysis

xy = (10n/2 xL + xR )(10n/2 yL + yR ) = 10n xL yL + 10n/2 (xL yR + xR yL ) + xR yR
4 recursive multiplications of number of size n/2 each plus 4 additions and left shifts
(adding enough 0’s to the right)
T (n) = 4T (n/2) + O(n) T (1) = O(1)
T (n) = Θ(n2 ). No better than grade school multiplication!
Can we invoke Gauss’s trick here?

5.8.3 Karatsuba’s Algorithm

5.8.3.1 Improving the Running Time
xy = (10n/2 xL + xR )(10n/2 yL + yR ) = 10n xL yL + 10n/2 (xL yR + xR yL ) + xR yR
Gauss trick: xL yR + xR yL = (xL + xR )(yL + yR ) − xL yL − xR yR
Recursively compute only xL yL , xR yR , (xL + xR )(yL + yR ).
Time Analysis
Running time is given by
T (n) = 3T (n/2) + O(n) T (1) = O(1)
which means T (n) = O(nlog2 3 ) = O(n1.585 )

11