Introduction to Algorithms: 6.

006
Massachusetts Institute of Technology
Instructors: Erik Demaine, Jason Ku, and Justin Solomon Lecture 15: Recursive Algorithms

Lecture 15: Recursive Algorithms

How to Solve an Algorithms Problem (Review)

• Reduce to a problem you already know (use data structure or algorithm)
Search Data Structures Sort Algorithms Graph Algorithms
Array Insertion Sort Breadth First Search
Linked List Selection Sort DAG Relaxation (DFS + Topo)
Dynamic Array Merge Sort Dijkstra
Sorted Array Counting Sort Bellman-Ford
Direct-Access Array Radix Sort Johnson
Hash Table AVL Sort
AVL Tree Heap Sort
Binary Heap
• Design your own recursive algorithm
– Constant-sized program to solve arbitrary input
– Need looping or recursion, analyze by induction
– Recursive function call: vertex in a graph, directed edge from A → B if B calls A
– Dependency graph of recursive calls must be acyclic (if can terminate)
– Classify based on shape of graph
Class Graph
Brute Force Star
Decrease & Conquer Chain
Divide & Conquer Tree
Dynamic Programming DAG
Greedy/Incremental Subgraph

– Hard part is thinking inductively to construct recurrence on subproblems

– How to solve a problem recursively (SRT BOT)
1. Subproblem definition
2. Relate subproblem solutions recursively
3. Topological order on subproblems (⇒ subproblem DAG)
4. Base cases of relation
5. Original problem solution via subproblem(s)
6. Time analysis
2 Lecture 15: Recursive Algorithms

Merge Sort in SRT BOT Framework

• Merge sorting an array A of n elements can be expressed in SRT BOT as follows:

– Subproblems: S(i, j) = sorted array on elements of A[i : j] for 0 ≤ i ≤ j ≤ n

– Relation: S(i, j) = merge(S(i, m), S(m, j)) where m = b(i + j)/2c
– Topo. order: Increasing j − i
– Base cases: S(i, i + 1) = [A[i]]
– Original: S(0, n)
– Time: T (n) = 2 T (n/2) + O(n) = O(n lg n)

• In this case, subproblem DAG is a tree (divide & conquer)

Fibonacci Numbers
• Suppose we want to compute the nth Fibonacci number Fn

• Subproblems: F (i) = the ith Fibonacci number Fi for i ∈ {0, 1, . . . , n}

• Relation: F (i) = F (i − 1) + F (i − 2) (definition of Fibonacci numbers)

• Topo. order: Increasing i

• Base cases: F (0) = 0, F (1) = 1

• Original prob.: F (n)

1 def fib(n):
2 if n < 2: return n # base case
3 return fib(n - 1) + fib(n - 2) # recurrence

• Divide and conquer implies a tree of recursive calls (draw tree)

• Time: T (n) = T (n − 1) + T (n − 2) + O(1) > 2T (n − 2), T (n) = Ω(2n/2 ) exponential... :(

• Subproblem F (k) computed more than once! (F (n − k) times)

• Can we avoid this waste?

Lecture 15: Recursive Algorithms 3

Re-using Subproblem Solutions

• Draw subproblem dependencies as a DAG

• To solve, either:

– Top down: record subproblem solutions in a memo and re-use

(recursion + memoization)
– Bottom up: solve subproblems in topological sort order (usually via loops)

• For Fibonacci, n + 1 subproblems (vertices) and < 2n dependencies (edges)

• Time to compute is then O(n) additions

1 # recursive solution (top down)

2 def fib(n):
3 memo = {}
4 def F(i):
5 if i < 2: return i # base cases
6 if i not in memo: # check memo
7 memo[i] = F(i - 1) + F(i - 2) # relation
8 return memo[i]
9 return F(n) # original

1 # iterative solution (bottom up)

2 def fib(n):
3 F = {}
4 F[0], F[1] = 0, 1 # base cases
5 for i in range(2, n + 1): # topological order
6 F[i] = F[i - 1] + F[i - 2] # relation
7 return F[n] # original

• A subtlety is that Fibonacci numbers grow to Θ(n) bits long, potentially  word size w

• Each addition costs O(dn/we) time

• So total cost is O(ndn/we) = O(n + n2 /w) time

4 Lecture 15: Recursive Algorithms

Dynamic Programming
• Weird name coined by Richard Bellman

– Wanted government funding, needed cool name to disguise doing mathematics!

– Updating (dynamic) a plan or schedule (program)

• Existence of recursive solution implies decomposable subproblems1

• Recursive algorithm implies a graph of computation

• Dynamic programming if subproblem dependencies overlap (DAG, in-degree > 1)

• “Recurse but re-use” (Top down: record and lookup subproblem solutions)

• “Careful brute force” (Bottom up: do each subproblem in order)

• Often useful for counting/optimization problems: almost trivially correct recurrences

How to Solve a Problem Recursively (SRT BOT)

1. Subproblem definition subproblem x ∈ X

• Describe the meaning of a subproblem in words, in terms of parameters

• Often subsets of input: prefixes, suffixes, contiguous substrings of a sequence
• Often record partial state: add subproblems by incrementing some auxiliary variables

2. Relate subproblem solutions recursively x(i) = f (x(j), . . .) for one or more j < i

3. Topological order to argue relation is acyclic and subproblems form a DAG

4. Base cases

• State solutions for all (reachable) independent subproblems where relation breaks down

5. Original problem

• Show how to compute solution to original problem from solutions to subproblem(s)

• Possibly use parent pointers to recover actual solution, not just objective function

6. Time analysis
P
• x∈X work(x), or if work(x) = O(W ) for all x ∈ X, then |X| · O(W )
• work(x) measures nonrecursive work in relation; treat recursions as taking O(1) time

1
This property often called optimal substructure. It is a property of recursion, not just dynamic programming
Lecture 15: Recursive Algorithms 5

DAG Shortest Paths

• Recall the DAG SSSP problem: given a DAG G and vertex s, compute δ(s, v) for all v ∈ V

• Subproblems: δ(s, v) for all v ∈ V

• Relation: δ(s, v) = min{δ(s, u) + w(u, v) | u ∈ Adj− (v)} ∪ {∞}

• Topo. order: Topological order of G

• Base cases: δ(s, s) = 0

• Original: All subproblems

P −
• Time: v∈V O(1 + | Adj (v)|) = O(|V | + |E|)

• DAG Relaxation computes the same min values as this dynamic program, just

– step-by-step (if new value < min, update min via edge relaxation), and
– from the perspective of u and Adj+ (u) instead of v and Adj− (v)

Bowling
• Given n pins labeled 0, 1, . . . , n − 1

• Pin i has value vi

• Ball of size similar to pin can hit either

– 1 pin i, in which case we get vi points

– 2 adjacent pins i and i + 1, in which case we get vi · vi+1 points

• Once a pin is hit, it can’t be hit again (removed)

• Problem: Throw zero or more balls to maximize total points

• Example: [ −1, 1 , 1 , 1 , 9, 9 , 3 , −3, −5 , 2, 2 ]

6 Lecture 15: Recursive Algorithms

Bowling Algorithms
• Let’s start with a more familiar divide-and-conquer algorithm:
– Subproblems: B(i, j) = maximum score starting with just pins i, i + 1, . . . , j − 1,
for 0 ≤ i ≤ j ≤ n
– Relation:
∗ m = b(i + j)/2c
∗ Either hit m and m + 1 together, or don’t
∗ B(i, j) = max{vm · vm+1 + B(i, m) + B(m + 2, j), B(i, m + 1) + B(m + 1, j)}
– Topo. order: Increasing j − i
– Base cases: B(i, i) = 0, B(i, i + 1) = max{vi , 0}
– Original: B(0, n)
– Time: T (n) = 4 T (n/2) + O(1) = O(n2 )
• This algorithm works but isn’t very fast, and doesn’t generalize well
(e.g., to allow for a bigger ball that hits three balls at once)

• Dynamic programming algorithm: use suffixes

– Subproblems: B(i) = maximum score starting with just pins i, i + 1, . . . , n − 1,
for 0 ≤ i ≤ n
– Relation:
∗ Locally brute-force what could happen with first pin (original pin i):
skip pin, hit one pin, hit two pins
∗ Reduce to smaller suffix and recurse, either B(i + 1) or B(i + 2)
∗ B(i) = max{B(i + 1), vi + B(i + 1), vi · vi+1 + B(i + 2)}
– Topo. order: Decreasing i (for i = n, n − 1, . . . , 0)
– Base cases: B(n) = B(n + 1) = 0
– Original: B(0)
– Time: (assuming memoization)
∗ Θ(n) subproblems · Θ(1) work in each
∗ Θ(n) total time
• Fast and easy to generalize!
• Equivalent to maximum-weight path in Subproblem DAG:

max{v0 , 0} max{v1 , 0} max{v2 , 0}

B0 B1 B2 B3 ··· Bn

v0 · v1 v1 · v2 v2 · v3
 Lecture 15: Recursive Algorithms 7

Bowling Code
• Converting a SRT BOT specification into code is automatic/straightforward
• Here’s the result for the Bowling Dynamic Program above:

1 # recursive solution (top down)

2 def bowl(v):
3 memo = {}
4 def B(i):
5 if i >= len(v): return 0 # base cases
6 if i not in memo: # check memo
7 memo[i] = max(B(i+1), # relation: skip pin i
8 v[i] + B(i+1), # OR bowl pin i separately
9 v[i] * v[i+1] + B(i+2)) # OR bowl pins i and i+1 together
10 return memo[i]
11 return B(0) # original

1 # iterative solution (bottom up)

2 def bowl(v):
3 B = {}
4 B[len(v)] = 0 # base cases
5 B[len(v)+1] = 0
6 for i in reversed(range(len(v))): # topological order
7 B[i] = max(B[i+1], # relation: skip pin i
8 v[i] + B(i+1), # OR bowl pin i separately
9 v[i] * v[i+1] + B(i+2)) # OR bowl pins i and i+1 together
10 return B[0] # original

How to Relate Subproblem Solutions

• The general approach we’re following to define a relation on subproblem solutions:
– Identify a question about a subproblem solution that, if you knew the answer to, would
reduce to “smaller” subproblem(s)
∗ In case of bowling, the question is “how do we bowl the first couple of pins?”
– Then locally brute-force the question by trying all possible answers, and taking the best
∗ In case of bowling, we take the max because the problem asks to maximize
– Alternatively, we can think of correctly guessing the answer to the question, and di-
rectly recursing; but then we actually check all possible guesses, and return the “best”
• The key for efficiency is for the question to have a small (polynomial) number of possible
answers, so brute forcing is not too expensive
• Often (but not always) the nonrecursive work to compute the relation is equal to the number
of answers we’re trying
MIT OpenCourseWare
https://ocw.mit.edu

6.006 Introduction to Algorithms

Spring 2020

For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms