Brute Force Algorithms

 Trying all possible solutions

 Main approach
◦ Generate and evaluate possible solutions until”
 Satisfactory solution is found
 Best solution is found (if can be determined)
 All possible solutions found
 Return best solution
 Return failure if no satisfactory solution
 Limitations:
Generally most expensive approach

2
 Brute Force Algorithms – Example
Traveling Salesman Problem (TSP)

o Given weighted undirected graph (map of cities)

o Find lowest cost path visiting all nodes (cities) once

o No known polynomial-time general solution

Brute force approach

o Find all possible paths using recursive backtracking

o Calculate cost of each path

o Return lowest cost path

3
 Backtracking Algorithms
 Backtracking.
◦ is used to solve problems in which a sequence of objects
is chosen from a specified set so that the sequence
satisfies some criterion.
◦ is a modified depth-first search of a tree.

 Backtracking involves only a tree search.

 Backtracking is the procedure whereby, after determining
that a node can lead to nothing but dead nodes, we go back
(“backtrack”) to the node’s parent and proceed with the
search on the next child
4
 Backtracking Algorithms
 Finding a solution to a problem can't be based on a straight path
to the goal.

 Example:

 Traversing a maze.

 We need a better approach than brute force (independently

evaluating all possible solutions).

 Think of the TSP problem – many possible solutions share

partial tours (why not treat identical partial tours as a single
partial solution?).

5
 Backtracking Algorithms
 We call a node non-promising if when visiting the node we
determine that it cannot possibly lead to a solution. Otherwise,
we call it promising.

 In summary, backtracking consists of

 Doing a depth-first search of a state space tree,

 Checking whether each node is promising, and, if it is nonpromising,

backtracking to the node’s parent.

 This is called pruning the state space tree, and the subtree
consisting of the visited nodes is called the pruned state space
tree.

6
 Example 1: Solving a Maze
 Clearly, at a single junction you
could have even more than 2
choices.

 The backtracking strategy says to try

each choice, one after the other,

◦ if you ever get stuck,

" backtr ack" to the junction and
C
try the next choice. B
A
 If you try all choices and never
found a way out, then there IS no
solution to the maze.
7
 Solving a Maze using Brute Force
Algorithm
 Try all possible sequences of moves:

most will fail (be blocked)

what does "all possible sequences" mean?

how many moves

need to avoid loops

generating all possible sequences may not

be possible

8
 Example 2: Eight Queens Problem
 Find an arrangement of 8 queens on a
single chess board such that no two
queens are attacking one another.
 In chess, queens can move all the way
down any row, column or diagonal (so
long as no pieces are in the way).
◦ Due to the first two restrictions, it's
clear that each row and column of
the board will have exactly one
queen.

9
 Example 2: Eight Queens Problem
Number of Possible placements
= 64 * 63 * 62 * 61 * 60 * 59 * 58 * 57
= 178,462, 987, 637, 760 / 8!
= 4,426,165,368

10
 Example: Eight Queens Problem
 The backtracking strategy is as Q
follows:
Q
Q
1. Place a queen on the first Q
available square in row 1. Q Q
2. Move onto the next row, Continue…
placing a queen on the
first available square there
Animated Example:
(that doesn't conflict with http://www.hbmeyer.de/backtrack/a
chtdamen/eight.htm#up
the previously placed
queens).
11
 Example: Eight Queens Problem
3. Continue in this fashion until Q
either: Q
Q
a. you have solved the
Q
problem, or Q Q
b. you get stuck.
Continue…
 When you get stuck, remove
the queens that got you
there, until you get to a row
Animated Example:
where there is another valid http://www.hbmeyer.de/backtrack/a
chtdamen/eight.htm#up
square to try.

12
 Divide-and-Conquer: Main Idea
 Divide and Conquer is a designing algorithms
technique to analyze the running time of such
algorithms
 Basic Procedure
◦ divide problem into subproblems
◦ For subproblems that are really small (trivial),
solve them directly. Else solve them recursively.
(conquer)
◦ combine solutions to subproblems to get solution
to original problem
 A good example of this approach is Mergesort.
13
 Divide-and-conquer algorithms
 Divide-and-conquer algorithms often follow a generic
pattern.
 They tackle a problem of size n by recursively
solving, say, a subproblems of size n/b and then
combining these answers in O(nd) time, for some a; b;
d > 0 (in the multiplication algorithm, a = 3, b = 2,
and d = 1).
 Their running time can therefore be captured by the
equation T(n) = a T (n/b) + O(nd).
 derive a closed-form solution to this general
recurrence so that we no longer have to solve it
explicitly in each new instance.
 Greedy Algorithm
 Used to solve optimization problem

◦ Recursion

◦ is one in which you want to find, not just a solution, but the best
solution

 A “greedy algorithm” sometimes works well for optimization problems

 A greedy algorithm works in phases. At each phase:

◦ You take the best you can get right now, without regard for future
consequences

◦ We hope that by choosing a local optimum at each step, you will end
up at a global optimum

15
 Greedy Algorithm
 Idea: When we have a choice to make, make the one
that looks best right now
◦ Make a locally optimal choice in hope of getting a
globally optimal solution
◦ a decision is made that appears to be good, without
regard for future consequences
 Greedy algorithms don’t always yield an optimal
solution
 Makes the choice that looks best at the moment in order
to get optimal solution 16
 Examples of Greedy Algorithms

• Counting Money
• Activity Selection
• Minimum Spanning Tree
• Dijkstra Shortest Path
• Huffman Codes
• Fractional Knapsack

17
 Example: Counting Money
 Suppose you want to count out a certain amount of
money, using the fewest possible bills and coins
 A greedy algorithm would do this would be:
At each step, take the largest possible bill or coin that
does not overshoot
◦ Example: To make $6.39, you can choose:
 a $5 bill
 a $1 bill, to make $6
 a 25¢ coin, to make $6.25
 A 10¢ coin, to make $6.35
 four 1¢ coins, to make $6.39
 For US money, the greedy algorithm always gives the
optimum solution
18
 Example: Counting Money
 Consider the problems of making n cents change with 50
cents, 20 cents, 10 cents, 5 cents and 1 cent using the least
total number of coins.
 At each step we choose the coin of the largest denomination
possible to add to the group of change without exceeding n
cents
Pseudocode:The Greedy Change-Making Algorithm
procedure change(c1,c2,…,cr: values of denomination of
coins where c1 > c2 >…> cr ; n: a positive integer)
for i := 1 to r
while n ≥ ci
begin
add a coin with value ci to the change
n := n - ci
end

19
 A Failure of Greedy Algorithm
 In some (fictional) monetary system, “krons” come in 1
kron, 7 kron, and 10 kron coins
 Using a greedy algorithm to count out 15 krons, you
would get
◦ A 10 kron piece
◦ Five 1 kron pieces, for a total of 15 krons
◦ This requires six coins
 A better solution would be to use two 7 kron pieces and
one 1 kron piece
◦ This only requires three coins
 The greedy algorithm results in a solution, but not in an
optimal solution

20
 A Scheduling Problem
 You have to run nine jobs(processes), with running times of 3,
5, 6, 10, 11, 14, 15, 18, and 20 minutes
 You have three processors on which you can run these jobs
 You decide to do the longest-running jobs first, on whatever
processor is available

P1 20 10 3
P2
18 11 6
P3
15 14 5

 Time to completion: 18 + 11 + 6 = 35 minutes

 This solution isn’t bad, but we might be able to do better
21
 A Scheduling Problem
 What would be the result if you ran the shortest job first?
 Again, the running times are 3, 5, 6, 10, 11, 14, 15, 18, and 20
minutes

P1 3 10 15
P2
5 11 18
P3
6 14 20

 That wasn’t such a good idea; time to completion is now

6 + 14 + 20 = 40 minutes
 Note, however, that the greedy algorithm itself is fast
◦ All we had to do at each stage was pick the minimum or maximum
22
 A Scheduling Problem

 Better solutions do exist:

P1 20 14
P2
18 11 5
P3
15 10 6 3

 This solution is clearly optimal (why?)

 Clearly, there are other optimal solutions (why?)
 How do we find such a solution?
◦ One way: Try all possible assignments of jobs to processors
◦ Unfortunately, this approach can take exponential time

23
 Data Compression
 Q. Given a text that uses 32 symbols (26 different letters, space,
and
some punctuation characters), how can we encode this text in bits?
 Ans. We can encode 25 different symbols using a fixed length of 5
bits per symbol. This is called fixed length encoding.
 Q. Some symbols (e, t, a, o, i, n) are used far more often than
others.
How can we use this to reduce our encoding?
 Ans. Encode these characters with fewer bits, and the others with
more bits.
 Q. How do we know when the next symbol begins?
 Ans. Use a separation symbol (like the pause in Morse), or make
sure that there is no ambiguity by ensuring that no code is a prefix
of another one.
 Ex. c(a) = 01, c(b) = 010 and c(e) = 1?
◦ What is 0101
24
 Prefix Codes
 Definition. A prefix code for a set S is a function c that maps
each x ∈ S to 1s and 0s in such a way that for x, y∈S, x≠y,
c(x) is not a prefix of c(y).
 Easy Definition: Prefix(-free) code: no codeword is also a
prefix of some other codewords (Un-ambiguous)
 Example:
◦ c(a) = 11, c(e) = 01, c(k) = 001, c(l) = 10, c(u) = 000
Q. What is the meaning of 1001000001 ?
Ans: “Leuk:
 Suppose frequencies are known in a text of 1G, where fa=0.4,
fe=0.2, fk=0.2, fl=0.1, fu=0.1
Q. What is the size of the encoded text?
Ans. 2*fa + 2*fe + 3*fk + 2*fl + 3*fu = 2.4G

25
 Optimal Prefix Codes
 Definition. The average bits per letter of a prefix code c
is the sum over all symbols of its frequency times the
number of bits of its encoding:

ABL(c ) = ∑ f x ⋅ c ( x )
x∈S

 We would like to find a prefix code that is has the lowest

possible average bits per letter.
 A prefix code that minimizes the average number of bits
per letter is called optimal
 Suppose we model a code in a binary tree…

26
 Representing Prefix Codes using
Binary Trees
 Q. What is the meaning of 111010001111101000 ?
Ans. 111010001111101000
s I m p e l

ABL(c ) = ∑ f x ⋅ c ( x )
x∈S

ABL(T ) = ∑ f x ⋅ depth T ( x)
x∈S

 Q. How can this prefix code be made more efficient?

Ans. Change encoding of p and s to a shorter one.
This tree is now full.

27
 Optimal Prefix Codes: Huffman
Encoding
 Observation. Lowest frequency items should be at the
lowest level in tree of optimal prefix code.
 Observation. For n > 1, the lowest level always contains
at least two leaves.
 Observation.The order in which items appear in a level
does not matter.
 Claim. There is an optimal prefix code with tree T* where
the two lowest-frequency letters are assigned to leaves
that are siblings in T*.
 Greedy template. [Huffman, 1952] Create tree bottom-up.
 Make two leaves for two lowest-frequency letters y and z.
 Recursively build tree for the rest using a meta-letter for
yz.

28
 Huffman Encoding: Greedy Analysis
 Huffman codes: compressing data (savings of 20% to 90%)
 Huffman’s greedy algorithm uses a table of the frequencies of
occurrence of each character to build up an optimal way of
representing each character as a binary string

29
 Huffman Encoding: Example

 Huffman codes: compressing data (savings of 20% to 90%)

 Huffman’s greedy algorithm uses a table of the frequencies of
occurrence of each character to build up an optimal way of
representing each character as a binary string

30
 Example, Step 1
 Assume that relative frequencies are:
◦ A: 40
◦ B: 20
◦ C: 10
◦ D: 10
◦ R: 20
 (We chose simpler numbers than the real frequencies)
 Smallest number are 10 and 10 (C and D), so connect those

31
 Example, Step 5
 Assign 0 to left branches, 1 to right branches
 Each encoding is a path from the root
 A=0
B = 100
C = 1010
D = 1011
R = 11
 Each path
terminates at a
leaf
 Do you see
why encoded
strings are
decodable?

32
 Spanning Tree

 Spanning tree. Let T = (V, F) be a subgraph of G = (V, E).

 T is a spanning tree of G: T that contains all the vertices of a graph G.
◦ T is acyclic and connected.
◦ T is connected and has |V| - 1 arcs.
◦ T is acyclic and has |V| - 1 arcs.
◦ T is minimally connected: removal of any arc disconnects it.
◦ T is maximally acyclic: addition of any arc creates a cycle.
◦ T has a unique simple path between every pair of vertices.
2 3 2 3
1 1

6 4 6 4

5 5

7 8 7 8
G = (V, E)
T = (V, F)
33
 Minimum Spanning Trees
•A spanning tree for G is a free tree that connects all the
vertices in V.
•The cost of a spanning tree is the sum of the costs of the edges
in the tree.
•Given: a Connected, undirected, weighted
b c graph, G =(V, E) in which each edge (u, v) in
11 5
E has a cost c (u, v) attached to it.
a 1 -3 •Find: Minimum - weight spanning tree, T
3

d e f
0 2 b c
5

a 1
3 -3

d e f
34 0
 Minimum Spanning Trees
Minimum spanning tree. Given connected graph G with real-
valued arc weights ce, an MST is a spanning tree of G whose
sum of arc weights is minimized.

2 24 3 2 3
4 4
1 1
23 9 9
6 18 6
6 6
5 4 5 4
16 11 11
8 5 8 5
7 7
10 14
7 21 8 7 8

G = (V, E) T = (V, F) w(T) = 50

Cayley's Theorem (1889). There are nn-2 spanning trees of Kn.

35
 Applications
 Designing physical networks.
◦ telephone, electrical, hydraulic, TV cable, computer, road
 Cluster analysis.
◦ delete long edges leaves connected components
◦ finding clusters of quasars and Seyfert galaxies
◦ analyzing fungal spore spatial patterns
 Approximate solutions to NP-hard problems.
◦ metric TSP, Steiner tree
 Indirect applications.
◦ describing arrangements of nuclei in skin cells for cancer
research
◦ learning salient features for real-time face verification
◦ modeling locality of particle interactions in turbulent fluid flow
◦ reducing data storage in sequencing amino acids in a protein
36
 Minimum Spanning Tree
 There can be many spanning trees of a graph

 In fact, there can be many minimum spanning

trees of a graph? WHY?

 In some cases, there is only one minimum

spanning tree. HW?

37
 Generic Algorithm
“Grows” a set A.

A is subset of some MST.

Edge is “safe” if it can be added to A without

destroying this invariant

A := ∅;
while A not complete tree do
find a safe edge (u, v);
A := A ∪ {(u, v)}
do

38
 Fundamental Cycle
Fundamental cycle.
◦ Adding any non-tree arc e to T forms unique cycle C.
◦ Deleting any arc f ∈ C from T ∪ {e} results in new
spanning tree.
2 3
1

6 4

5 f
9

e 8
7

Cycle optimality conditions: For every non-tree arc e, and for

every tree arc f in its fundamental cycle: cf ≤ ce.
Observation: If cf > ce then T is not a MST.
39
 Fundamental Cycle
Fundamental cut.
◦ Deleting any tree arc f from T disconnects tree into two
components with cut D.
◦ Adding back any arc e ∈ D to T - {f} results in new spanning
tree. 2 3
1

6 4
f

5
9

e 8
7
Cut optimality conditions: For every tree arc f, and for every
non-tree arc e in its fundamental cut: ce ≥ cf.
Observation: If ce < cf then T not a MST.

40
 Greedy Algorithms for Minimum
Spanning Trees
1. Kruskal's algorithm. Start with T = φ. Consider edges in
ascending order of cost. Insert edge e in T unless doing so
would create a cycle.
2. Reverse-Delete algorithm. Start with T = E. Consider edges
in descending order of cost. Delete edge e from T unless
doing so would disconnect T.
3. Prim's algorithm. Start with some root node s and greedily
grow a tree T from s outward. At each step, add the cheapest
edge e to T that has exactly one endpoint in T.
4. Remark. All three algorithms produce an MST.

41
 Greedy Algorithms for Minimum
Spanning Trees
Simplifying assumption. All edge costs ce are distinct.
Cut property. Let S be any subset of nodes, and let e be the min
cost edge with exactly one endpoint in S. Then the MST
contains e.
Cycle property. Let C be any cycle, and let f be the max cost
edge belonging to C. Then the MST does not contain f.

C
f

S
e

e is in the MST f is not in the MST

42
 Kruskal's Algorithm:

 First, sort edges in increasing order of cost.

Continue to pick edges in this order without
creating a cycle.

 When all vertices have been selected, then a

minimum spanning tree has been created.

43
 Kruskal's Algorithm: Proof of
Correctness
Kruskal's algorithm.
◦ Consider edges in ascending order of weight.
◦ Case 1: If adding e to T creates a cycle, discard e according
to cycle property.
◦ Case 2: Otherwise, insert e = (u, v) into T according to cut
property where S = set of nodes in u's connected component.

v S
e

e u

Case 1 Case 2
44
 Kruskal's Algorithm

16
5 4 11
12
7
3 14
6 9 2
10 8
15
17
13 18

consider the edges in increasing order of weight,

add in an edge iff it does not cause a cycle

45
 Prim’s Algorithm
MST-Prim(G, w, r)
6 4
Q = V[G];
5 9
for each u ∈ Q
key[u] = ∞;
14
10 2
key[r] = 0;
15
p[r] = NULL;
while (Q not empty) 3 8
u = ExtractMin(Q);
Run on example gr aph
for each v ∈ Adj[u]
if (v ∈ Q and w(u,v) < key[v])
p[v] = u;
key[v] = w(u,v);

46
Prim’s Algorithm
MST-Prim(G, w, r)
6 4 4
Q = V[G];
5 9
for each u ∈ Q
5 2 9
key[u] = ∞;
14
10 2 u
key[r] = 0;
15
p[r] = NULL; 0 8 15
while (Q not empty) 3 8
3
u = ExtractMin(Q);
for each v ∈ Adj[u]
if (v ∈ Q and w(u,v) < key[v])
p[v] = u;
key[v] = w(u,v);

47
 Minimum Spanning Tree
 Suppose in contradiction there are TWO MSTs,
M1 and M2.

 Let e be edge with minimum weight that is one

but not the other (say it is in M1).

 If e is added to M2, a cycle is formed.

 Let e' be an edge in the cycle that is not in M1

48
 Minimum Spanning Tree

e: in M1 but not M2

M2:
e': in M2 but not M1;
wt is less than wt of e

Replacing e with e' creates a new MST M3

whose weight is less than that of M2

49
 Correctness of Kruskal's Alg.
 Let e1, e2, …, en-1 be sequence of edges chosen

 Clearly they form a spanning tree

 Suppose it is not minimum weight

 Let ei be the edge where the algorithm goes wrong

 {e1,…,ei-1} is part of some MST M

 but {e1,…,ei} is not part of any MST

50
 Correctness of Kruskal's Algorithm

M: ei, forms a cycle in M

wt(e*) > wt(ei)

replacing e* w/ ei forms
e* :
a spanning tree with
min wt. edge smaller weight than M,
in cycle not in contradiction!
e1 to ei-1

white edges are part of MST M, which contains e1 to ei-1,

but not ei

51
 Barůvka‘s Algorithm
1. For all vertices search the edge with the smallest weight
of this vertex and mark these edges

2. Search connected vertices (clusters) and replace them

by
a “new“ vertex C (cluster)
i

3. Remove the cycles and, if two vertices are connected by

more than one edge, delete all edges except the
“cheapest“
 Barůvka‘s Algorithm

5
A B
4 6 2

2 D 3
C

3 1 2
E F
4
 Barůvka‘s Algorithm

minimum- spanning tree

A B
2

2 D
C

3 1 2
E F
The correctness of Barůvka‘s Algorithm

Crucial Fact about MSTs

Running time: O ( m log n )

The number of edges is at least reduced by half in

each step.
Number of steps: O ( log n )
 Kruskal‘s, Prim‘s and Borůvka‘s
Algorithms: A Comparison
Although each of the above algorithms has the
same worth-case running time, each one achieves
this running time using different data structures
and different approaches to build the MST.

there is no clear winner among these three

algorithms
 Minimum Spanning Trees
Kruskal’s and Boruvka’s have better running times if the
number of edges is low, while Prim’s has a better
running time if both the number of edges and the number
of nodes are low.

Boruvka’s avoids the complicated data structures needed

for the other two algorithms.

So, of course, the best algorithm depends on the graph

and if you want to bear the cost of complex data
structures.

57
 Minimum Spanning Trees

The best algorithm that I know of is a hybrid of Boruvka’s

and Prim’s, which I did not examine here.

It does O(log log n) passes of Boruvka’s and then

switches to Prim’s, resulting in a running time of O(m log
log n).

So, it’s the fastest algorithm, but would, of course, require

the Fibonacci heap for Prim’s which Boruvka’s avoids
when used by itself.

However, in order to keep things simple, I did not explore

it here.
58
 Knapsack Problem
 Given: A set S of n items, with each item i having

 bi - a positive benefit, wi - a positive weight

 Goal: Choose items with maximum total benefit but with weight at
most W.

◦ In this case, we let T denote the set of items we take

◦ Objective: maximize
∑b
i∈T
i

◦ Constraint: ∑w
i∈T
i ≤W

59
 Knapsack Problem
 Knapsack Problem: Given n items, with i th item worth bi
dollars and weighing wi pounds, a thief wants to take as valuable
a load as possible, but can carry at most W pounds in his
knapsack.
 Given: A set S of n items, with each item i having

 bi - a positive benefit, wi - a positive weight

 Goal: Choose items with maximum total benefit but with weight at
most W.

 If we are not allowed to take fractional amounts, then this is the 0/1
knapsack problem (Each item is either taken or not taken)
60
 Knapsack Problem: Example
 Given: A set S of n items, with each item i having
◦ bi - a positive benefit, wi - a positive weight
 Goal: Choose items with maximum total benefit but with
weight at most W.
“knapsack”

Solution:
Items:
• 5 (2 in)
1 2 3 4 5
• 3 (2 in)
• 1 (4 in)
Weight: 4 in 2 in 2 in 6 in 2 in
Benefit: 9 in
$20 $3 $6 $25 $80

61
 Knapsack Problem
Knapsack problem.
◦ Given n objects and a "knapsack."
◦ Item i weighs wi > 0 kilograms and has value vi > 0.
◦ Knapsack has capacity of W kilograms.
◦ Goal: fill knapsack so as to maximize total value.
Item Value Weight
Ex: { 3, 4 } has value 40. 1 1 1
2 6 2
W = 11
3 18 5
4 22 6
5 28 7

Greedy: repeatedly add item with maximum ratio vi / wi.

Ex: { 5, 2, 1 } achieves only value = 35 ⇒ greedy not optimal.

62 147
 Review: Greedy Algorithm

 Idea: When we have a choice to make, make the one that

looks best right now
◦ Make a locally optimal choice in hope of getting a
globally optimal solution
◦ a decision is made that appears to be good, without
regard for future consequences
 Greedy algorithms don’t always yield an optimal solution
 Makes the choice that looks best at the moment in order to
get optimal solution
 Review: Greedy Algorithms
 In order for greedy heuristic to solve the problem, it
must be that the optimal solution to the big problem
contains optimal solutions to sub-problems.
 Even when greedy algorithms do not produce the
optimal solution, they often provide fast heuristics (non-
optimal solution strategies), and are often used in finding
good approximations.
 Examples of Greedy Algorithms
Activity Selection, Minimum Spanning Tree, Shortest
Path, Huffman Codes, Fractional Knapsack
64
 Drawback of Divide & Conquer
 Sometimes can be inefficient

 Example: Fibonacci numbers

Sequence is 0, 1, 1, 2, 3, 5, 8, 13, …

F0 = 0, F1 = 1, Fn = Fn-1 + Fn-2 for n > 1

Obvious recursive algorithm:

Fib(n):

if n = 0 or 1 then return n

else return (Fib(n-1) + Fib(n-2))

65
 Computing Fibonacci Numbers
Recursion Tree for Fib(5)
Computing the nth Fibonacci
number recursively (top-down): Fib(5)

Fib(4) Fib(3)

Fib(3) Fib(2) Fib(2) Fib(1)

Fib(2) Fib(1) Fib(1) Fib(0) Fib(1) Fib(0)

Fib(1) Fib(0)

66
 How Many Recursive Calls?
• Run time: T(n) = T(n-1) + T(n-2)

• If all leaves had the same depth, then there would be about 2n
recursive calls.

• The run time doubles as n grows and is order O(2n).

• Our recursion tree has only 0s and 1s as leaves, thus we have

1.6n summations

• However with more careful counting it can be shown that it

is Ω((1.6)n)

• Running time is exponential!

67
 Save Work
 Wasteful approach - repeat work unnecessarily
 For our example: Fib(2) is computed three times

 Recursion adds overhead

 extra time for function calls
 extra space to store information on the runtime stack
about each currently active function call
 Avoid the recursion overhead by filling in the table
entries bottom up, instead of top down.
 For our example: Instead, compute Fib(2) once, store result in
a table, and access it when needed

68
 More Efficient Recursive Algorithm
F[0] := 0; F[1] := 1; F[n] := Fib(n);
Fib(n):
 if n = 0 or 1 then return F[n]
 if F[n-1] = NIL then F[n-1] := Fib(n-1)
 if F[n-2] = NIL then F[n-2] := Fib(n-2)
 return (F[n-1] + F[n-2])

computes each F[i] only once

69
 Automated Memoization
Automated memoization. Many functional programming languages
(e.g., Lisp) have built-in support for memoization.
Q. Why not in imperative languages (e.g., Java)?

(defun F (n) static int F(int n) {

(if if (n <= 1) return n;
(<= n 1) else return F(n-1) + F(n-
2);
n }
(+ (F (- n 1)) (F (- n 2)))))
Java (exponential)
Lisp (efficient)

F(40)

F(39) F(38)

F(38) F(37) F(37) F(36)

F(37) F(36) F(36) F(35) F(36) F(35) F(35) F(34)

70
 Example of Memoized Fib

F fills in F[4] with 3,

0 0 Fib(5) returns 3+2 = 5

1 1
fills in F[3] with 2,
2 1
NIL Fib(4) returns 2+1 = 3

3 2
NIL

4 3
NIL
Fib(3)
fills in F[2] with 1,
returns 1+1 = 2
5 5
NIL

Fib(2) returns 0+1 = 1

71
Dynamic Programming for Fibonacci
DP Solution for Fibonacci

Fib(n):
 F[0] := 0; F[1] := 1;
 for i := 2 to n do
 F[i] := F[i-1] + F[i-2]

◦ return F[n]
Can perform application-specific
optimizations
 e.g., save space by only keeping last
two numbers computed
72
Computing a Binomial Coefficient by DP
Binomial coefficients are coefficients of the binomial formula:
(a + b)n = C(n,0)anb0 + . . . + C(n,k)an-kbk + . . . + C(n,n)a0bn

Recurrence: C (n ,k ) = C (n-1,k ) + C (n -1,k -1) for n > k > 0

C (n ,0) = 1, C (n ,n ) = 1 for n ≥ 0

Value of C (n ,k ) can be computed by filling a table:

0 1 2 . . . k -1 k
0 1
1 1 1
.
.
.
n-1 C (n-1,k -1) C (n-1,k )
n C (n ,k )

73
 The General Dynamic Programming
Technique
 Applies to a problem that at first seems to require a lot of
time (possibly exponential), provided we have:
 Simple subproblems: the subproblems can be defined in
terms of a few variables, such as j, k, l, m, and so on.
 Subproblem optimality: the global optimum value can
be defined in terms of optimal subproblems
 Subproblem overlap: the subproblems are not
independent, but instead they overlap (hence, should be
constructed bottom-up).
74
 Dynamic Programming

 set up a recurrence relating a solution to a larger

instance to solutions of some smaller instances

 solve smaller instances once

 record solutions in a table

 extract solution to the initial instance from that

table

75 160
 Dynamic Programming

 Greedy algorithms v.s. dynamic programming

◦ Common:
 optimal substructure
◦ Difference:
 greedy-choice property
◦ Dynamic programming can be used
 if greedy solutions are not optimal

76 161
 Computing C(n, k)
Pseudocode and Analysis

Time efficiency: HW
Space efficiency: HW
77
 References
 Cormen, C. Leiserson, R. Rivest, and C. Stein,
Introduction to Algorithms, MIT Press and
McGraw-Hill Book Company, 2009
 Anany Levitin, Introduction to The Design &
Analysis of Algorithms.,Addison-Wesley , 2002
 J. Kleinberg, E. Tardos, Algorithm Design,
Addison-Wesley, 2005.

78