1

Chapter 4. The Greedy Approach

p Chapter 4 introduces an algorithm

design technique called “Greedy
Approach”.

CHAPTER 4
Foundations of Algorithms
 2
Greedy Algorithm
 Greedy Algorithm
arrives at a solution by making a sequence of choices,
each of which simply looks the best at the moment
- Once each choice has been made and added to a
partial solution, it will always be in the solution set.
- Each choice is locally optimal, but not necessarily
globally optimal.
 3
Greedy Algorithm
 Example: Giving Change for a Purchase
Solution: a smallest number of bills/coins totaling the
amount of change
\7,870 : one \5,000 bill,
two \1,000 bills,
one \500 coin,
three \100 coins,
one \50 coin,
two \10 coins.
 4
Greedy Algorithm
 Example: Giving Change for a Purchase
while (there are more bills/coins and the instance is not solved)
{
grab the largest remaining bill; //selection procedure
if (adding the bill/coin makes the change exceed the amount owed)
// feasibility check
reject the bill/coin ;
else
add the bill/coin to the change ;
if (the total value of the change equals the amount owed)
// solution check
the instance is solved ;
}
 5
Greedy Algorithm
 General Greedy Approach
- starts with an empty set, and adds items to the set in sequence
until the set represents a solution to an instance of a problem
- each iteration consists of the following components:
1. Selection Procedure
chooses the next item to add to the set according to a greedy criteria
2. Feasibility Check
checks whether it is possible to complete the set in such a way
as to give a solution to the instance.
3. Solution Check
determines whether the new set constitutes a solution to the instance.
 6
4.1 Minimum Spanning Trees
 Definitions
 Undirected Graph
- a graph where its edges do not have directions
 Connected Graph
- an undirected graph where there is a path between
every pair of vertices
 Acyclic Graph
- a graph with no cycles
 Tree
- an acyclic, connected, undirected graph
 7
4.1 Minimum Spanning Trees
 Definitions
 Spanning Tree for undirected graph
- a connected subgraph that contains all the vertices in G
and is a tree
 Minimum Spanning Tree for undirected graph
- a spanning tree of G with minimum weight
 8
4.1 Minimum Spanning Trees
 Examples of Graph
1 1 G4,G5:
V1 V2 V1 V2
G1: 3 G2: minimum
3 6 6
4 4 spanning trees
V3 V4 V3 V4
2 5 2 5
of G1
V5 V5
1 1 1
V1 V2 V1 V2 V1 V2
G3: 3 G4: G5: 3
6 3
4 4
V3 V4 V3 V4 V3 V4
5 2 2
V5 V5 V5
 9
4.1 Minimum Spanning Trees
4.1.1 Prim’s Algorithm
F=Ø;
Y = {v1}
while (the instance is not solved)
{
select a vertex in V-Y that is nearest to Y ;

add the vertex to Y ;

add the edge to F ;

if (Y==V)
the instance is solved ;
}
 10
4.1 Minimum Spanning Trees
4.1.1 Prim’s Algorithm
1 Y V1 1 Y V1 1
V1 V2 V2 V2
3 3 6 3 3 6 3 3 6
4 4 4
V3 V4 V3 V4 V3 V4
2 5 2 5 2 5
V5 V5 V5
Y V1 1 Y V1 1 Y V1 1
V2 V2 V2
3 3 6 3 3 6 3 3 6
4 4 4
V3 V4 V3 V4 V3 V4
2 5 2 5 2 5
V5 V5 V5
 11
4.1 Minimum Spanning Trees
4.1.1 Prim’s Algorithm
- To find the nearest vertex from Y, we use the following
arrays:
W[i][j] : Weight table (symmetric)
nearest[i] = index of the vertex in Y nearest to Vi
distance[i] = weight on edge between Vi and the
vertex indexed by nearest[i]
- nearest[i] is initialized to 1
- distance[i] is initialized to W[1][i]
- as vertices are added to Y, two arrays are updated to
reference the new vertex in Y nearest to each vertex outside Y
 12
4.1 Minimum Spanning Trees
4.1.1 Prim’s Algorithm
public static set_of_edges prim(
int n, const number W[][]) min = ∞ ;
{ for (i=2; i<=n; i++)
index i, vnear; if (0<= distance[i] < min) {
index [ ] nearest = new index[2..n] ; min = distance[i] ; vnear = i ;
number min; }
number[ ] distance = new number[2..n] ; e = edge connecting vertices indexed
edge e;
by vnear and nearest[vnear] ;
set_of_edges F = Ø;
add e to F ;
for (i=2; i<=n; i++) { distance[vnear] = -1 ;
nearest[i]= 1; for (i=2; i<=n; i++)
distance[i] = W[1][i]; if (W[i][vnear] < distance[i]) {
} distance[i] = W[i][vnear] ;
repeat( n-1 times ) { … } nearest[i] = vnear ;
return F ;
}
}
 13
4.1 Minimum Spanning Trees
 Time Complexity of Prim’s Algorithm
Basic Operation:
if-statement inside two for-i loops
Input Size:
n, the number of vertices

 T(n) = 2*(n-1)*(n-1) ∈ Θ(n2)

 14
4.1 Minimum Spanning Trees
 Does Prim’s Algorithm always produce an
optimal solution?
- Although greedy algorithms are often easier to develop
than dynamic programming algorithms, usually it is more
difficult to determine whether or not a greedy algorithm
always produces an optimal solution.

 For a greedy algorithm, we usually need a formal

proof to show that it actually does.
 15
4.1 Minimum Spanning Trees
 Optimality of Prim’s Algorithm
- We will use the concept of “a promising set of edges” to
prove the optimality of Prim’s Algorithm.
 Definition
- A subset F of E is called promising if edges can be
added to it so as to form a minimum spanning tree.
 Proof:
- Will show by induction that the set F is promising
after each iteration of the repeat loop.
 16
4.1 Minimum Spanning Trees
 Example:
1(e1)
V1 V2 1. F = { }  promising
G: 3(e2) 3(e3) 6(e4) 2. F = {e4}  Not promising
4(e5)
V3 V4 3. F = {e1,e2}  promising
2(e6) 5(e7)
V5 4. F = {e1,e2,e3 }  Not promising
V1
1
V2 V1
1
V2 5. F = {e1,e2,e5}  promising
3 3 6 3 3 6 6. F = {e1,e2,e5,e6}  promising
4 4
V3 V4 V3 V4
2 5 2 5
V5 V5
2 minimum spanning trees of G
 17
4.1 Minimum Spanning Trees

 Proof of the Optimality of Prim’s Algorithm

 Induction Basis
Obviously, the empty set is promising.
 Induction Hypothesis

Assume that, after a given iteration of the repeat loop, the

set of edges so far selected, namely F, is promising.
 Induction Step
We need to show that F ∪{e} is promising, where e
is an edge of minimum weight that connects a vertex
in Y to a vertex in V-Y.
 18
4.1 Minimum Spanning Trees

 Proof of the Optimality of Prim’s Algorithm

Induction Step - continued
Because F is promising, there must be some set of
edges F′ such that F ⊆ F′ and (V,F′) is a minimum
spanning tree.
Case 1: e ∈ F′
F ∪{e} ⊆ F′ , so F ∪{e} is promising. Done.

e e
F F ∪{e} F′
 19
4.1 Minimum Spanning Trees

 Proof of the Optimality of Prim’s Algorithm

Induction Step - continued
Case 2: e ∉ F′
Because (V,F′) is a spanning tree, F′ ∪{e} must contain
exactly one cycle and e must be in the cycle. Thus, there
must be another edge e′∈ F′ in the cycle that also
connects a vertex in Y to one in V-Y.
If we remove e′ from F′ ∪{e}, the cycle disappears,
which means we have a spanning tree. Because e is an
edge of minimum weight that connects a vertex in Y to
one in V-Y, the weight of e must be less than or equal to
that of e′ (in fact they must be equal).
 20
4.1 Minimum Spanning Trees

 Proof of the Optimality of Prim’s Algorithm

Induction Step
Case 2: e ∉ F′ - continued
So, F′ ∪{e} – {e′} is a minimum spanning tree.
Therefore, F ∪{e} ⊆ F′ ∪{e} – {e′},
which means F ∪{e} is promising.

e e e
e′
F F ∪{e} F′ F′∪{e} F′∪{e} – {e′}
 21
4.1 Minimum Spanning Trees
4.1.2 Kruskal’s Algorithm
1. F = Ø ;
2. create disjoint subsets of V, one for each vertex and containing only
that vertex;
3. sort the edges in E in non-decreasing order ;
4. while (the instance is not solved) {
select next edge from the sorted list
if (the edge connects two vertices in disjoint subsets) {
merge the subsets ;
add the edge to F ;
}
if (all the subsets are merged)
the instance is solved ;
}
 22
4.1 Minimum Spanning Trees
4.1.2 Kruskal’s Algorithm
 Example:
1. Sorted Edges:
1 (v1,v2) 1
V1 V2
3 3 6 (v3,v5) 2
4 (v1,v3) 3
V3 V4
2 5 (v2,v3) 3
V5
(v3,v4) 4
(v4,v5) 5
(v2,v4) 6
 23
4.1 Minimum Spanning Trees
4.1.2 Kruskal’s Algorithm
1 1 1
Sorted Edges V1 V2 V1 V2 V1 V2
3 3 3 3 6
(v1,v2) 1 3 6 3 6
4 4 4
(v3,v5) 2 V3 V4 V3 V4 V3 V4
2 5 2 5 2 5
(v1,v3) 3 V5 V5 V5
(v2,v3) 3 1 1 V1
1
V2
V1 V2 V1 V2
(v3,v4) 4 3 3 6 3 3 6 3 3 6
(v4,v5) 5 V3 4
V4 V3 4
V4 V3 4
V4
(v2,v4) 6 2 5 2 5 2 5
V5 V5 V5
 24
4.1 Minimum Spanning Trees
4.1.2 Kruskal’s Algorithm
- To write a formal version of Kruskal’s Algorithm, we use
the following:
- Initial(n) initializes n disjoint subsets, each of which
contains exactly one of the indices between 1 and n
- p= find(i) makes p point to the set containing index i
- merge(p,q) merges the two sets, to which p and q
point, into a set.
- equal(p,q) returns true if both p and q point to the
same set.
 25
4.1 Minimum Spanning Trees
4.1.2 Kruskal’s Algorithm
public static set_of_edges kruskal(
int n, int m, set_of_edges E) {
index i, j;
e = edge with the least weight
set_pointer p,q ;
not yet considered ;
edge e;
i,j = indices of vertices connected by e;
set_of_edges F = Ø ;
p = find(i) ;
Sort the m edges in E by weight
q = find(j) ;
in nondecreasing order;
if (! Equal(p,q)) {
initial(n) ;
merge(p,q) ;
while( |F| < n-1)
add e to F ;
{ }
…
return F ;
}
}
 26
4.1 Minimum Spanning Trees

 Time Complexity of Kruskal’s Algorithm

Basic Operation:
A comparison operation
Input Size:
n, the number of vertices
m, the number of edges
1. Time to sort the edges:
W(m) ∈ Θ(m lg m) using MergeSort
2. Time in the while loop:
W(m) ∈ Θ(m lg m) using DisjointSet implementation
 27
4.1 Minimum Spanning Trees

 Time Complexity of Kruskal’s Algorithm

3. Time to initialize n disjoint sets
T(n) ∈ Θ(n)

 W(m,n) ∈ Θ(m lg m) where n-1 ≤ m ≤ n(n-1)/2.

 Thus, in the worst case, we have
W(m,n)∈Θ(m lg m)= Θ(n2 lg n2)=Θ(n2 lg n).
 28
4.1 Minimum Spanning Trees

 Optimality of Kruskal’s Algorithm

- Similar to the proof of Prim’s Algorithm
- Will use the concept of “a promising set of edges” to
prove the optimality
 29
4.1 Minimum Spanning Trees

 Proof of the Optimality of Kruskal’s Algorithm

 Induction Basis
Obviously, the empty set is promising.
 Induction Hypothesis

Assume that, after a given iteration of the repeat loop, the

set of edges so far selected, namely F, is promising.
 Induction Step
We need to show that F ∪{e} is promising, where e is
an edge of minimum weight in E-F such that F ∪{e}
has no cycles.
 30
4.1 Minimum Spanning Trees

 Proof of the Optimality of Kruskal’s Algorithm

Induction Step - continued
Because F is promising, there must be some set of
edges F′ such that F ⊆ F′ and (V,F′) is a minimum
spanning tree.
Case 1: e ∈ F′
F ∪{e} ⊆ F′, so F ∪{e} is promising. Done.

e e

F F ∪{e} F′
 31
4.1 Minimum Spanning Trees

 Proof of the Optimality of Kruskal’s Algorithm

Induction Step - continued
Case 2: e ∉ F′
Because (V,F′) is a spanning tree, F′ ∪{e} must contain
exactly one cycle and e must be in the cycle. Thus, there
must be another edge e′∈ F′ in the cycle that is not in F.
That is, e′∈ E – F.
If we remove e′ from F′ ∪{e}, the cycle disappears,
which means we have a spanning tree. Because e is an
edge of minimum weight that is not in F, the weight of e
must be less than or equal to that of e′ (in fact they must
be equal).
 32
4.1 Minimum Spanning Trees

 Proof of the Optimality of Kruskal’s Algorithm

Induction Step
Case 2: e ∉ F′ - continued
So, F′ ∪{e} – {e′} is a minimum spanning tree.
Therefore, F ∪{e} ⊆ F′ ∪{e} – {e′},
which means F ∪{e} is promising.

e e e
e′ e′

F F ∪{e} F′ F′∪{e} F′∪{e} – {e′}

 33
4.1 Minimum Spanning Trees

4.1.3. Prim’s Algorithm vs. Kruskal’s Algorithm

 Prim’s Algorithm
T(n) ∈ Θ(n2)
 Kruskal’s Algorithm

W(m,n) ∈ Θ(m lg m) where n-1 ≤ m ≤ n(n-1)/2.

 1. If m is close to n-1, i.e., the graph is sparse,
then Θ(m lg m) becomes Θ(n lg n), which is better than Θ(n2).
 2. If m is close to n(n-1)/2, i.e., the graph is dense,
then Θ(m lg m) becomes Θ(n2 lg n), which is worse than Θ(n2).
 34
4.4 Huffman Code

 Huffman Code
- an efficient encoding method for data compression
- uses a variable-length binary code to represent a text file
Note:
 Variable-Length Binary Code

- represents different characters using different numbers

of bits
 Fixed-Length Binary Code

- represents each character using the same number

of bits
 35
4.4 Huffman Code

 Fixed-Length vs. Variable-Length Binary Code

Example: Character Set { a, b, c },
String “ababcbbbc”
 Fixed-Length Binary Code

 a: 00, b: 01, c: 11
ababcbbbc: 000100011101010111 - 18 bits
 Variable-Length Binary Code
 a: 10, b: 0, c: 11
ababcbbbc: 1001001100011 - 13 bits
 36
4.4 Huffman Code

4.4.1 Prefix Code

- a variable-length code in which no codeword for
one character constitutes the beginning of the codeword
for another character
 Example:
 a: 10, b: 0, c: 11
0 1
b
0 1
a c
 37
4.4 Huffman Code

p The number of bits to encode a text file

p Given the binary tree T corresponding to some
prefix code

Minimize ∑ frequency(v) * depth(v)

0 1
frequency(b) * 1 b
+ frequency(a) * 2 0 1
+ frequency(c) * 2 a c
 38
4.4 Huffman Code

Character Frequency Code 1 Code 2 Code 3

(Fixed) (Huffman)
a 16 000 10 00
b 5 001 11110 1110
c 12 010 1110 110
d 17 011 110 01
e 10 100 11111 1111
f 25 101 0 10

∑ frequency(v) * depth(v) = 255 231 212

 39
4.4 Huffman Code
Code 3
Code 2
0 1
0 1
f:25 0 1
0 1 0 1
a:16 f:25
a:16 d:17 0 1
0 1
d:17 c:12
0 1
0 1
b:5 e:10
c:12
0 1
b:5 e:10
 40
4.4 Huffman Code

4.4.2 Huffman’s Algorithm

Regard characters as a forest with n single-node trees

repeat
merge two trees with least frequencies
until it becomes a single tree
 41
4.4 Huffman Code

4.4.2 Huffman’s Algorithm

Public class nodetype
for (i = 1; i < n; i++) { {
remove(PQ, p); char symbol;
remove(PQ, q); Min int frequency;
r = new nodetype(); priority
r.left = p; nodetype left;
r.right = q; quene
nodetype right;
r.frequency = p.frequency + q.frequency; }
insert(PQ, r);
}
remove(PQ, r); Time complexity
return r;
Θ(n log n)
 42
4.4 Huffman Code

p Example
b:5 e:10 c:12 a:16 d:17 f:25 c:12 15 a:16 d:17 f:25
0 1
b:5 e:10

a:16 d:17 f:25 27 f:25 27 33

0 1 0 1 0 1
c:12 15 c:12 15 a:16 d:17
0 1 0 1
b:5 e:10 b:5 e:10
 43
4.4 Huffman Code

p Example (cont’d) 0 1

0 0 1
33 52
1
a:16 f:25
0 1 0 1 d:17 0 1
a:16 d:17 f:25 27
c:12
0 1 0 1
c:12 15 b:5 e:10
0 1
b:5 e:10
 44
4.4 Huffman Code
p Proof of the Optimality of Huffman’s Algorithm
p Induction Basis
pThe set of single nodes in the 0th step
 branches in an optimal prefix binary tree
p Induction Hypothesis
pThe set of trees in the ith step
 branches in an optimal prefix binary tree T
p Induction Step
p u & v: roots of trees combined in the (i+1)th step
p NEXT PAGE..
 45
4.4 Huffman Code

p Proof of the Optimality of Huffman’s Algorithm

pInduction Step – continued
Case 1: in T
u v
Done.
Case 2: parent of u ≠parent of v in T
WLOG, depth(u) ≥ depth(v) in T
There exists w such that or in T

w u u w
 46
4.4 Huffman Code

p Proof of the Optimality of Huffman’s Algorithm

p Induction Step – continued
frequency(w) ≥ frequency(v) – why?
depth(w) ≥ depth(v) in T
Create a new tree T’ by swapping the positions of the branches
rooted at v & w.
cost(T’) = cost(T) + (depth(w) – depth(v)) *
(frequency(v) – frequency(w)) ≤ cost(T)
v
Hence, T’ is optimal.
u w
 47
4.5 Knapsack Problem

 A Greedy Approach to the 0-1 Knapsack Problem

 The 0-1 Knapsack Problem
Given n items, let
S = { item1 ,item2, … , itemn}
wi = weight of itemi
pi = profit of itemi
W = maximum weight the knapsack can hold,
where wi, pi, and W are positive integers.
Determine a subset A of S such that
∑ pi is maximized subject to ∑ wi ≤ W.
itemi∈A itemi∈A
 48
4.5 Knapsack Problem

 A Greedy Approach to the 0-1 Knapsack Problem

 A Brute-Force Solution to the 0-1 Knapsack Problem
1. Consider all possible subsets of S.
2. Discard those subsets whose total weight > W.
3. Of those remaining, take one with the max total profit.
 Since there are 2n subsets containing up to n items,
the complexity is exponential in n.
 49
4.5 Knapsack Problem

 A Greedy Approach to the 0-1 Knapsack Problem

 A Simple Greedy Approach
Idea: Take items in non-increasing order according to profit.
 This approach wouldn’t work very well if the most profitable
item had a large weight in comparison with its profit.
Example:
item1 : $1 M and 25 Kg  Greedy Solution:
item2 : $0.8 M and 15 Kg {item1} (Profit : $ 1M)
item3 : $0.5 M and 10 Kg  Optimal Solution:
item4 : $0.24 M and 8 Kg {item2, item3}
maximum weight = 30 Kg (Profit : $ 1.3M)
 50
4.5 Knapsack Problem

 A Greedy Approach to the 0-1 Knapsack Problem

 Another Simple Greedy Approach
Idea: Take items in non-decreasing order according to weight.
 This approach would fail badly when the light items
have small profits compared with their weights.
Example:
item1 : $1 M and 25 Kg  Greedy Solution:
item2 : $0.8 M and 15 Kg {item3, item4} (Profit: $ 0.74M)
item3 : $0.5 M and 10 Kg  Optimal Solution:
item4 : $0.24 M and 8 Kg {item2, item3} (Profit: $ 1.3M)
maximum weight = 30 Kg
 51
4.5 Knapsack Problem

 A Greedy Approach to the 0-1 Knapsack Problem

 More Sophisticated Greedy Approach
Idea: Take items in non-increasing order according to
profit per unit weight.
Example 1:
item1 : $1 M and 25 Kg  $40000/Kg
item2 : $0.8 M and 15 Kg  $53333/Kg
item3 : $0.5 M and 10 Kg  $50000/Kg
item4 : $0.24 M and 8 Kg  $30000/Kg
maximum weight = 30 Kg
 Greedy Solution: {item2, item3} (Profit: $ 1.3M)  Optimal
 52
4.5 Knapsack Problem

 A Greedy Approach to the 0-1 Knapsack Problem

 More Sophisticated Greedy Approach
Idea: Take items in non-increasing order according to
profit per unit weight.
Example 2:
item1 : $5 M and 5 Kg  $1 M/Kg
item2 : $6 M and 10 Kg  $0.6 M/Kg
item3 : $14 M and 20 Kg  $0.7 M/Kg
maximum weight = 30 Kg
 Greedy Solution: {item1, item3} (Profit: $ 19 M)
 Optimal Solution: {item2, item3} (Profit: $ 20 M)
 53
4.5 Knapsack Problem

 A Greedy Approach to the fractional Knapsack

Problem
 If we allow a fraction of item to be put in the knapsack,
the greedy approach based on the profit per unit weight
produces an optimal solution.
 54
4.5 Knapsack Problem

Dynamic Programming Approach to the 0-1

Knapsack Problem
Let P[i][w] be the optimal profit obtained from choosing
items only from the first i items under the restriction that
the total weight cannot exceed w.

item1 item2 … itemi-1 itemi … itemn

P[i][w] = max( P[i-1][w] , pi+P[i-1][w-wi ] ) if wi ≤ w

P[i-1][w] if wi > w
 55
4.5 Knapsack Problem

Dynamic Programming Approach to the 0-1

Knapsack Problem
P[i][w] = max( P[i-1][w] , pi+P[i-1][w-wi ] ) if wi ≤ w
P[i-1][w] if wi > w
 The value we are looking for is P[n][W].
 We can determine this value using a two dimensional
array P[0..n][0..W] where
P[0][w] = 0 0≤ w ≤ W
P[i][0] = 0 0≤ i ≤ n
 56
4.5 Knapsack Problem

 Dynamic Programming Approach

P[i][w] = max( P[i-1][w] , pi+P[i-1][w-wi ] ) if wi ≤ w
P[i-1][w] if wi > w
P 0 1 2 … W-w … W
n
0 0 0 0 0 0 0 0 0
1 0  2n-1 entries:
2 0 P[1][W] , P[1][W – w1] , …
… P[1][W–wn-wn-1 …-w3 ],
i P[1][(W–wn-wn-1…-w3 )–w2 ]

n-1 ? ?  2 entries: P[n-1][W] , P[n-1][W – wn]

n 0 ?  1 entry: P[n][W]
∴ 1 + 2 + 4 + … + 2n-1 ∈ Θ(2n )