Greedy algorithm

~ Ravi Sheth
 Greedy Approach
• Also for optimization problem
• Best choice at any moment
It makes a locally optimal choice in hopes of achieving a
globally optimal solution.
• Do not always yield optimal solutions, but
for many problems they do
• Examples: Activity selection problem, Fractional
knapsack problem, Huffman coding.
Activity-selection Problem
 Activity-Selection Problem
i 1 2 3 4 5 6 7 8 9 10 11
start_timei 1 3 0 5 3 5 6 8 8 2 12
finish_timei 4 5 6 7 8 9 10 11 12 13 14
time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11
0
1
2
3
4 Compatible activities:
5 {a3, a9, a11},
6
7 {a1,a4,a8,a11},
8 {a2,a4,a9,a11}
9
10
11
12
13
14
 Activity-Selection Problem
Greedy Strategy Solution time a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12

0
Hence, to solve the Si,j: 1
2
3
1. Choose the activity am with the earliest 4
5
6
finish time.(starting time should not 7
8
overlap with Finishing time) 9
10
11
12
2. Solution of Si,j = {am} U Solution of 13
14
subproblem Sm,j

That is,
To solve S0,12, we select a1 that will finish earliest, and solve for S1,12.
To solve S1,12, we select a4 that will finish earliest, and solve for S4,12.
To solve S4,12, we select a8 that will finish earliest, and solve for S8,12.
To solve S8,12, we select a11 that will finish earliest
 Activity-Selection Problem
Greedy Strategy Solution
Recursive-Activity-Selector(i,j)
1 m = i+1
// Find first activity in Si,j m=2 m=3 m=4
2 while m < j and start_timem <= finish_timei Okay Okay break
3 do m = m + 1 the loop
4 if m <= j
5 then return {am} U Recursive-Activity-Selector(m,j)
6 else return Ø time a a a a a a a a a a a a a
0 1 2 3 4 5 6 7 8 9 10 11 12

Order of calls: 0
{1,4,8,11}
1
2
Recursive-Activity-Selector(0,11) 3
{4,8,11}
4
5
Recursive-Activity-Selector(1,11) 6
{8,11}
7
8
Recursive-Activity-Selector(4,11) 9
{11}
10
11
Recursive-Activity-Selector(8,11) 12
13
Ø Recursive-Activity-Selector(11,11) 14


 Elements of Greedy Strategy
1. Determine the optimal substructure of the problem.
2. Develop a recursive solution.
3. Show that if we make the greedy choice, then only one
sub problem remains.
4. Prove that it is always safe to make the greedy choice.
5. Develop a recursive algorithm that implements the greedy
strategy.
 Knapsack Problem
• A thief robbing a store finds n items. The i th item is worth vi dollars
and weighs wi pounds, where vi and wi are integers. The thief wants
to take as valuable a load as possible, but he can carry at most W
pounds in his napsack, for some integer W.

•
• 0-1 knapsack problem:
– How to fill the knapsack with best total value w/o breaking each
item
• Fractional knapsack problem:
– How to fill the knapsack with best total value
• we can solve the fractional knapsack problem by a greedy strategy,
but we cannot solve the 0-1 problem by such a strategy. To solve the
fractional problem, we first compute the vi/wi for each item.
• To see that this greedy strategy does not work for the 0-1 knapsack
problem, consider the problem instance illustrated in Figure (a).
• This example has 3 items and a knapsack that can hold 50pounds.
Item1 weighs 10pounds and is worth 60dollars. Item 2 weighs
20pounds and is worth100 dollars. Item3 weighs 30 pounds and is
worth 120dollars.
• Thus, the value per pound of item1 is 6dollars per pound, which is
greater than the value per pound of either item2 (5dollars per pound)
or item3 (4dollars per pound).
 Solution
• DP for 0-1 knapsack
• Greedy for fractional knapsack
– Compute value per pound (VPP) for each item
– Take as much as possible of the item with
largest VPP
– Continue till reach weight limit
 Huffman Codes
Huffman Codes
• For compressing data (sequence of characters)
• Widely used
• Very efficient (saving 20-90%)
• Use a table to keep frequencies of occurrence
of characters.
• Output binary string.

“Today’s “001 0110 0 0 100

weather is nice” 1000 1110”
 Huffman Codes
Example: Frequency Fixed-length Variable-length
codeword codeword
‘a’ 45000 000 0
‘b’ 13000 001 101
A file of 100,000 characters. ‘c’ 12000 010 100
Containing only ‘a’ to ‘f’
‘d’ 16000 011 111
‘e’ 9000 100 1101
‘f’ 5000 101 1100

eg. “abc” = “000001010” eg. “abc” = “0101100”

1*45000 + 3*13000 + 3*12000 +

300,000 bits 3*16000 + 4*9000 + 4*5000
= 224,000 bits
 Huffman Codes A file of 100,000
characters.
The coding schemes can be represented by trees:
Frequency Fixed-length Frequency Variable-length
(in thousands) codeword (in thousands) codeword
‘a’ 45 000 ‘a’ 45 0
‘b’ 13 001 ‘b’ 13 101
‘c’ 12 010 ‘c’ 12 001
‘d’ 16 011 ‘d’ 16 111
‘e’ 9 100 ‘e’ 9 1011
‘f’ 5 101 ‘f’ 5 0011

Not a full 0
100 100 A full binary tree
1 0 1 every nonleaf node
binary tree 86 14 a:45 55 has 2 children
0 1 0 0 1
58 28 14 25 30
0 1 0 1 0 1 0 1 0 1
a:45 b:13 c:12 d:16 e:9 f:5 c:12 b:13 14 d:16
0 1
f:5 e:9
 Huffman Codes
How do we find the Huffman code (1952) was invented to solve it.
optimal prefix code? A Greedy Approach.

Q: A min-priority queue f:5 e:9 c:12 b:13 d:16 a:45

c:12 b:13 14 d:16 a:45 14 d:16 25 a:45

f:5 e:9 f:5 e:9 c:12 b:13

25 30 a:45 a:45 55 100

a:45 55
c:12 b:13 14 d:16 25 30
25 30
f:5 e:9 c:12 b:13 14 d:16
b:13 c:12 14 d:16
f:5 e:9
e:9 f:5
 Huffman Codes
Q: A min-priority queue f:5 e:9 c:12 b:13 d:16 a:45

c:12 b:13 14 d:16 a:45 14 d:16 25 a:45

f:5 e:9 f:5 e:9 c:12 b:13

….

HUFFMAN(C)
1 n=C and Build Q from C (Q=C) // initialize queue Q with the characters in C
2 For i = 1 to |n-1 Repeatedly extracts two
3 Allocate a new node z nodes x and y of lowest
4 z.left = x = EXTRACT_MIN(Q) frequency from the queue
5 z.right = y = EXTRACT_MIN(Q)
6 z.freq = x.freq + y.freq // sum of x and y
7 Insert(Q,z)Insert z into Q in correct position.
8 Return EXTRACT_MIN(Q) // returns the one node left in the queue
 Greedy Algorithms
Summary
Casual Introduction: Two Knapsack Problems
An Activity-Selection Problem
Greedy Algorithm Design
Steps of Greedy Algorithm Design
Optimal Substructure Property
Greedy-Choice Property
Comparison with Dynamic Programming
Huffman Codes
Backtracking
 Backtracking
• Suppose you have to make a series of
decisions, among various choices, where
– You don’t have enough information to know
what to choose
– Each decision leads to a new set of choices
– Some sequence of choices (possibly more than
one) may be a solution to your problem
• Backtracking is a methodical way of trying
out various sequences of decisions, until
you find one that “works”
19
 Solving a maze/Network
• Given a maze, find a path from start to finish
• At each intersection, you have to decide
between three or fewer choices:
– Go straight
– Go left
– Go right
• You don’t have enough information to choose correctly
• Each choice leads to another set of choices
• One or more sequences of choices may (or may not) lead to a
solution
• Many types of maze problem can be solved with backtracking
20
 Backtracking (animation)
dead end
?
dead end
dead end

?
start ? ?
dead end
dead end
?

success!

21
 Terminology I
A tree is composed of nodes

There are three kinds of

nodes:

The (one) root node

Backtracking can be thought of as
Internal nodes
searching a tree for a particular
Leaf nodes “goal” leaf node

22
 Terminology II
• Each non-leaf node in a tree is a parent of one
or more other nodes (its children)
• Each node in the tree, other than the root, has
exactly one parent
parent
Usually, however, we
draw our trees
downward, with the
parent root at the top

children children

23
 The backtracking algorithm
• Backtracking is really quite simple--we
“explore” each node, as follows:
• To “explore” node N:
1. If N is a goal node, return “success”
2. If N is a leaf node, return “failure”
3. For each child C of N,
3.1. Explore C
3.1.1. If C was successful, return “success”
4. Return “failure”
24
 Heuristic Search Methods
(Informed search)
Methods that use a heuristic function to
provide specific knowledge about the
problem:

Heuristic Functions
 Heuristic Functions
• A Heuristic is a function that, when applied
to a state, returns a number that is an
estimate of the merit of the state, with
respect to the goal.

• In other words, the heuristic tells us

approximately how far the state is from the
goal state.

26
 8-puzzle
• Goal State:

1 2 3

8 4

7 6 5
 Examples (2): 8-puzzle
• f1(T) = the number correctly placed tiles on
the board: 1 3 2
f1 8 4 =4
5 6 7

 f2(T) = number or incorrectly placed tiles on board:

gives (rough!) estimate of how far we are from goal
1 3 2
f2 8 4 =4
5 6 7

Most often, ‘distance to goal’ heuristics are more useful !

28
 Example

1 2 3 1 2 3

4 8 4 5 6

7 6 5 7 8

Current State Goal State

• Heuristic ?????
• f2(T) = number or incorrectly placed tiles on
board:

1 2 3
4 8
F2(T)= 4
7 6 5