210CT PROGRAMMING ALGORITHMS AND

DATA STRUCTURES

Greedy Algorithm (Part 1)

Learning Outcome
Successful students should be able to:
▪ Explain the greedy algorithm and its main objective
▪ Use Knapsack algorithm to solve a problem
▪ Explain and use Huffman codes to reach optimal solution
Introduction : Greedy Algorithms
■ Greedy algorithms are mainly used for solving mathematical
optimization problem
■ A greedy algorithm divides a given problem into some stages.
■ The main idea is to get an locally optimal result at each stage using
heuristics (based on some knowledge/information).
■ Then use the solution of each stage as an input for the next stage
and with the hope to find the globally optimal solution
■ Usually, greedy algorithm is faster and simpler
■ Take note that a greedy algorithm doesn’t guarantee to produce an
optimal solution all the time, but it can give us a globally
approximate solution of a given problem in efficient time.
Example: Counting money
■ Suppose you want to count out a certain amount of money, using
the fewest possible bills and coins

■ A greedy algorithm would do this:

At each step, take the largest possible bill or coin that does not
overshoot
– Example: To make RM 6.39, you can choose:
■ a RM 5
■ a RM 1 , to make RM 6
■ a 20¢ coin, to make RM 6.20
■ a 10¢ coin, to make RM 6.30
■ a 5¢ coin, to make RM 6.35
■ four 1¢ coins, to make RM 6.39

■ For Ringgit Malaysia, the greedy algorithm almost always gives the
optimum solution
 A failure of the greedy algorithm
■ In some (fictional) monetary system, eg: in sweden, “krons” come in 1 kron, 7
kron, and 10 kron coins

■ Using a greedy algorithm to count out 15 krons, you would get

– A 10 kron piece
– Five 1 kron pieces, for a total of 15 krons
– This requires six coins

■ A better solution would be to use two 7 kron pieces and one 1 kron piece
– This only requires three coins

■ Doesn’t guarantee to produce an optimal solution all the time, but it can give
us a globally approximate solution of a given problem in efficient time.
5
Another approach
■ What would be the result if you ran the shortest job first?
■ Again, the running times are 3, 5, 6, 10, 11, 14, 15, 18, and 20 minutes

P1 3 10 15

P2 5 11 18

P3 6 14 20

■ That wasn’t such a good idea; time to completion is now

6 + 14 + 20 = 40 minutes

6
An optimum solution
■ Better solutions do exist:

P1 20 14

P2 18 11 5
P3 15 10 6 3

■ This solution is clearly optimal (why?)

■ Clearly, there are other optimal solutions (why?)
■ How do we find such a solution?
– One way: Try all possible assignments of jobs to
processors
– Unfortunately, this approach can take exponential time
7
Some others Greedy Algorithms
■ Factional knapsack algorithm

■ Huffman codes

■ Kruskal's MST algorithm

■ Prim's MST algorithm

■ Dijkstra's algorithm (shortest path)

 Knapsack Problem
■ There are n different items in a store

■ Item i :
– weighs wi
– worth $vi

■ A thief breaks in

■ Can carry up to W in weight in his knapsack

■ What should he take to maximize the value of his haul?

0-1 vs. Fractional Knapsack
■ 0-1 Knapsack Problem:
– the items cannot be divided
– thief must take entire item or leave it behind
– This required dynamic programming method or using
backtracking method (required testing of the all possibilities)

■ Fractional Knapsack Problem:

– thief can take partial items
– for instance, items are liquids or powders
– solvable with a greedy algorithm…
– Use in the world of e-commerce when it comes to warehousing
and shipping
Greedy Fractional Knapsack Algorithm
■ Sort items in decreasing order of value per unit weight

■ While still room in the knapsack (limit of W unit weight) do

– consider next item in sorted list
– take as much as possible (all there is or as much as will fit)

■ O(n log n) running time (for the sort)

Greedy Fractional Knapsack Algorithm
■ 3 items:
item 1 Item 2 Item 3
Profit (RM) 60 100 120
Weight (kg) 10 20 30
Profit/weigth RM 6 /kg RM5 /kg RM 4 / kg

■ knapsack can hold 50 kg. (Reminder: objective maximize the

profit)
item 1 Item 2 Item 3
Profit (RM) 60 100 120
■ greedy strategy: Weight (kg) 10 20 30
Profit/weigth RM 6 /kg RM5 /kg RM 4 / kg
Item to pick 1 (10kg) 1 (20kg) 2/3 (20kg)
Total profit RM60 + RM100 + RM4 * 20 = RM240
Huffman Codes
■ Huffman codes compress data very effectively: savings of 20%
to 90% are typical, depending on the characteristics of the data
being compressed.

■ We consider the data to be a sequence of characters.

■ Huffman’s greedy algorithm uses a table giving how often each

character occurs (i.e., its frequency) to build up an optimal way
of representing each character as a binary string.
Huffman Code

If Q is implemented as a binary min-heap,

“Build Q from C” is O(n)
“EXTRACT_MIN(Q)” is O(lg n)
“Insert z into Q” is O(lg n)
Huffman(C) is O(n lg n)
Huffman Codes
Example: Frequency Fixed-length Variable-length
codeword codeword
‘a’ 45000 000 0
A file of 100,000 characters. ‘b’ 13000 001 101
Containing only ‘a’ to ‘f’ ‘c’ 12000 010 100
‘d’ 16000 011 111
‘e’ 9000 100 1101
‘f’ 5000 101 1100

eg. “abc” = “000001010” eg. “abc” = “0101100”

1*45000 + 3*13000
1*45000 + 3*13000+ 3*12000 ++
+ 3*12000
300,000 bits Saving 3*16000 + 4*9000
3*16000 + 4*5000
+ 4*9000 + 4*5000
= 224,000 bits
= 224,000 bits
25%
Huffman Codes A file of 100,000
The coding schemes can be represented by trees: characters.
Frequency Fixed-length Frequency Variable-length
(in thousands) codeword (in thousands) codeword
‘a’ 45 000 ‘a’ 45 0
‘b’ 13 001 ‘b’ 13 101
‘c’ 12 010 ‘c’ 12 100
‘d’ 16 011 ‘d’ 16 111
‘e’ 9 100 ‘e’ 9 1101
‘f’ 5 101 ‘f’ 5 1100

Not a full
binary tree

A full binary tree

every nonleaf node
has 2 children
Huffman Codes
To find an optimal code for a file:
Frequency Codeword
‘a’ 45000 0 1. The coding must be unambiguous.
Consider codes in which no codeword is also a
‘b’ 13000 101
prefix of other codeword. => Prefix Codes
‘c’ 12000 100
‘d’ 16000 111 Prefix Codes are unambiguous.
‘e’ 9000 1101 Once the codewords are decided, it is easy to
‘f’ 5000 1100 compress (encode) and decompress (decode).

Eg. “abc” is
2. File size must be smallest.
coded as => Can be represented by a full binary tree.
“0101100” => Usually less frequent characters are at bottom
Let C be the alphabet (eg. C={‘a’,’b’,’c’,’d’,’e’,’f’})
For each character c, no. of bits to encode all c’s
occurrences = freqc*depthc
File size B(T) = cCfreqc*depthc
Huffman Code
Huffman Code