CSE 246

Design & Analysis of Algorithms

Greedy Algorithms (Part 1)

 Greedy Algorithm
• Greedy algorithms make the choice that looks
best at the moment.
• This locally optimal choice may lead to a globally
optimal solution (i.e. an optimal solution to the
entire problem).

2
When can we use Greedy algorithms?

We can use a greedy algorithm when the following are true:

1) The greedy choice property: A globally optimal solution can

be arrived at by making a locally optimal (greedy) choice.
E.g., choosing the alley with the least crowd at this moment

2) The optimal substructure property: The optimal solution

contains within its optimal solutions to subproblems.

3
 Designing Greedy Algorithms
1. Cast the optimization problem as one for which:
• we make a choice and are left with only one subproblem
to solve
2. Prove the GREEDY CHOICE
• that there is always an optimal solution to the original
problem that makes the greedy choice
3. Prove the OPTIMAL SUBSTRUCTURE:
• the greedy choice + an optimal solution to the resulting
subproblem leads to an optimal solution of the whole
problem
4
 Optimal Substructure
Find
Problem
optimal
soln. Greedy
Example - Always taking smaller
Choice
time route providing the optimal
time route from source to
Find
destination.
optimal Subproblem 1
soln.
Greedy
Choice
Find
optimal Subproblem 2
soln.
Remaining Problem
Greedy
Choice

Find
optimal Subproblem n
soln.
 Example: Making Change
• Instance: amount (in cents) to return to customer
• Problem: do this using the fewest number of
coins
• Example:
– Assume that we have an unlimited number of coins of
various denominations:
– 1c (pennies), 5c (nickels), 10c (dimes), 25c (quarters), 1$
(loonies)
– Objective: Pay out a given sum $5.64 with the
smallest number of coins possible.

6
 The Coin Changing Problem
• Assume that we have an unlimited number of coins of various
denominations:
• 1c (pennies), 5c (nickels), 10c (dimes), 25c (quarters), 1$==100c (loonies)
• Objective: Pay out a given sum S with the smallest number of
coins possible.

• The greedy coin changing algorithm:

• This is a Θ(m) algorithm where m = number of denominations.

while S > 0 do
c := value of the largest coin no larger than S;
num := S / c;
pay out num coins of value c;
S := S - num*c;

7
 Example: Making Change
• E.g.:
$5.64 = 1+1+1+1+1+
.25 + .25 + .10 +
.01 + .01 + .01 +.01

- Greedy Choice: In optimal Solution, we will always have k * P, P=maximum coin unit possible
which can be taken, k = maximum can be taken. If it contradicts -> Solution Fails
- Optimal Substructure Property: Lets, optimal solution is S= C1 * k1 + C2 * K2 +C3 * k3 where
C1 > C2 > C3 and each are coin unit, Now a subproblem is S` = S - (C1 * k1). S`’s best
solution has to be gained from (C2 * K2 +C3 * k3), Otherwise optimal substructure property
fails.

8
 Making Change – A big problem
• Example 2: Coins are valued $.30, $.20, $.05,
$.01
– Does not have greedy-choice property, since $.40 is
best made with two $.20’s, but the greedy solution will
pick three coins (which ones?)

9
 More examples
- Euro Coins (in cents) : Supports Greedy prop.
1 , 2, 5, 10, 20, 50, 100, 200
- {1, 3, 4} -> Not supports Greedy prop, ex: 6

How can you find if the greedy property is working,

- Simulate some cases, find some contradiction
that fails the greedy property.
 The Fractional Knapsack Problem
• Given: A set S of n items, with each item i having
– bi - a positive benefit
– wi - a positive weight
• Goal: Choose items with maximum total benefit but with weight at
most W.
• If we are allowed to take fractional amounts, then this is the fractional
knapsack problem.
– In this case, we let xi denote the amount we take of item i

– Objective: maximize

– Constraint:

11
 Fractional Knapsack: Greedy Intuition
As we can take fractional amount
- we can take whatever we want not contradicting the
maximum weight of an element and maximum
weight of the sack
- to fill up a single unit of sack we should try to
maximize the benefit gain
 Example
• Given: A set S of n items, with each item i having
– bi - a positive benefit
– wi - a positive weight
• Goal: Choose items with maximum total benefit but with total weight at
most W.

“knapsac
k”
Solution:
Items P
1 2 3 4 5 • 1 ml of 5
:
50$
Weigh 4 8 2 6 1
• 2 ml of 3
t:
Benefi ml
$1 ml
$3 ml
$4 ml
$3 ml
$5 10 40$
t:
Value 2
3 2
4 0
2 0
5 0
5 ml • 6 ml of 4
($ per 0 0 30$
:
ml) • 1 ml of 132
4$
 The Fractional Knapsack Algorithm
• Greedy choice: Keep taking item with the highest value (benefit to
weight ratio)
– Since

Algorithm fractionalKnapsack(S, W)
Input: set S of items w/ benefit bi and weight wi; max. weight W
Output: amount xi of each item i to maximize benefit w/ weight at most W

for each item i in S

xi ← 0
vi ← bi / wi {per unit benefit}
w←0 {total weight}
while w < W
remove item i with highest vi
xi ← min{wi , W - w}
w ← w + min{wi , W - w}
14
 The Fractional Knapsack Algorithm
• Running time: Given a collection S of n items, such that each item i
has a benefit bi and weight wi, we can construct a maximum-benefit
subset of S, allowing for fractional amounts, that has a total weight W in
O(nlogn) time.
– Use heap-based priority queue to store S
– Removing the item with the highest value takes O(logn) time
– In the worst case, need to remove all items

15
 An Activity Selection Problem
(Conference Scheduling Problem)

• Input: A set of activities S = {a1,…, an}

• Each activity has a start time and a finish time
– ai=(si, fi)
• Two activities are compatible if and only if their
interval does not overlap
• Output: a maximum-size subset of mutually
compatible activities

16
 The Activity Selection Problem
• Here are a set of start and finish times

• What is the maximum number of activities that can be

completed?
• {a3, a9, a11} can be completed
• But so can {a1, a4, a8’ a11} which is a larger set
• But it is not unique, consider {a2, a4, a9’ a11}

17
 The Activity Selection Problem
Input: list of time-intervals L

Output: a non-overlapping subset S of the intervals

Objective: maximize |S| 3,7

2,4
5,8
6,9
1,11
10,1
2 18
 The Activity Selection Problem
Input: list of time-intervals L

Output: a non-overlapping subset S of the

intervals
3,7
Objective: maximize |S| 2,4
5,8
6,9
1,11
Answer = 10,1
3 2 19
 The Activity Selection Problem
Algorithm 1:

1. sort the activities by the starting time

2. pick the first activity a
3. remove all activities conflicting with
a
4. repeat

20
 The Activity Selection Problem
Algorithm 1:

1. sort the activities by the starting time

2. pick the first activity “a”
3. remove all activities conflicting with
“a”
4. repeat

21
 The Activity Selection Problem
Algorithm 1:

1. sort the activities by the starting time

2. pick the first activity “a”
3. remove all activities conflicting with
“a”
4. repeat

22
 The Activity Selection Problem
Algorithm 2:

1. sort the activities by length

2. pick the shortest activity “a”
3. remove all activities conflicting with
“a”
4. repeat

23
 The Activity Selection Problem
Algorithm 2:

1. sort the activities by length

2. pick the shortest activity “a”
3. remove all activities conflicting with
“a”
4. repeat

24
 The Activity Selection Problem
Algorithm 2:

1. sort the activities by length

2. pick the shortest activity “a”
3. remove all activities conflicting with
“a”
4. repeat

25
 The Activity Selection Problem
Algorithm 3:

1. sort the activities by ending time

2. pick the activity which ends first
3. remove all activities conflicting with
a
4. repeat

26
 The Activity Selection Problem
Algorithm 3:

1. sort the activities by ending time

2. pick the activity which ends first
3. remove all activities conflicting with
a
4. repeat

27
 The Activity Selection Problem
Algorithm 3:

1. sort the activities by ending time

2. pick the activity which ends first
3. remove all activities conflicting with
a
4. repeat

28
 The Activity Selection Problem
Algorithm 3:

1. sort the activities by ending time

2. pick the activity which ends first
3. remove all activities conflicting with
a
4. repeat

29
 The Activity Selection Problem
Algorithm 3:

1. sort the activities by ending time

2. pick the activity a which ends first
3. remove all activities conflicting with a
4. repeat

Theorem:

Algorithm 3 gives an optimal solution to

the activity selection problem.
30
 Activity Selection Algorithm

Idea: At each step, select the activity with the smallest finish time
that is compatible with the activities already chosen.

Greedy-Activity-Selector(s, f)
sort the activities by ending time
n <− length[s]
A <− {1} {Automatically select first
activity}
j <− 1 {Last activity selected so far}
for i <− 2 to n do
if si >= fj then
A <− A U {i}{Add activity i to the set}
j <− i {record last activity added}
return A 31
 The Activity Selection Problem
• Here are a set of start and finish times

• What is the maximum number of activities that can be

completed?
• {a3, a9, a11} can be completed
• But so can {a1, a4, a8’ a11} which is a larger set
• But it is not unique, consider {a2, a4, a9’ a11}

32
Interval Representation

33
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3415
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3515
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3615
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3715
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3815
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 3915
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 4015
 Why this Algorithm is Optimal?
• We will show that this algorithm uses the
following properties
• The problem has the optimal substructure
property
• The algorithm satisfies the greedy-choice
property
• Thus, it is Optimal

41
 Greedy-Choice Property

• Show there is an optimal solution that begins with a greedy

choice (with activity 1, which as the earliest finish time)
• Suppose A ⊆ S in an optimal solution
– Order the activities in A by finish time. The first activity in A is k
• If k = 1, the schedule A begins with a greedy choice
• If k ≠ 1, show that there is an optimal solution B to S that begins with the
greedy choice, activity 1
– Let B = A – {k} ∪ {1}
• f1 ≤ fk 🡪 activities in B are disjoint (compatible)
• B has the same number of activities as A
• Thus, B is optimal

42
 Optimal Substructures
– Once the greedy choice of activity 1 is made, the problem reduces
to finding an optimal solution for the activity-selection problem over
those activities in S that are compatible with activity 1
• Optimal Substructure
• If A is optimal to S, then A’ = A – {1} is optimal to S’={i ∈S: si ≥ f1}
• Why?
– If we could find a solution B’ to S’ with more activities than A’, adding
activity 1 to B’ would yield a solution B to S with more activities than A 🡺
contradicting the optimality of A
– After each greedy choice is made, we are left with an optimization
problem of the same form as the original problem
• By induction on the number of choices made, making the greedy choice
at every step produces an optimal solution

43
 Huffman Codes

• Widely used technique for data compression

• Assume the data to be a sequence of characters

• Looking for an effective way of storing the data

• Binary character code

– Uniquely represents a character by a binary string

44
 Fixed-Length Codes

E.g.: Data file containing 100,000 characters

a b c d e f
Frequency (thousands) 45 13 12 16 9 5

• 3 bits needed
• a = 000, b = 001, c = 010, d = 011, e = 100, f = 101
• Requires: 100,000 ⋅ 3 = 300,000 bits

45
 Huffman Codes

• Idea:
– Use the frequencies of occurrence of characters to
build a optimal way of representing each character

a b c d e f
Frequency (thousands) 45 13 12 16 9 5

46
 Variable-Length Codes
E.g.: Data file containing 100,000 characters

a b c d e f
Frequency (thousands) 45 13 12 16 9 5

• Assign short codewords to frequent characters and

long codewords to infrequent characters
• a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100
• (45 ⋅ 1 + 13 ⋅ 3 + 12 ⋅ 3 + 16 ⋅ 3 + 9 ⋅ 4 + 5 ⋅ 4) ⋅
1,000
= 224,000 bits
47
 Prefix Codes

• Prefix codes:
– Codes for which no codeword is also a prefix of some
other codeword
– Better name would be “prefix-free codes”

• We can achieve optimal data compression using

prefix codes
– We will restrict our attention to prefix codes

48
Encoding with Binary Character Codes

• Encoding
– Concatenate the codewords representing each
character in the file

• E.g.:
– a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100
– abc = 0 ⋅ 101 ⋅ 100 = 0101100

49
Decoding with Binary Character Codes
• Prefix codes simplify decoding
– No codeword is a prefix of another ⇒ the codeword
that begins an encoded file is unambiguous
• Approach
– Identify the initial codeword
– Translate it back to the original character
– Repeat the process on the remainder of the file
• E.g.:
– a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100
– 001011101 = 0 ⋅ ⋅ ⋅ 1101 = aabe
0 101
50
 Prefix Code Representation
• Binary tree whose leaves are the given characters
• Binary codeword
– the path from the root to the character, where 0 means “go to the
left child” and 1 means “go to the right child”
• Length of the codeword
– Length of the path from root to the character leaf (depth of node)
1 1
0 0
0 1 0 1
0 0
8 1 a: 5
6 4 45 0 5 1
0 1 0
2 3
5 2 1 5 0
8 8 4 0 1 0 1
0 1 0 1 0 1 c: b: 1 d:
a: b: c: d: 12 13 16
e: 9 f: 5 4
45 13 12 16 0 1
f: 5 e: 9 51
 Optimal Codes
• An optimal code is always represented by a full
binary tree
– Every non-leaf has two children
– Fixed-length code is not optimal, variable-length is
• How many bits are required to encode a file?
– Let C be the alphabet of characters
– Let f(c) be the frequency of character c
– Let dT(c) be the depth of c’s leaf in the tree T
corresponding to a prefix code
the cost of tree T

52
 Constructing a Huffman Code
• A greedy algorithm that constructs an optimal prefix code
called a Huffman code
• Assume that:
– C is a set of n characters
– Each character has a frequency f(c)
– The tree T is built in a bottom up manner
• Idea: c: b: d: a:
f: 5 e: 9
12 13 16 45
– Start with a set of |C| leaves
– At each step, merge the two least frequent objects: the frequency of
the new node = sum of two frequencies
– Use a min-priority queue Q, keyed on f to identify the two least
frequent objects
53
 Example
c: b: d: a: c: b: 1 d: a:
f: 5 e: 9
12 13 16 45 12 13 0 4 1 16 45
f: 5 e: 9
1 d: 2 a: 2 3 a:
0 4 1 16 0
5 1 45
0 5 1 0 0 1 45
c: b: c: b: 1 d:
f: 5 e: 9
12 13 12 13 16
0 4 1
f: 5 e: 9
1
a: 5 0 0 1
45 0 5 1 a: 0 5
2 3 45 0 5 1
0
5 1 0 0 1 2 3
c: b: 1 d:
12 13 16 0 5 1 0 0 1
4 c: b: 1 d:
0 1
f: 5 e: 9 12 13 0 4 1 16
f: 5 e: 9 54
 Building a Huffman Code
Alg.: HUFFMAN(C) Running time:
O(nlgn)
1. n ← ⎜C ⎜
2. Q ← C O(n
3. for i ← 1 to n – 1 )

4. do allocate a new node z

5. left[z] ← x ← EXTRACT-MIN(Q)
O(nlgn
6. right[z] ← y ← EXTRACT-MIN(Q) )
7. f[z] ← f[x] + f[y]
8. INSERT (Q, z)
9. return EXTRACT-MIN(Q)
55