Unit 2

Greedy
Method/Technique

1
Lecture Content

1.The Knapsack Problem

2. Job Sequencing with deadline
3. Activity Selection Problem
4. Huffman Algorithm
5. Kruskal’s Algorithm

2
•The greedy approach suggests constructing a
solution through a sequence of steps, each
expanding a partially constructed solution
obtained so far, until a complete solution to the
problem is reached. On each step, the choice made
must be:

3
 Greedy Technique

Constructs a solution to an optimization problem piece by

piece through a sequence of choices that are:

feasible, i.e. satisfying the constraints Defined by an

objective function and
a set of constraints
locally optimal (with respect to some neighborhood definition)

greedy (in terms of some measure), and irrevocable

For some problems, it yields a globally optimal solution for every

instance. For most, does not but can be useful for fast approximations.
We are mostly interested in the former case in this class.
 Applications of the Greedy Strategy
Optimal solutions:
change making for “normal” coin denominations
minimum spanning tree (MST)
single-source shortest paths
simple scheduling problems
Huffman codes

Approximations/heuristics:
traveling salesman problem (TSP)
knapsack problem
other combinatorial optimization problems
 Change-Making Problem

Given unlimited amounts of coins of denominations d1 > … > dm ,

give change for amount n with the least number of coins
Q: What are the objective function and constraints?

Example: d1 = 25c, d2 =10c, d3 = 5c, d4 = 1c and n = 48c

25c =1
Greedy solution: 10c=2
<1, 2, 0, 3> 1c=3
Greedy solution is 25*1+10*2+1*3=48c
optimal for any amount and “normal’’ set of denominations
Ex: Prove the greedy algorithm is optimal for the below denominations.

For example, d1 = 25c, d2 = 10c, d3 = 1c, and n = 30c

Greedy Solution may not be optimal for arbitrary coin denominations
 -feasible: it has to satisfy the problem’s
constraints
-locally optimal: it has to be the best local
choice among all feasible choices available on that
step
-irrevocable: once decision was made, it
cannot be changed on subsequent steps of the
algorithm

7
• The greedy approach is based on the principle:
- a sequence of locally optimal choices will
yield a globally optimal solution to the entire
problem.
• However, a sequence of locally optimal choices
does not always yield a globally optimal solution

8
1. The Knapsack Problem

Knapsack Problem-
• A knapsack (kind of shoulder bag) with limited weight capacity.
• Few items each having some weight and value.
The problem states-
Which items should be placed into the knapsack such that-
• The value or profit obtained by putting the items into the knapsack is
maximum.
• And the weight limit of the knapsack does not exceed.

9
1. The Knapsack Problem

1
1. The Knapsack Problem

Knapsack Problem Variants-

Knapsack problem has the following three variants-

• Fractional Knapsack Problem
• Brute force method
• 0/1 Knapsack Problem

In this lecture, we will discuss about Fractional Knapsack Problem.

Fractional Knapsack Problem-

In Fractional Knapsack Problem,

• As the name suggests, items are divisible here.
• We can even put the fraction of any item into the knapsack if taking the complete item
is not possible.
• It is solved using Greedy Method.

1
1. The Knapsack Problem

Fractional Knapsack Problem Using Greedy Method-

Fractional knapsack problem is solved using greedy method in the following steps-

Step-01:

For each item, compute its value / weight ratio.

Step-02:

Arrange all the items in decreasing order of their value / weight ratio.

Step-03:

Start putting the items into the knapsack beginning from the item with the highest ratio.
Put as many items as you can into the knapsack.

1
1. The Knapsack Problem

•The knapsack problem: Given a knapsack of

capacity W and n items of weights w1, ..., wn and
values v1, ..., vn, where wi and W are positive
inetegrs, find the most valuable subset of the items
that fits into the knapsack.

1
 1. The Knapsack Problem

•The continuous (or fractional) knapsack

problem: any fraction of any given item can be
taken into the knapsack. Let xi, i = 1, ..., n, be a
variable representing a fraction of item i taken into
the knapsack. Obviously, xi must satisfy the
inequality 0 ≤ xi ≤ 1. Then the total weight of the
n

n the sum 
selected items can be expressed by i w i xi ,
and their total value by the sum  vi xi . 1

i 1

1
1. The Knapsack Problem

•Thus, the continuous version of the knapsack

problem can be expressed as the following
optimization problem:

1
 The knapsack algorithm

Problem-

For the given set of items and knapsack capacity = 60 kg, find the optimal solution for
the fractional knapsack problem making use of greedy approach.

16
 The knapsack example
OR
Find the optimal solution for the fractional knapsack problem making use of greedy
approach. Consider-
n=5
w = 60 kg
(w1, w2, w3, w4, w5) = (5, 10, 15, 22, 25)
(b1, b2, b3, b4, b5) = (30, 40, 45, 77, 90)

OR
A thief enters a house for robbing it. He can carry a maximal weight of 60 kg into his
bag. There are 5 items in the house with the following weights and values. What items
should thief take if he can even take the fraction of any item with him?

17
 The knapsack example
Solution-

Step-01:

Compute the value / weight ratio for each item-

Step-02:

Sort all the items in decreasing order of their value / weight ratio-

I1 I2 I5 I4 I3
(6) (4) (3.6) (3.5) (3)

Step-03:

Start filling the knapsack by putting the items into it one by one.

18
 The knapsack example
Step-03:

Start filling the knapsack by putting the items into it

one by one.

Now,
Knapsack weight left to be filled is 20 kg but item-4 has a weight of 22 kg.
Since in fractional knapsack problem, even the fraction of any item can be taken.
So, knapsack will contain the following items-
< I1 , I2 , I5 , (20/22) I4 >

Total cost of the knapsack

= 160 + (20/27) x 77
= 160 + 70
= 230 units 19
 The knapsack problem

n objects, each with a weight wi > 0

a profit pi > 0
capacity of knapsack:
M
 pi xi
1i n
Maximize
 w i x i M
Subject to 1in
0  xi  1, 1  i  n

20
The knapsack algorithm

The greedy algorithm:

Step 1: Sort pi/wi into nonincreasing order.
Step 2: Put the objects into the knapsack according
to the sorted sequence as possible as we can.
e. g.
n = 3, M = 20, (p1, p2, p3) = (25, 24, 15)
(w1, w2, w3) = (18, 15, 10)

21
The knapsack algorithm

The greedy algorithm:

Step 1: Sort pi/wi into nonincreasing order.
Step 2: Put the objects into the knapsack according
to the sorted sequence as possible as we can.
e. g.
n = 3, M = 20, (p1, p2, p3) = (25, 24, 15)
(w1, w2, w3) = (18, 15, 10)
Sol: p1/w1 = 25/18 = 1.39
p2/w2 = 24/15 = 1.6
p3/w3 = 15/10 = 1.5
Optimal solution: x1 = 0, x2 = 1, x3 = 1/2
total profit = 24 + 7.5 = 31.5

22
Knapsack algorithm

Algorithm fractionalKnapsack(S, W)
Input: set S of items w/ benefit bi
and weight wi; max. weight W
Output: amount xi of each item i
to maximize benefit w/ weight
at most W
for each item i in S
xi  0
vi  bi / wi {value}
w0 {total
weight}
while w < W
remove item i w/ highest vi
xi  min{wi , W - w}
w  w + min{wi , W - w}
 0/1 Knapsack Problem - Example 1

• The brute-force strategy

- generate all the subsets of the
set of n items given,
- computing the total weight of each subset
to identify feasible subsets (i.e., the
total
ones weight with
not exceeding
the the knapsack’s
capacity), and
- finding a subset of the largest value among them.

24
 0/1 Knapsack Problem - Example 1

•Consider the following instance of the

0/1 knapsack problem.

25
 0/1 Knapsack Problem - Example 1

26
 0/1 Knapsack Problem - Example 2

•Consider the following instance of the

0/1 knapsack problem.

27
 0/1 Knapsack Problem - Example 2 - Brute Force

28
1. The Knapsack Problem

•Since the number of subsets of an n-element set

is 2n, the brute-force approach leads to a Ω(2n)
algorithm no matter how efficiently individual
subsets are generated.
•The greedy strategy: We can think of several
greedy approaches to the knapsack problem.
- One is to select the items in decreasing order
of their weights. However, heavier items may not
be the most valuable in the set.
29
1. The Knapsack Problem

-Alternatively, if we pick up the items in

decreasing order of their value, there is no
guarantee that the knapsack’s capacity will be
used efficiently.
-Can we find a greedy strategy that takes into
account both the weights and values? Yes, we can,
by computing the value-to-weight ratios vi/wi, i =
1, 2, …, n, and selecting the items in decreasing
order of these ratios.

30
1. The Knapsack Problem

•The discrete (or 0-1) knapsack problem: we are

allowed either to take an item in its entirety or not
to take it at all. Hence, we have the following
optimization problem:

3
1. The Knapsack Problem

• The greedy strategy for the 0/1

knapsack
problem
- compute value-to-weight ratios ri = vi / wi.
- sort computed ratios ri in descending order.
-select the items in decreasing order of these
ratios. That is, make greedy selection with regard
to ri. Specifically, select items in decreasing order
ri until their total weight exceeds the knapsack
capacity W.
32
 0/1 Knapsack Problem - Example 1

33
 0/1 Knapsack Problem - Example 1

Select item 3, total weight = 4 < W. Thus, item 3

is selected.
Add item 1 to knapsack, total weight = 11 > W.
Thus, item 1 is not selected.
Add item 4 to knapsack, total weight = 9 < W. Thus, item
4 is selected.
Add item 2 to knapsack, total weight = 12 > W.
Thus, item 2 is not selected.
Therefore, the most valuable subset of the items that fits
into the knapsack is {3, 4} with the total value of $65.
34
 0/1 Knapsack Problem - Example 2 - Greedy

35
 0/1 Knapsack Problem - Example 2 - Greedy

Select the item 1, total weight = 10 < W. Thus, the

item 1 is selected.
Add the item 2 to the knapsack, total weight =
17
> W. Thus, the item 2 is not selected.
Add the item 3 to the knapsack, total weight =
18
> W. Thus, the item 3 is not selected.
Add the item 4 to the knapsack, total weight =
14 36
 0/1 Knapsack Problem - Example 2 - Greedy

•Therefore, the most valuable subset of the items

that fits into the knapsack is {1, 4} with the total
value of $112.
•Actually, this is not the optimal solution. The
most valuable subset of the items that fits into the
knapsack is {2, 3} with the total value of $119.

37
1. The Knapsack Problem

Remarks:
-The greedy strategy does not always generate the
(globally) optimal solution.
-Some optimization problem can not be solved by
the greedy strategy.

38
Job sequencing with deadlines

The sequencing of jobs on a single processor with deadline constraints is

called as Job Sequencing with Deadlines.
Here-
You are given a set of jobs.
Each job has a defined deadline and some profit associated with it.
The profit of a job is given only when that job is completed within its
deadline.
Only one processor is available for processing all the jobs.
Processor takes one unit of time to complete a job.
The problem states-
“How can the total profit be maximized if only one job can be completed
at a time?”

39
Job sequencing with deadlines
Approach to Solution-

A feasible solution would be a subset of jobs where each job of the subset
gets completed within its deadline.
Value of the feasible solution would be the sum of profit of all the jobs
contained in the subset.
An optimal solution of the problem would be a feasible solution which
gives the maximum profit.

Greedy Algorithm-

Greedy Algorithm is adopted to determine how the next job is selected for
an optimal solution.
The greedy algorithm described below always gives an optimal solution to
the job sequencing problem-

40
 Job sequencing with deadlines

Step-01:

Sort all the given jobs in decreasing order of their profit.

Step-02:

Check the value of maximum deadline.

Draw a Gantt chart where maximum time on Gantt chart is the value of maximum
deadline.

Step-03:

Pick up the jobs one by one.

Put the job on Gantt chart as far as possible from 0 ensuring that the job gets
completed before its deadline.

41
 Job sequencing with deadlines
Problem-

Given the jobs, their deadlines and associated profits as shown-

Answer the following questions-

Write the optimal schedule that gives maximum profit.
Are all the jobs completed in the optimal schedule?
What is the maximum earned profit?

42
 Job sequencing with deadlines
Solution-

Step-01:

Sort all the given jobs in decreasing order of their profit-

Step-02:

Value of maximum deadline = 5.

So, draw a Gantt chart with maximum time on Gantt chart = 5 units as shown-

43
Job sequencing with deadlines
Now,
We take each job one by one in the order they appear in Step-01.
We place the job on Gantt chart as far as possible from 0.

Step-03:

We take job J4.

Since its deadline is 2, so we place it in the first empty cell before deadline 2 as-

Step-04:

We take job J1.

Since its deadline is 5, so we place it in the first empty cell before deadline 5 as-

44
Job sequencing with deadlines
Step-05:

We take job J3.

Since its deadline is 3, so we place it in the first empty cell before deadline 3 as-

Step-06:

We take job J2.

Since its deadline is 3, so we place it in the first empty cell before deadline 3.
Since the second and third cells are already filled, so we place job J2 in the first cell as-

45
 Job sequencing with deadlines
Step-07:

Now, we take job J5.

Since its deadline is 4, so we place it in the first empty cell before deadline 4 as-

Now,
The only job left is job J6 whose deadline is 2.
All the slots before deadline 2 are already occupied.
Thus, job J6 can not be completed.
Now, the given questions may be answered as-
Part-01:
The optimal schedule is-
J2 , J4 , J3 , J5 , J1
This is the required order in which the jobs must be completed in order to obtain
the maximum profit.
46
 Job sequencing with deadlines

Part-02:

All the jobs are not completed in optimal schedule.

This is because job J6 could not be completed within its deadline.

Part-03:

Maximum earned profit

= Sum of profit of all the jobs in optimal schedule
= Profit of job J2 + Profit of job J4 + Profit of job J3 + Profit of job J5 + Profit of
job J1
= 180 + 300 + 190 + 120 + 200

47
Job sequencing with deadlines

Problem: n jobs, S={1, 2, …, n}, each job i has a deadline di  0 and a

profit pi  0. We need one unit of time to process each job and we can do
at most one job each time. We can earn the profit pi if job i is completed by
its deadline.

i 1 2 3 4 5
pi 20 15 10 5 1
di 2 2 1 3 3

48
Algorithm:
Step 1: Sort pi into nonincreasing order. After sorting p1  p2 

p3  …  pn.
Step 2: Add the next job i to the solution set if i can be
completed by its deadline. Assign i to time slot [r-1, r], where r
is the largest integer such that 1  r  di and [r-1, r] is free.
Step 3: Stop if all jobs are examined. Otherwise, go to step 2.

Time complexity: O(n2)

49
e.g.
i pi di
1 20 2 assign to [1, 2]
2 15 2 assign to [0, 1]
3 10 1 reject
4 5 3 assign to [2, 3]
5 1 3 reject

solution = {1, 2, 4}
total profit = 20 + 15 + 5 = 40

50
 A simple example

Problem: Pick k numbers out of n numbers

such that the sum of these k numbers is the
largest.
Algorithm:
FOR i = 1 to k
pick out the largest number and
delete this number from the input.
ENDFOR

51
 The greedy method

Suppose that a problem can be solved by a sequence of

decisions. The greedy method has that each decision is locally
optimal. These locally optimal solutions will finally add up to
a globally optimal solution.
Only a few optimization problems can be solved by the greedy
method.

52
 The activity selection problem

Problem: n activities, S = {1, 2, …, n}, each activity i has a

start time si and a finish time fi, si  fi.

Activity i occupies time interval [si, fi].

i and j are compatible if si  fj or sj  fi.

The problem is to select a maximum-size set of mutually
compatible activities

53
The activity selection problem

54
The activity selection problem

55
The activity selection problem

56
 Algorithm

Input Data for the Algorithm:

•act[] array containing all the activities.
•s[] array containing the starting time of all the activities.
•f[] array containing the finishing time of all the activities.

Ouput Data from the Algorithm:

•sol[] array referring to the solution set containing the maximum number of non-conflicting

activities.
Steps for Activity Selection Problem
Following are the steps we will be following to solve the activity selection problem,
Step 1: Sort the given activities in ascending order according to their finishing time.
Step 2: Select the first activity from sorted array act[] and add it to sol[] array.
Step 3: Repeat steps 4 and 5 for the remaining activities in act[].
Step 4: If the start time of the currently selected activity is greater than or equal to
the finish time of previously selected activity, then add it to the sol[] array.
Step 5: Select the next activity in act[] array.
Step 6: Print the sol[] arra
 Activity Selection Problem
Example

Start Time (s) Finish Time (f) Activity Name

5 9 a1
1 2 a2
3 4 a3
0 6 a4
5 7 a5
8 9 a6
 Activity Selection Problem
Example

A
possible solution wo
uld be:
Step 1: Sort the
Start Time Finish Time (f) Activity Name
(s)

given activities in 4
1 2 a2
3 a3

ascending order
0
5
6
7
a4
a5
according to 5their9 a1

finishing time.
8 9 a6

Sorted table
 Activity Selection Problem
Example

Step 2: Select the first activity from sorted array act[] and
add it to the sol[] array, thus sol = {a2}.
Step 3: Repeat the steps 4 and 5 for the remaining activities
in act[].
Step 4: If the start time of the currently selected activity is
greater than or equal to the finish time of the previously
selected activity, then add it to sol[].
Step 5: Select the next activity in act[]
 Activity Selection Problem
Example

For the data given in the above table,

A.Select activity a3. Since the start time of a3 is greater than the
finish time of a2 (i.e. s(a3) > f(a2)), we add a3 to the solution
set. Thus sol = {a2, a3}.
B.Select a4. Since s(a4) < f(a3), it is not added to the solution
set.
C.Select a5. Since s(a5) > f(a3), a5 gets added to solution set.
Thus sol = {a2, a3, a5}
D.Select a1. Since s(a1) < f(a5), a1 is not added to the solution
set.
E.Select a6. a6 is added to the solution set since s(a6) > f(a5).
Thus sol = {a2, a3, a5, a6}.
Step 6: At last, print the array sol[]
Example:
i 1 2 3 4 5 6 7 8 9 10 11
si 1 3 0 5 3 5 6 8 8 2 12
fi 4 5 6 7 8 9 10 11 12 13 14

The solution set = {1, 4, 8, 11}

Algorithm:
Step 1: Sort fi into nondecreasing order. After sorting, f1  f2 
f 3  …  fn.
Step 2: Add the next activity i to the solution set if i is
compatible with each in the solution set.
Step 3: Stop if all activities are examined. Otherwise, go to
step 2.

Time complexity: O(nlogn) 62

Solution of the example:
i si fi accept
1 1 4 Yes
2 3 5 No
3 0 6 No
4 5 7 Yes
5 3 8 No
7 6 10 No
8 8 11 Yes
9 8 12 No
10 2 13 No
11 12 14 Yes

Solution = {1, 4, 8, 11} 63

The activity selection problem

64
The activity selection problem

65
 Activity-selection
problem
Greedy_Activity_Selector(s,f)
{ n = length(s)
A = {1}
j=1
for i = 2 to n
if (s[i] >= f[j])
A = A U {i}
j=i
end if
end for
Return A
}
Assume the array f already sorted
Complexity: T(n) = O(n)
The activity selection problem

67
5. Huffman Code

•Idea: assign shorter codewords to more frequent

symbols and longer codewords to less frequent
symbols.
•Huffman tree: is a binary tree with all the left
edges are labeled by 0 and all the right edges are
labeled by 1.
•The codeword of a symbol can then be obtained
by recording the labels on the simple path from
the root to the symbol’s leaf.
68
 Huffman codes

Any binary tree with edges labeled with 0’s and 1’s yields a prefix-free code of
characters assigned to its leaves
0 1

Optimal binary tree minimizing the average 0 1

length of a codeword can be constructed 1
as follows:
Huffman’s algorithm represents {00, 011, 1}

Initialize n one-node trees with alphabet characters and the tree weights with
their frequencies.
Repeat the following step n-1 times: join two binary trees with smallest
weights into one (as left and right subtrees) and make its weight equal the
sum of the weights of the two trees.
Mark edges leading to left and right subtrees with 0’s and 1’s, respectively.
 Huffman codes

1. Huffman Coding is a famous Greedy Algorithm.

2. It is used for the lossless compression of data.
3. It uses variable length encoding.
4. It assigns variable length code to all the characters.
5. The code length of a character depends on how frequently it occurs in the
given text.
6. The character which occurs most frequently gets the smallest code.
7. The character which occurs least frequently gets the largest code.
8. It is also known as Huffman Encoding.
 Prefix rule for Huffman codes

1. Huffman Coding implements a rule known as a prefix rule.

2. This is to prevent the ambiguities while decoding.

3. It ensures that the code assigned to any character is not a prefix of the code

assigned to any other character.

 Construct Huffman Tree

There are mainly two major parts in Huffman Coding

1) Build a Huffman Tree from input characters.
2) Traverse the Huffman Tree and assign codes to characters.
Steps to build Huffman Tree
Input is an array of unique characters along with their frequency of occurrences
and output is Huffman Tree.
1. Create a leaf node for each unique character and build a min heap of all leaf
nodes (Min Heap is used as a priority queue. The value of frequency field is used
to compare two nodes in min heap. Initially, the least frequent character is at root)
2. Extract two nodes with the minimum frequency from the min heap.
3. Create a new internal node with a frequency equal to the sum of the two nodes
frequencies. Make the first extracted node as its left child and the other extracted
node as its right child. Add this node to the min heap.
4. Repeat steps#2 and #3 until the heap contains only one node. The remaining
node is the root node and the tree is complete.
5. Huffman Code - Example

• Consider the five-symbol alphabet {A, B, C, D, -

} with the following occurrence frequencies in
a text made up of these symbols:

73
 Constructed Huffman Tree

74
5. Huffman Code - Example

• The resulting codewords are as follows:

Encoding: DAD is encoded as 011101.

Decoding: 10011011011101 is decoded as
BAD-AD. (Huffman tree is needed, weakness)
75
 Huffman Tree Construction Time

Algorithm Huffman(W[0..n − 1])

//Constructs Huffman’s tree
//Input: An array W[0..n − 1] of weights
//Output: A Huffman tree with the given weights
assigned to its leaves

initialize priority queue Q of size n with one-node

trees and priorities equal
to the elements of W[0..n − 1]

77
 Huffman Tree Construction Time

while Q has more than one element do

Tℓ ← the minimum-weight tree in Q
delete the minimum-weight tree in Q
Tr ← the minimum-weight tree in Q
delete the minimum-weight tree in Q
create a new tree T with Tℓ and Tr as its left and
right subtrees
and the weight equal to the sum of Tℓ and
Tr

eights 78
 Huffman Tree Construction Time Complexity

T(n) = O(n log n)

Time Complexity-

The time complexity analysis of Huffman Coding is as follows-

extractMin( ) is called 2 x (n-1) times if there are n nodes.
As extractMin( ) calls minHeapify( ), it takes O(logn) time.

Thus, Overall time complexity of Huffman Coding becomes O(nlogn).

Here, n is the number of unique characters in the given text.

79
 Huffman Tree Construction Time Complexity

Important Formulas-

The following 2 formulas are important to solve the problems based on Huffman Coding-

Formula-02:

Total number of bits in Huffman encoded message

= Total number of characters in the message x Average code length per character
= ∑ ( frequencyi x Code lengthi )

80
Example

A file contains the following characters with the frequencies as shown.

If Huffman Coding is used for data compression, determine-
1. Huffman Code for each character
2. Average code length
3. Length of Huffman encoded message (in bits)

81
Huffman Tree
Huffman Tree after assigning weight
 Huffman Code For Characters-

To write Huffman Code for any character, traverse the Huffman Tree from
root node to the leaf node of that character.
Following this rule, the Huffman Code for each character is-
a = 111
e = 10
i = 00
o = 11001
u = 1101
s = 01
t = 11000

From here, we can observe-

1. Characters occurring less frequently in the text are assigned the larger
code.
2. Characters occurring more frequently in the text are assigned the
smaller code.
 Huffman Code For Characters-

2. Average Code Length-

Using formula-01, we have-

Average code length
= ∑ ( frequencyi x code lengthi ) / ∑ ( frequencyi )
= { (10 x 3) + (15 x 2) + (12 x 2) + (3 x 5) + (4 x 4) + (13 x 2) + (1 x 5) } / (10 +
15 + 12 + 3 + 4 + 13 + 1)
= 2.52

3. Length of Huffman Encoded Message-

Using formula-02, we have-

Total number of bits in Huffman encoded message
= Total number of characters in the message x Average code length per
character
= 58 x 2.52
= 146.16
≅ 147 bits
 Huffman codes

In telecommunication, how do we represent a set of

messages, each with an access frequency, by a
sequence of 0’s and 1’s?
To minimize the transmission and decoding costs,
we may use short strings to represent more
frequently used messages.
This problem can by solved by using an extended
binary tree which is used in the 2-way merging
problem.

3 -86
An example of Huffman algorithm
Symbols: A, B, C, D, E, F, G
freq. : 2, 3, 5, 8, 13, 15, 18

Huffman codes:
A: 10100 B: 10101 C:
1011
D: 100 E: 00 F: 01
G: 11

A Huffman code Tree

3-
87
Exercises

1. a. Construct a Huffman code for the following

data:

b.Encode ABACABAD using the code of question

(a).
c.Decode 100010111001010 using the code
of question (a).
88
 Minimum Spanning Tree (MST)

Spanning tree of a connected graph G: a connected

acyclic subgraph of G that includes all of G’s vertices

Minimum spanning tree of a weighted, connected

graph G: a spanning tree of G of the minimum total
weight

Example:
6 c 6 c c
a a a
1 4 1 1
4
2 2

d d d
b 3 b b 3
Another greedy algorithm for MST: Kruskal’s

Sort the edges in nondecreasing order of lengths

“Grow” tree one edge at a time to produce MST through a series

of expanding forests F1, F2, …, Fn-1

On each iteration, add the next edge on the sorted list unless this
would create a cycle. (If it would, skip the edge.)
 Example

4 c 4 c 4 c
a a a
1 1 1
6 6 6
2 2 2
d d d
b 3 b 3 b 3

4 c c c
a a a
1 1 1
6 6
2 2 2
d d d
b 3 b 3 b 3
 Notes about Kruskal’s algorithm

Algorithm looks easier than Prim’s but is harder to implement

(checking for cycles!)

Cycle checking: a cycle is created iff added edge connects

vertices in the same connected component

Union-find algorithms – see section 9.2

Runs in O(m log m) time, with m = |E|. The time is mostly spent
on sorting.
 Minimum Spanning Tree (MST) Problem

•Given n points, connect them in the cheapest

possible way so that there will be a path between
every pair of points.
•The solution is called the minimum spanning
tree (MST).
•The MST is a tree connecting all points and has
the minimum cost (also called length, or weight).

93
 Minimum Spanning Tree (MST) Problem

•The minimum spanning tree problem is the

problem of finding a minimum spanning tree
(MST) for a given weighted connected undirected
graph.
•The total number of possible spanning trees for n
vertices (nodes, points, or cities) is nn-2 (it grows
exponentially).
•Thus, it is impossible to employ the brute-force
strategy to solve the MST problem. It is solved by
the greedy method. 94
2. Kruskal’s Algorithm

[R. C. T. Lee]
Algorithm: Kruskal’s algorithm to find
a minimum spanning tree
Input: A weighted, connected, undirected graph G
= (V, E).
Output: A minimum spanning tree for G.
// n = |V|: number of vertices
// m = |E|: number of edges
95
2. Kruskal’s Algorithm

Step 1: [sorting, O(m log m)] Select the edge

with the smallest weight from the
set of sorted edges.
This constitutes the initial partially
constructed subgraph which will
later be developed into a minimum
spanning tree.
96
2. Kruskal’s Algorithm

Step 2: [union and finding operations, O(m)]

Add the next smallest weight edge to this
partially constructed graph if this will not
cause a cycle to be formed.
Discard the selected edge if otherwise.
Step 3: [O(m)] Terminate if the spanning tree
contains (n - 1) edges.
Otherwise, go to Step 2. 29
2. Kruskal’s Algorithm

•The time complexity of Kruskal’s algorithm is

O(m log m), where m = |E| is the number of edges
of G.

98
2. Kruskal’s Algorithm

[Anany Levitin]
ALGORITHM Kruskal(G)
//Kruskal’s algorithm for constructing a minimum
spanning tree
//Input: A weighted connected graph G = (V, E)
//Output: ET, the set of edges composing
a minimum spanning tree of G

99
2. Kruskal’s Algorithm

sort E in nondecreasing order of the edge weights

w( ei ) ≤ … ≤ w( i|E|
1
e )
E ← ; ecounter ← 0 //initialize the set of tree
T
edges and its size
k ← 0 //initialize the number of processed edges

10
2. Kruskal’s Algorithm

while ecounter < |V| - 1 do

k←k+1
if ET ∪ ei k} is acyclic
ET ← ET ∪ ei k}; ecounter ← ecounter + 1
{
{
return ET

10
2. Kruskal’s Algorithm - Example 1

•Consider the following weighted,

connected, undirected graph

10
2. Kruskal’s Algorithm - Example 1

• The edges are sorted into the

following
sequence:
(A, E) (C, D) (B, C) (B, D) (A, D) (D, E) (B, E) (A, C)

[ 50 60 70 75 80 90 200 300 ]

10
2. Kruskal’s Algorithm - Example 1

10
2. Kruskal’s Algorithm - Example 1

10
2. Kruskal’s Algorithm - Example 1

10
2. Kruskal’s Algorithm - Example 1

10
2. Kruskal’s Algorithm

• Remarks:
-If |E| is small, then Kruskal’s algorithm (O(m
log m)) is preferred.
-If |E| is large (i.e., |E| = (|V|2)), then Prim’s
algorithm (O(m log n)) is preferred.

10
 Prim’s MST algorithm

Start with tree T1 consisting of one (any) vertex and “grow” tree
one vertex at a time to produce MST through a series of
expanding subtrees T1, T2, …, Tn

On each iteration, construct Ti+1 from Ti by adding vertex not in

Ti that is closest to those already in Ti (this is a “greedy” step!)

Stop when all vertices are included

 Example

4 c 4 c 4 c
a a a
1 1 1
6 6 6
2 2 2
d d d
b 3 b 3 b 3

4 c
4 c
a
a 1
1 6
6
2
2
d
d b 3
b 3
 Notes about Prim’s algorithm

Proof by induction that this construction actually yields an MST

(CLRS, Ch. 23.1). Main property is given in the next page.

Needs priority queue for locating closest fringe vertex. The

Detailed algorithm can be found in Levitin, P. 310.

Efficiency
O(n2) for weight matrix representation of graph and array
implementation of priority queue
O(m log n) for adjacency lists representation of graph with n
vertices and m edges and min-heap implementation of the priority
queue
 The Crucial Property behind Prim’s
Algorithm

Claim: Let G = (V,E) be a weighted graph and (X,Y) be a

partition of V (called a cut). Suppose e = (x,y) is an edge of
E across the cut, where x is in X and y is in Y, and e has the
minimum weight among all such crossing edges (called a
light edge). Then there is an MST containing e.

Y
y

X
3. Prim’s Algorithm

[R. C. T. Lee]
Algorithm: The basic Prim’s algorithm to find
a minimum spanning tree
Input: A weighted, connected, undirected graph
G = (V, E).
Output: A minimum spanning tree for G.
Step 1: Let x be any vertex in V. Let X = {x} and
Y = V - {x}.
11
3. Prim’s Algorithm

Step 2: Select an edge (u, v) from E such that

u  X, v  Y and (u, v) has the smallest
weight among edges between X and Y.
Step 3: Connect u to v.
Let X = X  {v} and Y = Y -
{v}.
Step 4: If Y is empty, terminate and the resulting
tree is a minimum
Otherwise, go to Stepspanning tree. 43
3. Prim’s Algorithm

[Anany Levitin]
ALGORITHM Prim(G)
//Prim’s algorithm for constructing a
minimum spanning tree
//Input: A weighted connected graph G = (V, E)
//Output: ET, the set of edges composing
a minimum spanning tree of G

11
3. Prim’s Algorithm

VT ← {v0} //the set of tree vertices can be

initialized with any vertex
ET ← 

11
3. Prim’s Algorithm

for i ← 1 to |V| - 1 do
find a minimum-weight edge e* = (u*, v*)
among all the edges (u, v) such that
u is in VT and v is in V - VT
VT ← VT ∪ {v*} // add v* to VT
ET ← ET ∪ {e*} // add (u*, v*) to ET
return ET // all vertices included the
currently constructed tree 11
3. Prim’s Algorithm - Example 1

•Consider the following weighted,

connected, undirected graph

11
3. Prim’s Algorithm - Example 1

• B is selected as starting vertex.

11
3. Prim’s Algorithm - Example 1

• B is selected as starting vertex.

12
3. Prim’s Algorithm - Example 1

• B is selected as starting vertex.

12
3. Prim’s Algorithm - Example 1

• B is selected as starting vertex.

12
3. Prim’s Algorithm

•If a graph is represented by its weight matrix,

then the running time of Prim’s algorithm is (n2),
where n = |V|.

12
3. Prim’s Algorithm

•If graph is represented by its adjacency lists and

priority queue is implemented as a min-heap,
running time of Prim’s algorithm is in O(m log n),
where m = |E|, n = |V|.

12
3. Prim’s Algorithm

•This is because Prim’s algorithm performs n - 1

deletions of smallest element and makes m
verifications and, changes of an element’s priority
in a min-heap of size not exceeding n. Each of
these operations is a O(log n) operation. Hence,
running time of this implementation of Prim’s
algorithm is in
(n - 1 + m) O(log n) = O(m log n)
because, in a connected graph, n - 1 ≤ m,
where m = |E|, n = |V| 54
3. Prim’s Algorithm

• Remarks:
-If |E| is small, then Kruskal’s algorithm (O(m
log m)) is preferred.
-If |E| is large (i.e., |E| = (|V|2)), then Prim’s
algorithm (O(m log n)) is preferred.

12
3. Prim’s Algorithm - Example 2

•Consider the following weighted,

connected, undirected graph

12
3. Prim’s Algorithm - Example 2

• Attach two labels to a vertex:

- the name of the nearest tree vertex, and
- the weight of the corresponding edge.
• Example: select a as starting vertex
- a(-, -)
- b(a, 3)
- c(-, )
12
3. Prim’s Algorithm - Example 2

12
3. Prim’s Algorithm - Example 2

13
3. Prim’s Algorithm - Example 2

13
 Single-Source Shortest-Paths Problem

•Given a weighted connected graph G = (V, E),

each edge has a non-negative weight, m = |E|, |V| =
n + 1, i.e., V = {v0, v1, …, vn}.
•The single-source shortest-paths problem: is to
find all the shortest paths from a given vertex v0
(called source vertex) to all other vertices vi in V.

13
 Single-Source Shortest-Paths Problem

•Similar to the MST algorithm, the set of vertices

is divided into two sets S and V - S, where S
contains all the i nearest neighbors of v0 which
have been found in the first i steps (1 ≤ i ≤ n).
• Thus, in the (i + 1)th step, our job is to find the
(i
+ 1)th nearest neighbor of v0.
• Initially, let S = {v0}.

13
 Single-Source Shortest-Paths Problem

• Let us denote L(vi) as the shortest distance from

v0 to vi presently found, where vi  V - S.
• For example, L(v1) = a and L(v2) = b, where a <
b. Thus, v1 is added into S. That is, S = S  {v1}. If
the edge (v1, v2) with weight c exists, then L(v2)
must be updated as follows.
L(v2) = min{L(v2), L(v1) + c}.

13
 Single-Source Shortest-Paths Problem

•In other words, let u be the latest vertex added to

S (i.e., u  S). Let L(w) denote the presently found
shortest distance from v0 to w (i.e., w  V - S).
•If the edge (u, w) exists, let c(u, w) denote the
length of the edge connecting u and w, then L(w)
will need to be updated according to the following
formula:
L(w) = min{L(w), L(u) + c(u, w)}.

13
4. Dijkstra’s Algorithm

[R. C. T. Lee]
Algorithm: Dijkstra’s algorithm to generate
single-source shortest paths
Input: A weighted connected graph G = (V, E)
and a source vertex v0. For each edge
(u, v)  E, there is a non-negative number
c(u, v) associated with it. |V| = n + 1.
Output: For each v  V, the length of a shortest path
from v0 to v. 66
4. Dijkstra’s Algorithm

S = {v0} // (1)
for i = 1 to n do // consider v1, v2, …, vn // n times
{
if ((v0, vi)  E) L(vi) = c(v0, vi); // (1)
// vi is adjacent to v0, fringe vertex
else L(vi) = ∞; // vi is not adjacent to
v0 // (1)
} // (n) 67
4. Dijkstra’s Algorithm

for i = 1 to n do // n times
{
Choose u from V - S such that L(u) is the smallest
// greedy choice, // (n)
S = S  {u} // put u into S, move u from V - S to S
for all w in V - S do // (m) throughout algorithm
if (u, w)  E, then // u is latest vertex added to S
L(w) = min{L(w), L(u) + c(u, w)} // update
L(w)
} // (n2) 68
4. Dijkstra’s Algorithm - Example 1

•Consider the following weighted,

connected, directed graph

13
4. Dijkstra’s Algorithm - Example 1

• The shortest paths from vertex v0 are

14
4. Dijkstra’s Algorithm - Example 2

•Consider the following weighted,

connected, undirected graph

14
4. Dijkstra’s Algorithm - Example 2

• Attach two labels to a vertex:

-the name of the parent of the vertex on the
shortest path from the source to this vertex found
so far, and
-the length of the shortest path from the source
to this vertex found so far.
• Example: select a as source vertex
- a(-, 0), - b(a, 3), - c(-, )
14
4. Dijkstra’s Algorithm - Example 2

14
4. Dijkstra’s Algorithm - Example 2

14
4. Dijkstra’s Algorithm - Example 2

from a to b : a - b of length 3
from a to d : a - b - d of length 5
from a to c : a - b - c of length
7
from a to e : a - b - d - e of
length 9

14
4. Dijkstra’s Algorithm

Remarks:
• Dijstra’s algorithm does not always
work if the edge weight is negative.
correctly
• The algorithm that is used to solve the negative-
weighted, single-source shortest-paths path
problem is known as the Bellman-Ford’s
algorithm (using dynamic programming).

14
4. Dijkstra’s Algorithm

•Consider the weighted connected

following undirected graph

Dijkstra’s algorithm yields the shortest path from

a to b of length 2, that is longer than the actual
shortest path a - c - b of length 1. This is because
Dijkstra’s algorithm follows the greedy decision
making.
14
4. Dijkstra’s Algorithm - Time Complexity

• Graph is represented by weight matrix

T(n) = (m + n2) = (n2), where n = |V|.
• Graph is represented by adjacency list
T(n) = O(m log n), where m = |E|.

14
Exercises

2.Give a graph illustrating that Dijkstra’s

algorithm may not work for a weighted connected
graph with negative weights.

3.Write a program to implement Huffman coding

method.

14