Introduction to Algorithms

Class 11: Greedy Algorithm 2

Jianxi Gao

Department of Computer Science

Rensselaer Polytechnic Institute
www.gaojianxi.com
Minimum Spanning Tree

Input: an undirected graph G = (V,E), edge weights 𝑤𝑒

Output: A tree T = (V,E’), with 𝐸’ ⊆ 𝐸, that minimizes the sum weights

𝑊𝑒𝑖𝑔ℎ𝑡 𝑇 = ෍ 𝑤𝑒
𝑒∈𝐸′
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1
A B C
2 1 1
1
D E F

G H
1
Kruskal’s algorithm

Procedure Kruskal(𝐺, 𝑤)
Input: A connected undirected graph G = (𝑉, 𝐸) with the edge
weights 𝑤𝑒

Output: A minimum spanning tree defined by the edges 𝑋.

Makeset(x): create a singleton set containing just x.

Find(x): to which set does x belong to

Union(x,y): merge the sets containing x and y

Kruskal’s algorithm
Procedure Kruskal(𝐺(𝑉, 𝐸), 𝑤) Procedure find(𝑥)(find the root of x)
for all vertex v ∈ 𝑉 While 𝑥 ≠ 𝜋 𝑥 : 𝑥 = 𝜋 𝑥
makeset(v) Return 𝑥
𝑋 = {}. Procedure union 𝑥, 𝑦
Sort the edges of 𝐸 by weights. 𝑟𝑥 = 𝑓𝑖𝑛𝑑 𝑥
For all edges u, v ∈ 𝐸, in 𝑟𝑦 = 𝑓𝑖𝑛𝑑 𝑦
increasing order of weight If 𝑟𝑦 = 𝑟𝑥 return
If find u ≠ 𝑓𝑖𝑛𝑑 𝑣 : If rank(𝑟𝑥 ) > 𝑟𝑎𝑛𝑘 𝑟𝑦
Add edge {u, v} to 𝑋. 𝜋 𝑟𝑦 = 𝑟𝑥
Union(u, v). Else:
𝜋 𝑟𝑥 = 𝑟𝑦
Procedure makeset(𝑥)
𝜋 𝑥 = 𝑥 (parent pointer) If rank(𝑟𝑥 ) = 𝑟𝑎𝑛𝑘 𝑟𝑦 :
rank 𝑥 = 0 𝑟𝑎𝑛𝑘 𝑟𝑦 = 𝑟𝑎𝑛𝑘 𝑟𝑦 +1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C 𝜋 𝐴 = 𝐴, rank (A)=0
1 2 𝜋 𝐵 = 𝐵, rank (B)=0
A B C
𝜋(𝐶) = 𝐶, rank (C)=0
3 2 1 𝜋 𝐷 = 𝐷, rank (D)=0
D E F 2 1
𝜋 𝐸 = 𝐸, rank (E)=0
1 4
D E F 𝜋 𝐹 = 𝐹, rank (F)=0
3 𝜋 𝐺 = 𝐺, rank (G)=0
G H 2 4 𝜋 𝐻 = 𝐻, rank (H)=0
2
for all vertex v ∈ 𝑉 G H
1
makeset(v)
Procedure makeset(𝑥)
𝜋 𝑥 = 𝑥 (parent pointer)
rank 𝑥 = 0
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
𝑋 = {}.
Sort the edges of 𝐸 by weights.

For all edges u, v ∈ 𝐸, in increasing order of weight

Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2 find A ≠ 𝑓𝑖𝑛𝑑 𝐵
A B C
3 1
𝑋 = {(𝐴, 𝐵)}.
D E F 2 2 1
Union(A, B).
1 4
D E F
𝐴 = 𝑓𝑖𝑛𝑑 𝐴
3
G H 2 4 𝐵 = 𝑓𝑖𝑛𝑑 𝐵
2
G H If rank(𝐴) > 𝑟𝑎𝑛𝑘 𝐵
1
𝜋 𝐴 =𝐵
If rank(𝐴) = 𝑟𝑎𝑛𝑘 𝐵 :
𝑟𝑎𝑛𝑘 𝐵 = 𝑟𝑎𝑛𝑘 𝐵 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2 find A ≠ 𝑓𝑖𝑛𝑑 𝐵
A B C
3 1
𝑋 = {(𝐴, 𝐵)}.
D E F 2 2 1
Union(A, B).
1 4
D E F
𝐴 = 𝑓𝑖𝑛𝑑 𝐴
3
G H 2 4 𝐵 = 𝑓𝑖𝑛𝑑 𝐵
2
G H If rank(𝐴) > 𝑟𝑎𝑛𝑘 𝐵
1
𝜋 𝐴 =𝐵
If rank(𝐴) = 𝑟𝑎𝑛𝑘 𝐵 :
𝑟𝑎𝑛𝑘 𝐵 = 𝑟𝑎𝑛𝑘 𝐵 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2 find C ≠ 𝑓𝑖𝑛𝑑 𝐹
A B C
3 1
𝑋 = { 𝐴, 𝐵 , (𝐶, 𝐹)}.
D E F 2 2 1
Union(C, F).
1 4
D E F
𝐶 = 𝑓𝑖𝑛𝑑 𝐶
3
G H 2 4 𝐹 = 𝑓𝑖𝑛𝑑 𝐹
2
G H If rank(C) > 𝑟𝑎𝑛𝑘 𝐹
1
𝜋 𝐶 =𝐹
If rank(C) = 𝑟𝑎𝑛𝑘 𝐹 :
𝑟𝑎𝑛𝑘 𝐹 = 𝑟𝑎𝑛𝑘 𝐹 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2 find C ≠ 𝑓𝑖𝑛𝑑 𝐹
A B C
3 1
𝑋 = { 𝐴, 𝐵 , (𝐶, 𝐹)}.
D E F 2 2 1
Union(C, F).
1 4
D E F
𝐶 = 𝑓𝑖𝑛𝑑 𝐶
3
G H 2 4 𝐹 = 𝑓𝑖𝑛𝑑 𝐹
2
G H If rank(C) > 𝑟𝑎𝑛𝑘 𝐹
1
𝜋 𝐶 =𝐹
If rank(C) = 𝑟𝑎𝑛𝑘 𝐹 :
𝑟𝑎𝑛𝑘 𝐹 = 𝑟𝑎𝑛𝑘 𝐹 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2 find D ≠ 𝑓𝑖𝑛𝑑 𝐸
A B C
3 1
𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , (𝐷, 𝐸)}.
D E F 2 2 1
Union(D, E).
1 4
D E F D= 𝑓𝑖𝑛𝑑 𝐷
3
G H 2 4 E= 𝑓𝑖𝑛𝑑 𝐸
2
G H If rank(D) > 𝑟𝑎𝑛𝑘 𝐸
1
𝜋 𝐷 =𝐸
If rank(D) = 𝑟𝑎𝑛𝑘 𝐸 :
𝑟𝑎𝑛𝑘 𝐸 = 𝑟𝑎𝑛𝑘 𝐸 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2 find D ≠ 𝑓𝑖𝑛𝑑 𝐸
A B C
3 1
𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , (𝐷, 𝐸)}.
D E F 2 2 1
Union(D, E).
1 4
D E F D= 𝑓𝑖𝑛𝑑 𝐷
3
G H 2 4 E= 𝑓𝑖𝑛𝑑 𝐸
2
G H If rank(D) > 𝑟𝑎𝑛𝑘 𝐸
1
𝜋 𝐷 =𝐸
If rank(D) = 𝑟𝑎𝑛𝑘 𝐸 :
𝑟𝑎𝑛𝑘 𝐸 = 𝑟𝑎𝑛𝑘 𝐸 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

find G ≠ 𝑓𝑖𝑛𝑑 𝐻
A B C
1 2 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C (𝐺, 𝐻)}.
3 2 1
D E F 2 1
Union(G, H).
1 4
D E F G= 𝑓𝑖𝑛𝑑 𝐺
3
G H 2 4 H= 𝑓𝑖𝑛𝑑 𝐻
2
G H If rank(G) > 𝑟𝑎𝑛𝑘 𝐻
1
𝜋 𝐺 =𝐻
If rank(G) = 𝑟𝑎𝑛𝑘 𝐻 :
𝑟𝑎𝑛𝑘 𝐻 = 𝑟𝑎𝑛𝑘 𝐻 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

find G ≠ 𝑓𝑖𝑛𝑑 𝐻
A B C
1 2 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C (𝐺, 𝐻)}.
3 2 1
D E F 2 1
Union(G, H).
1 4
D E F G= 𝑓𝑖𝑛𝑑 𝐺
3
G H 2 4 H= 𝑓𝑖𝑛𝑑 𝐻
2
G H If rank(G) > 𝑟𝑎𝑛𝑘 𝐻
1
𝜋 𝐺 =𝐻
If rank(G) = 𝑟𝑎𝑛𝑘 𝐻 :
𝑟𝑎𝑛𝑘 𝐻 = 𝑟𝑎𝑛𝑘 𝐻 +1=1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

F = find C ≠ 𝑓𝑖𝑛𝑑 𝐸 = 𝐸
A B C
1 2 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C 𝐺, 𝐻 , (𝐶, 𝐸)}.
3 2 1
D E F 2 1
Union(C, E).
1 4
D E F F= 𝑓𝑖𝑛𝑑 𝐶
3
G H 2 4 E= 𝑓𝑖𝑛𝑑 𝐸
2
G H If rank(C) > 𝑟𝑎𝑛𝑘 𝐸
1
𝜋 𝐶 =𝐸
If rank(C) = 𝑟𝑎𝑛𝑘 𝐸 :
𝑟𝑎𝑛𝑘 𝐸 = 𝑟𝑎𝑛𝑘 𝐸 +1=2
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

F = find C ≠ 𝑓𝑖𝑛𝑑 𝐸 = 𝐸
A B C
1 2 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C 𝐺, 𝐻 , (𝐶, 𝐸)}.
3 2 1
D E F 2 1
Union(C, E).
1 4
D E F F= 𝑓𝑖𝑛𝑑 𝐶
3
G H 2 4 E= 𝑓𝑖𝑛𝑑 𝐸
2
G H If rank(C) > 𝑟𝑎𝑛𝑘 𝐸
1
𝜋 𝐶 =𝐸
If rank(C) = 𝑟𝑎𝑛𝑘 𝐸 :
𝑟𝑎𝑛𝑘 𝐸 = 𝑟𝑎𝑛𝑘 𝐸 +1=2
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

B = find B ≠ 𝑓𝑖𝑛𝑑 𝐸 = 𝐸
A B C
1 2 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C 𝐺, 𝐻 , 𝐶, 𝐸 , (𝐵, 𝐸)}.
3 2 1
D E F 2 1
Union(B, E).
1 4
D E F B= 𝑓𝑖𝑛𝑑 𝐵
3
G H 2 4 E= 𝑓𝑖𝑛𝑑 𝐸
2
G H If rank(B) > 𝑟𝑎𝑛𝑘 𝐸
1
𝜋 𝐵 =𝐸
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

B = find B ≠ 𝑓𝑖𝑛𝑑 𝐸 = 𝐸
A B C
1 2 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C 𝐺, 𝐻 , 𝐶, 𝐸 , (𝐵, 𝐸)}.
3 2 1
D E F 2 1
Union(B, E).
1 4
D E F B= 𝑓𝑖𝑛𝑑 𝐵
3
G H 2 4 E= 𝑓𝑖𝑛𝑑 𝐸
2
G H If rank(B) > 𝑟𝑎𝑛𝑘 𝐸
1
𝜋 𝐵 =𝐸
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1
2 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1 2
A B C find A = 𝑓𝑖𝑛𝑑 𝐷 = 𝐸
3 2 1
D E F 1
1 4
D E F
3
G H 2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1 2
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

A B C
1
A B C find B = 𝑓𝑖𝑛𝑑 𝐶 = 𝐸
3 2 1
D E F 1
1 4
D E F
3
G H 2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

1
A B C
3 2 1 1
1 4
D E F
3
2 4
2
G H
1
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

E = find E ≠ 𝑓𝑖𝑛𝑑 𝐺 = 𝐻
A B C
1 𝑋 = { 𝐴, 𝐵 , 𝐶, 𝐹 , 𝐷, 𝐸 ,
A B C 𝐺, 𝐻 , 𝐶, 𝐸 , 𝐵, 𝐸 , (𝐸, 𝐺)}
3 2 1
D E F 1
Union(E, G).
1 4
D E F E= 𝑓𝑖𝑛𝑑 𝐸
3
G H 2 4 H= 𝑓𝑖𝑛𝑑 𝐺
2
G H If rank(E) > 𝑟𝑎𝑛𝑘 𝐺
1
𝜋 𝐺 =𝐸
Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

E

1
A B C
2 1 1 F D H B
1
D E F

2
C G
G H A
1

Q: What if we do not Rank and just randomly union the nodes, correctness and efficiency?

A: correct, but not efficient.

Kruskal’s Algorithm example

Using Kruskal’s algorithm to find the minimum spanning tree.

If we use a bad strategy,

1
A B C
3 2 1 C F D B
1
1 4
D E F
3 E
2 4
2 H A
G H
1

G
Huffman
encoding
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAE AACD EBAC AABC BCAA DCAE AACD EBAC AABC
Size of the message:
ASCII 8-bit table 40 X 8 = 320 bits

Q: how to decrease the

size?

A: Using our own code.

Huffman encoding

Suppose we want to send the sequence to someone:

BCAA BCAD AACD EBAC AABC BCAA B CAD AACD EBAC AABC

Q: what is minimal digital do we

Character frequency code need to represent 5 letters?
A 16 000 A: 3
B 8 001
C 10 010 Q: what is the size of the message
D 4 011 now?
E 2 100

A: message size: 3 X 40 = 120 A: table size: 5 x 8 + 3 x 5 = 55

A: The total = message size + table size = 175 < 320

Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

Character frequency code

A 16
B 8
C 10
D 4
E 2

2 4 8 10 16
E D B C A
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

2 4 8 10 16
E D B C A
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

14

2 4 8 10 16
E D B C A
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

24

10 14
C
6

2 4 8 16
E D B A
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

40

16 24
A

10 14
C
6

2 4 8
E D B
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

Character frequency code 40

A 16
B 8
C 10 16 24
D 4 A
E 2
10 14
C
6

2 4 8
E D B
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

Character frequency code 40

A 16 0 1
B 8
C 10 16 24
D 4 A 1
0
E 2
10 14
0
C
6 1
0 1
2 4 8
E D B
Huffman encoding

Suppose we want to send the sequence to someone:

BCAA DCAD AACD EBAC AABC BCAA DCAD AACD EBAC AABC

Character frequency code size 40

A 16 0 16 x 1 = 16 0 1
B 8 111 3 x 8 = 24
C 10 10 10 x 2 = 20 16 24
D 4 1101 4 x 4 = 16 A 1
0
E 2 1100 2x4=8
10 14
0
C
message size: 84 1
6
0 1
table size: 5 x 8 +
1+3+2+4+4 = 54 2 4 8
The total = message size + E D B
table size = 137 < 175 < 320
Huffman encoding
Procedure Huffman(𝑓)
Input: An array 𝑓 1 … 𝑛 of frequencies.
Output: An encoding tree with 𝑛 leaves.
Let H be a priority queue of integers, ordered by 𝑓.
for i=0 to n; insert(H,i)
for k=n+1 to 2n-1
i=deletemin(H), j= deletemin(H)
Create a node numbered k with children i,j.
f[k] = f[i] + f[j].
Inser(H,k).
Huffman encoding -- example

for i=0 to n; insert(H,i)

2 4 8 10 16
E D B C A
for k=n+1 to 2n-1
i=deletemin(H), j= deletemin(H)
8 10 16
B C A
Create a node numbered k with children i,j.
f[k] = f[i] + f[j]. 6 8 10 16
Inser(H,k). B C A

2 4
E D
Huffman encoding -- example

6 8 10 16
B C A

2 4
E D
Huffman encoding -- example

6 8 10 16
B C A

2 4
E D
Huffman encoding -- example

10 16
C A
Huffman encoding -- example

14 10 16
C A

6 8
B

2 4
E D
10 14 16
C A

6 8
B

2 4
E D
Huffman encoding -- example

14 10 16
C A

6 8
B

2 4
E D
Huffman encoding -- example

24 16
A

14 10
C 24 16
A
6 8
B 10 14
C
2 4
E D
6 8
B

2 4
E D
Huffman encoding -- example

40 40

24 16 16 24
A A

14 10 10 14
C C

6 8 6 8
B B

2 4 2 4
E D E D
Huffman encoding -- example

40 40
0 1 0 1

24 16 16 24
0 A A 0
1 1

14 10 10 14
C C
0 0 1
1
6 8 6 8
0 1 B 0 B
1
2 4 2 4
E D E D
Huffman encoding -- example

40
0 1

24 16
0 A
1

14 10
0 C Character Frequency Code 1
1
A 16 1
6 8 B 8 001
0 1 B
C 10 01
2 4 D 4 0001
E D
E 2 0000
Huffman encoding -- example

Table size = 5*8 + 1+3+2+4+4 = 54

40
Message size = 1*16+3*8+2*10+4*4+4*2 = 84 0 1

Total size = Table size + Message size = 138 16 24

A 0 1

Character Frequency Code 2 Code 1 10 14

A 16 0 1 C
0 1
B 8 111 001
6 8
C 10 10 01
0 B
D 4 1101 0001 1
E 2 1100 0000 2 4
E D
Huffman encoding -- example

Character Frequency Code 1

A 4
B 7
C 3
D 2
E 4

15 mins
Huffman encoding -- example
letter Frequency Code
2 3 4 4 7 A 4
D C A E B
B 7
C 3
D 2
E 4
Huffman encoding -- example
letter Frequency Code
2 3 4 4 7 A 4
D C A E B
B 7
C 3
D 2
E 4
Huffman encoding -- example
letter Frequency Code
20
A 4 00
0 1
B 7 11
8 12 C 3 101
0 1 0 1
D 2 100
4 4 5 7 E 4 01
A E 0 1 B
2 3
D C
Table size = 5*8 + 2+2+3+3+2 = 52

Message size = 4*2+7*2+3*3+2*3+4*2 = 45

Total size = Table size + Message size = 97

Huffman encoding -- example
letter Frequency Code
20
A 4 00
0 1
B 7 11
8 12 C 3 101
0 1 0 1
D 2 100
4 4 5 7 E 4 01
A E 0 1 B
2 3
D C

Decoding: 01111110110000101 5 mins

Huffman encoding -- Summary

• David Huffman in 1951

• Encoding follows the prefix rule: no codeword can

be a prefix of another codeword.

• Most frequent letter will get small code and least

frequent letter will get large code
Huffman encoding – Prefix property
Example
?
letter Frequency Code 0 1
A 0 A B
B 1 0 1
C 01 ? C
Q: For a message 001, what is the decoding?
A: AC or AAB
Q: Why we have this problem?
A: because 0(A) is the prefix of 01(C).

Any prefix-free encoding can be represented by a full binary tree– that is, a binary tree in
which every node has either zero or two children – where the symbols are at the leaves, and
where each codeword is generated by a path from root to leaf, interpreting left as 0 and right as
1.