ECE 515

Information Theory

Distortionless Source Coding 1

1
Source Coding

Output Alphabet
Y={y1,…,yJ}

Source Encoder

2
 Source Coding
Two requirements
1. The source sequence can be recovered from
the encoded sequence with no ambiguity.
2. The average number of output symbols per
source symbol is as small as possible.

3
Variable Length Codes

Output Alphabet
Y={y1,…,yJ}

Lengths

4
Variable Length Codes

5
 Variable Length Codes
• Let K = 4, X = {x1, x2, x3, x4}, J = 2
• Prefix code (also prefix-free or instantaneous)
C1={0,10,110,111}
• Example sequence of codewords:
001110100110
• Decodes to:
0 0 111 0 10 0 110
x 1 x1 x4 x1 x2 x1 x3

6
 Instantaneous Codes
• Definition:
A uniquely decodable code is said to be
instantaneous if it is possible to decode each
codeword in a sequence without reference to
succeeding codewords.

A necessary and sufficient condition for a code to be

instantaneous is that no codeword is a prefix of
some other codeword.

7
 Variable Length Codes
• Uniquely decodable code (which is not prefix)
C2={0,01,011,0111}
• Example sequence of codewords:
001110100110
• Decodes to:
0 0111 01 0 011 0
x1 x4 x2 x1 x3 x1

8
 Variable Length Codes
• Non-singular code (which is not uniquely
decodable)
C3={0,1,00,11}
• Example sequence of codewords:
001110100110
• Decodes to:
0 0 1 1 1 0 1 0 0 1 1 0
x1 x1 x2 x2 x2 x1 x2 x1 x1 x2 x2 x1
00 11 1 0 1 00 11 0
x3 x4 x2 x1 x2 x3 x4 x1
9
 Variable Length Codes
• Singular code
C4={0,10,11,10}
• Example sequence of codewords:
001110100110
• Decodes to:
0 0 11 10 10 0 11 0
x 1 x1 x3 x2 x2 x1 x3 x1
x1 x1 x3 x4 x2 x1 x3 x1

10
Variable Length Codes

Output Alphabet
Y={y1,…,yJ}

Lengths

11
Variable Length Codes

12
Average Codeword Length

K
L(C) = ∑ p(xk )lk
k =1

13
 Two Binary Prefix Codes
• Five source symbols: x1, x2, x3, x4, x5
• K = 5, J = 2

• c1 = 0, c2 = 10, c3 = 110, c4 = 1110, c5 = 1111

– codeword lengths 1,2,3,4,4
• c1 = 00, c2 = 01, c3 = 10, c4 = 110, c5 = 111
– codeword lengths 2,2,2,3,3

14
Kraft Inequality for Prefix Codes

15
Code Tree

16
 Five Binary Codes

Source
symbols Code A Code B Code C Code D Code E

x1
x2
x3
x4

17
 Ternary Code Example
• Ten source symbols: x1, x2, …, x9, x10
• K = 10, J = 3

• lk = 1,2,2,2,2,2,3,3,3,3
• lk = 1,2,2,2,2,2,3,3,3
• lk = 1,2,2,2,2,2,3,3,4,4

18
Average Codeword Length Bound

H(X)
L(C) ≥
log b J

X C
Information Variable Length
Source Source Encoder

H(X) L(C)logbJ

19
 Four Symbol Source
Information
X
Source

• p(x1) = 1/2 p(x2) = 1/4 p(x3) = p(x4) = 1/8

• H(X) = 1.75 bits

x1 0 x1 00
x2 10 x2 01
x3 110 x3 10
x4 111 x4 11
L(C) = 1.75 bits L(C) = 2 bits
20
 Code Efficiency
H(X)
=ζ ≤1
L(C)log b J

• First code ζ = 1.75/1.75 = 100%

• Second code ζ = 1.75/2.0 = 87.5%

21
 Compact Codes
A code C is called compact for a source X if its
average codeword length L(C) is less than or
equal to the average length of all other uniquely
decodable codes for the same source and
alphabet Y size J.

22
Codeword Lengths

K
H(X) = −∑ p(xk )logp(xk )
k =1

K
L(C) = ∑ p(xk )lk
k =1

23
 Upper and Lower Bounds for a
Compact Code

H(X) H(X)
≤ L(C) < +1
log b J log b J

if b = J

H(X) ≤ L(C) < H(X) + 1

24
 The Shannon Algorithm
• Order the symbols from largest to smallest
probability
• Choose the codeword lengths according to
lk =  − log J p(xk )
• Construct the codewords according to the
cumulative probability Pk
k −1
Pk = ∑ p(xi )
i =1
expressed as a base J number

25
 Example
• K = 10, J = 2
• p(x1) = p(x2) = 1/4
• p(x3) = p(x4) = 1/8
• p(x5) = p(x6) = 1/16
• p(x7) = p(x8) = p(x9) = p(x10) = 1/32

26
Converting Decimal Fractions to Binary
• To convert a fraction to binary, multiply it by 2
• If the integer part is 1, the binary digit is 1,
otherwise it is 0
• Delete the integer part
• Continue multiplying by 2 and obtaining
binary digits until the resulting fractional part
is 0 or the required number of binary digits
have been obtained

27
 Example
• Convert 5/8 = 0.62510 to binary
• 2 × 0.625 = 1.25 = 1 + 0.25 MSB
• 2 × 0.250 = 0.50 = 0 + 0.50
• 2 × 0.500 = 1.00 = 1 + 0.00 LSB
• 0.62510 = 0.1012

28
Example

29
 Shannon Algorithm

• p(x1) = .4 p(x2) = .3 p(x3) = .2 p(x4) = .1

• H(X) = 1.85 bits

Shannon Code Alternate Code

x1 00 x1 0
x2 01 x2 10
x3 101 x3 110
x4 1110 x4 111
L(C) = 2.4 bits L(C) = 1.9 bits
ζ = 77.1% ζ = 97.4%
30
Shannon’s Noiseless Coding Theorem

NH(X) NH(X)
≤ L N (C) < +1
log b J log b J

H(X) L N (C) H(X) 1

≤ < +
log b J N log b J N

31
Shannon’s Noiseless Coding Theorem

If b = J

NH(X) ≤ L N (C) < NH(X) + 1

L N (C) 1
H(X) ≤ < H(X) +
N N

32
Robert M. Fano (1917-2016)

33
 The Fano Algorithm
• Arrange the symbols in order of decreasing
probability
• Divide the symbols into J approximately
equally probable groups
• Each group receives one of the J code symbols
as the first codeword symbol
• This division process is repeated within the
groups as many times as possible

34
Example

35
Shannon Algorithm vs Fano Algorithm

• p(x1) = .4 p(x2) = .3 p(x3) = .2 p(x4) = .1

• H(X) = 1.85 bits

Shannon Code Fano Code

x1 00 x1 0
x2 01 x2 10
x3 101 x3 110
x4 1110 x4 111
L(C) = 2.4 bits L(C) = 1.9 bits
ζ = 77.1% ζ = 97.4%
36
Upper Bound for the Fano Code J∈{2,3}
H(X)
L(C) ≤ + 1 − pmin
log b J
where pmin is the smallest nonzero symbol
probability
if b = J
L(C) ≤ H(X) + 1 − pmin
37
David A. Huffman (1925-1999)

38
• ``It was the most singular moment in my life.
There was the absolute lightning of sudden
realization.’’
– David Huffman

• ``Is that all there is to it!’’

– Robert Fano

39
 The Binary Huffman Algorithm
1. Arrange the K symbols of the source X in order of
decreasing probability.
2. Assign a 1 to the last digit of the Kth codeword cK and a 0
to the last digit of the (K-1)th codeword cK-1. Note that
this assignment is arbitrary.
3. Form a new source X´ with x´k = xk, k = 1, …, K-2, and
x´K-1 = xK-1 U xK p(x´K-1) = p(xK-1) + p(xK)
4. Set K = K-1.
5. Repeat Steps 1 to 4 until all symbols have been
combined.

To obtain the codewords, trace back to the original

symbols.
40
 Five Symbol Source
• p(x1)=.35 p(x2)=.22 p(x3)=.18 p(x4)=.15 p(x5)=.10

• H(X) = 2.2 bits

41
L(C) = 2.25 bits ζ = 97.8%
42
 Shannon and Fano Codes
• p(x1)=.35 p(x2)=.22 p(x3)=.18 p(x4)=.15 p(x5)=.10
• H(X) = 2.2 bits
Shannon Code Fano Code
x1 00 x1 00
x2 010 x2 01
x3 100 x3 10
x4 110 x4 110
x5 1110 x5 111
L(C) = 2.75 bits L(C) = 2.25 bits
ζ = 80.4% ζ = 97.8%
43
Huffman Code for the English Alphabet

44
 Six Symbol Source
• p(x1)=.4 p(x2)=.3 p(x3)=.1 p(x4)=.1 p(x5)=.06
p(x6)=.04
• H(X) = 2.1435 bits
First Code Second Code
x1 1 x1 1
x2 00 x2 00
x3 0100 x3 010
x4 0101 x4 0110
x5 0110 x5 01110
x6 0111 x6 01111
45
 Second Five Symbol Source
• p(x1)=.4 p(x2)=.2 p(x3)=.2 p(x4)=.1 p(x5)=.1
• H(X) = 2.1219 bits

x1 x1
x2 x2
x3 x3
x4 x4
x5 x5
C1 C2
46
Second Five Symbol Source

C1 C2
x1 0 11
x2 10 01
x3 111 00
x4 1101 101
x5 1100 100

Which code is preferable?

47
 Second Five Symbol Source
• p(x1)=.4 p(x2)=.2 p(x3)=.2 p(x4)=.1 p(x5)=.1
• H(X) = 2.122 bits L(C) = 2.2 bits

• variance of code C1
𝜎𝜎12 = 0.4(1-2.2)2+0.2(2-2.2)2+0.2(3-2.2)2+0.2(4-2.2)2 = 1.36
• variance of code C2
𝜎𝜎22 = 0.8(2-2.2)2+0.2(3-2.2)2 = 0.16

48
 Midterm Test
• Friday, October 21, 2022
• During class time (11:30 – 12:20)
• Counts for 20% of the final mark
• Aids allowed
– One page of notes on 8.5” × 11.5” paper (both
sides)
– Calculator
• Cellphones, tablets, laptops, or any other
electronic devices are NOT ALLOWED
49
 Nonbinary Codes
• The Huffman algorithm for nonbinary codes
(J>2) follows the same procedure as for binary
codes except that J symbols are combined at
each stage.
• This requires that the number of symbols in
the source X is K’=J+c(J-1), K’≥K

K − J 
c= 
 J − 1 
50
 Nonbinary Example
• J=3 K=6

• p(x1)=1/3 p(x2)=1/6 p(x3)=1/6 p(x4)=1/9 p(x5)=1/9

p(x6)=1/9

• H(X) = 1.544 trits

51
 Nonbinary Example
• J=3 K=6

K − J 
• c=  =2 so K’=J+c(J-1)=3+2(2)=7
 J −1 

• Add an extra symbol x7 with p(x7)=0

• p(x1)=1/3 p(x2)=1/6 p(x3)=1/6 p(x4)=1/9 p(x5)=1/9

p(x6)=1/9 p(x7)=0

52
Nonbinary Example with an Extra Symbol
x1 1
x2 00
x3 01
x4 02
x5 20
x6 21
x7 22
L(C) = 1.667 trits
H(X) = 1.544 trits
ζ = 92.6%
53
Nonbinary Example with no Extra Symbol
x1 1
x2 01
x3 02
x4 000
x5 001
x6 002

L(C) = 2.0 trits

H(X) = 1.544 trits
ζ = 77.2%
54
 Codes for Different Output Alphabets
• K=13

• p(x1)=1/4 p(x2)=1/4
p(x3)=1/16 p(x4)=1/16 p(x5)=1/16 p(x6)=1/16
p(x7)=1/16 p(x8)=1/16 p(x9)=1/16
p(x10)=1/64 p(x11)=1/64 p(x12)=1/64 p(x13)=1/64

• J=2 to 13

55
Codes for Different Output Alphabets
J
p(xi) xi

x1
x2
x3
x4
x5
x6
x7
x8
x9
x10
x11
x12
x13

L(C)

56
Codes for Different Output Alphabets
J L(C)
2 3.125
3 2.047
4 1.563
5 1.438
6 1.359
7 1.250
8 1.188
9 1.125
10 1.063
11 1.047
12 1.031
13 1.000

57
Codes for Different Output Alphabets
J L(C) ζ
2 3.125 1.000
3 2.047 0.963
4 1.563 1.000
5 1.438 0.936
6 1.359 0.889
7 1.250 0.891
8 1.188 0.877
9 1.125 0.876
10 1.063 0.885
11 1.047 0.863
12 1.031 0.845
13 1.000 0.844

58
 Code Efficiency
ζ

J
59
Binary and Quaternary Codes
x1 00 0
x2 01 1
x3 1000 20
x4 1001 21
x5 1010 22
x6 1011 23
x7 1100 30
x8 1101 31
x9 1110 32
x10 111100 330
x11 111101 331
x12 111110 332
x13 111111 333

60
 Huffman Codes
• Symbol probabilities must be known a priori
• The redundancy of the code
L(C)-H(X) (for J=b)
is typically nonzero
• Error propagation can occur
• Codewords have variable length

61
 Variable to Fixed Length Codes

M sourcewords M ≤ JL
Lengths m1,m2,…,mM
Variable to fixed
length encoder
Fixed length L

62
 Variable to Fixed Length Codes
• Two questions:
1. What is the best mapping from sourcewords to
codewords?
2. How to ensure unique encodability?

63
 Average Bit Rate
average codeword length
ABR =
average sourceword length
L
=
L(S)
M
L(S) = ∑ p(si )mi
i =1

M - number of sourcewords
si - sourceword i
mi - length of sourceword i
p(si) - probability of sourceword i
64
 Average Bit Rate
• For fixed to variable length codes
average codeword length
ABR =
average sourceword length
L(C) L N (C)
= or
1 N
• Design criterion: minimize L(C) or LN(C)
– minimize the ABR

65
 Variable to Fixed Length Codes
• Design criterion: minimize the Average Bit Rate

L
ABR=
L(S)

• ABR ≥ H(X) (L(C) ≥ H(X) for fixed to variable

length codes)
• L(S) should be as large as possible so that the
ABR is close to H(X)

66
 Code Efficiency
• Fixed to variable length codes
H(X)
ζ= ≤1
L(C)

• Variable to fixed length codes

H(X)
ζ= ≤1
ABR

67
Binary Tunstall Code K=3, L=3
Let x1 = a, x2 = b and x3 = c

Unused codeword is 111

68
 Tunstall Codes
Tunstall codes must satisfy the Kraft inequality
M

∑ ≤1
K − mi

i =1

M - number of sourcewords
K - source alphabet size
mi - length of sourceword i

69
 Binary Tunstall Code Construction
• Source X with K symbols
• Choose a codeword length L where 2L > K
1. Form a tree with a root and K branches labelled
with the symbols
2. If the number of leaves is greater than 2L - (K-1),
go to Step 4
3. Find the leaf with the highest probability and
extend it to have K branches, go to Step 2
4. Assign codewords to the leaves

70
K=3, L=3
p(a) = .7, p(b) = .2, p(c) = .1

71
H(X)
ζ = H(X)/ABR = 84.7%

72
The Codewords
aaa 000
aab 001
aac 010
ab 011
ac 100
b 101
c 110

73
• What if a or aa is left at the end of the
sequence of source symbols?
– there are no corresponding codewords
• Solution: use the unused codeword 111
– a 1110 or 111 000
– aa 1111 or 111 001

74
 Tunstall Codes for a Binary Source
• L = 3, K = 2, J = 2, p(x1) = 0.7, p(x2) = 0.3
• JL = 8
Seven sourcewords Eight sourcewords Codewords
x1x1x1x1x1 x1x1x1x1x1 000
x1x1x1x1x2 x1x1x1x1x2 001
x1x1x1x2 x1x1x1x2 010
x1x1x2 x1x1x2 011
x1x2 x1x2x1 100
x2x1 x1x2x2 101
x2x2 x2x1 110
x2x2 111

75
• The end of the sequence of source symbols can
be
x1, x2, x1x1 , x1x1x1, or x1x1x1x1
• With M=7 sourcewords the codeword 111 is
unused so they can be assigned as follows
– x1 111 000
– x2 111 001
– x1x1 111 010
– x1x1x1 111 011
– x1x1x1x1 111 100
76
 Huffman Code for a Binary Source
• N = 3, K = 2, p(x1) = 0.7, p(x2) = 0.3
• Eight sourcewords
• A = x1x1x1 p(A) = .343 00
• B = x1x1x2 p(B) = .147 11
• C = x1x2x1 p(C) = .147 010
• D = x2x1x1 p(D) = .147 011
• E = x2x2x1 p(E) = .063 1000
• F = x2x1x2 p(F) = .063 1001
• G = x1x2x2 p(G) = .063 1010
• H = x2x2x2 p(H) = .027 1011

77
 Code Comparison
• H(X) = .8813
• Tunstall Code L=3 (7 codewords)
ABR = .9762 ζ = 90.3%
• Tunstall Code L=3 (8 codewords)
ABR = .9138 ζ = 96.4%
• Huffman Code N=1 (2 codewords)
L(C) = 1.0 ζ = 88.1%
• Huffman Code N=3 (8 codewords)
L3(C)/3 = .9087 ζ = 97.0%

78
 Error Propagation
• Received Huffman codeword sequence
00 11 00 11 00 11 …
A B A B A B …

• Sequence with one bit error

011 1001 1001 1 …
D F F …

79
 Error Propagation
• The corresponding Tunstall codeword
sequence
000 110 001 000 110 001 …
x1x1x1x1x1 x2x1 x1x1x1x1x2 …

• Sequence with one bit error

010 110 001 000 110 001 …
x1x1x1x2 x2x1 x1x1x1x1x2 …

80