SOURCE CODING

1. Definition of Source Coding Terms

(a) Code Length
(b) Code efficiency
(c) Code redundancy
2. Source Coding Theory
3. Classification of Codes
4. Entropy Encoding
(a) Shannon-Fano
(b) Huffman

ECE 416 – Digital Communication

Friday, 23 March 2018
1
SYLLABUS

2
 DIGITAL COMMUNICATION

Baseband signal
Signal Source Source Channel Modulator
& Transducer Encoder Encoder

Microphone Sampling Error Control

TV Camera Companding
Flow-sensor Encrypting
etc
Channel
Free space
Co-axial cable
Baseband signal Water
fibre

Signal Output Message Channel Demodulator

Recovery decoder
 SOURCE CODING DEFINITION & OBJECTIVE

1. Sources Coding is the conversion of the

output of a Discrete Memory-less Source
(DMS) into a binary sequence.
2. The Objective of Source Coding is to
minimize the average bit rate required to
represent signal by reducing the redundancy
of the information source.

4
 CLASSIFICATION OF INFORMATION SOURCES

Information Sources fall into two categories:

a) Memory sources where the current symbol

depends on the previous symbols,

b) Memory-less sources where current symbols

are independent of previous symbols.

5
 CODE LENGTH DEFINITION

• Assume X is a DMS with finite entropy H(X)

and alphabet {x1,x2,..,xn) and with
corresponding probabilities P(xi) for i =
1,2,..,m.
• If the binary code word assigned to a symbol xi
by the source encoder has length n, then the
codeword length is the number of binary
digits in the code word.

6
 OTHER DEFINITIONS

1. The average Codeword Length is given by:

m
L   P ( xi )ni
i 1

2. Code Efficiency is defined as:

Lmin
 
L
where Lmin is the minimum possible value of the average
code length, L.
3. The code is said to be efficient when the code length,η
tends to 1.

3. Code redundancy is defined as:

  1  7
 SOURCE CODING THEOREM

• The Source coding theorem states that a DMS

X with entropy H(X), and average code length
L per symbol is bound by
L ≥ H(X)
• Further, that L can be made closer to H(X) as
desired through some suitable code.

8
 CLASSIFICATION OF CODES

1. Fixed Length Code:

A code whose code length is fixed.

9
 CLASSIFICATION OF CODES

2. Variable Length Code

A code whose length varies for different
symbols.

10
 CLASSIFICATION OF CODES

3. Distinct Code is a code in which each code

word is distinguishable from other code words

11
 CLASSIFICATION OF CODES

4. Prefix-free Codes
Codes in which no codeword can be formed
by adding code symbols to another codeword

12
 CLASSIFICATION OF CODES

5. Uniquely Decodable Codes

A code in which the original sequence can be
reconstructed perfectly from the encoded
binary sequence.

13
 CLASSIFICATION OF CODES

6. Instantaneous Codes
1. A code which has the end of any codeword is
recognizable without examining subsequent code
symbols.
2. Instantaneous Codes have the property that no
codeword is a prefix of another codeword.

14
 CLASSIFICATION OF CODES

7. Optimal Codes:
A code is said to be optimal if it is
instantaneous and has a minimum average
Length, 𝐿𝑚𝑖𝑛

15
 WORKED EXAMPLE - 1

1. A Discrete Memory-less Source X has alphabet

{x1, x2} and associated probabilities, P(x1)=0.9,
P(x2)=0.1 where the symbols are encoded as:

Find the efficiency and redundancy of the code

16
 SOLUTION

Entropy is:

Code efficiency is

Code redundancy is:

17
 EXAMPLE 2

2. A Discrete Memory-Less Source X has

alphabet {x1,x2,x3,x4} and a source coding as
shown below.
xi P(xi) Code
X1 0.81 0
X2 0.09 10
X3 0.09 110
X4 0.01 111

Determine the efficiency and redundancy of

the code.
18
 SOLUTION

• The average code word length, L is:

• The entropy H(X) is given by:

H(X)

• Code Efficiency is therefore:

H ( X ) 0.938
   0.727
L 1.29

• Code redundancy is therefore:

  1    0.273  27%
19
 ENTROPY CODING

1. Entropy coding refers to the design of a

variable length code such that its average
codeword length approaches the entropy.
There are two main type of entropy coding,
i.e
(a) Shannon-Fano Coding
(b) Huffman Coding

20
 SHANNON-FANO CODING

• Named after Claude Shannon and Robert Fano, is

a technique for constructing a prefix-code based
on a set of symbols and their probabilities
(estimated or measured).
• Shannon–Fano coding is suboptimal in the sense
that it does not achieve the lowest possible
expected code word length like Huffman coding.
• The technique was first proposed in Shannon's "A
Mathematical Theory of Communication", his
1948 article introducing the field of information
theory.

21
 SHANNON-FANO CODING

The Shannon-Fano Code is generated by using

the following procedure:
1. List the source symbols in the order of
decreasing probability.
2. Partition the set into two sets with as nearly
equal probabilities as possible and assign 0 to the
upper set and 1 to the lower set.
3. Continue with the process each time partitioning
the sets with as nearly equal probabilities as
possible until further partitioning of sets is not
possible.
22
 SHANNON-FANO CODING
1. Original symbol list and Probabilities
x(i) P(x(i))
x1 0.05
x2 0.30
x3 0.08
x4 0.25
x5 0.20
x6 0.12

2. Sort in the order of decreasing probabilities

x(i) P(x(i))
x2 0.30
x4 0.25
x5 0.20
x6 0.12
x3 0.08
x1 0.05

3. Partition into 2 sets above and below 0.5 (approx.)

x(i) P(x(i)) Step 1

x2 0.30 0
Assign 0
x4 0.25 0

x5 0.20 1

x6 0.12 1

x3 0.08 1
Assign 1
x1 0.05 1

23
 SHANNON-FANO CODING

4. Partition into 2 sets above and below the middle points

x(i) P(x(i)) Step 1 Step 2
x2 0.30 0 0
x4 0.25 0 1
x5 0.20 1 0
x6 0.12 1 1
x3 0.08 1 1
Remaining
x1 0.05 1 1

5. Partition the remaining into 2 sets above and below the middle points
x(i) P(x(i)) Step 1 Step 2 Step 3
x2 0.30 0 0
x4 0.25 0 1
x5 0.20 1 0
x6 0.12 1 1 0
x3 0.08 1 1 1
x1 0.05 1 1 1 Remaining

5. Partition the remaining into 2 sets above and below the middle points
x(i) P(x(i)) Step 1 Step 2 Step 3 Step 4 Code
x2 0.30 0 0 00
x4 0.25 0 1 01
x5 0.20 1 0 10
x6 0.12 1 1 0 110
x3 0.08 1 1 1 0 1110
24
x1 0.05 1 1 1 1 1111
 SHANNON-FANO CODING EXAMPLE 1

A DMS has four symbols, i.e x1. x2, x3 and x4 with

probabilities P(x1)=1/2, P(x2)=1/4, P(x3)=P(x4)=1/8.

Construct the Shannon-Fano Code and determine the

code efficiency.

25
 SHANNON-FANO CODE-EXAMPLE 1 - SOLUTION

1. Shannon-Fano Code
x(i) P(x(i)) Step 1 Step 2 Step 3 Code
x1 0.500 0 0
x2 0.250 1 0 10
x3 0.125 1 1 0 110
x4 0.125 1 1 1 111

𝐼 𝑥1 = 𝑙𝑜𝑔2 2 = 1 = n1
𝐼 𝑥2 = 𝑙𝑜𝑔2 4 = 2 = n2
𝐼 𝑥3 = 𝑙𝑜𝑔2 8 = 3 = n3
𝐼 𝑥4 = 𝑙𝑜𝑔2 8 = 3 = n4
4
1 1 1 1
𝐻 𝑋 = ෍ 𝑃 𝑥𝑖 𝐼 𝑥𝑖 = 1 + 2 + 3 + 3 = 1.75
2 4 8 8
𝑖=1
4
1 1 1 1
𝐿 = ෍ 𝑃 𝑥𝑖 𝑛𝑖 = 1 + 2 + 3 + 3 = 1.75
2 4 8 8
𝑖=1
1. Efficiency
𝐻(𝑋)
𝑛= = 1 𝑜𝑟 100% 26
𝐿
 HUFFMAN CODE

1. Huffman coding is a lossless data compression

algorithm using variable length codes.
2. Lengths of the assigned codes are based on the
frequencies of corresponding characters.
a) Most frequent character gets the shortest code.
b) Least frequent character gets the longest code.
3. The variable-length codes assigned to input
characters are Prefix Codes, i.e
– the codes are assigned in such a manner that the
code assigned to one character is not prefix of code
assigned to any other character.

27
 HUFFMAN CODE

1. The Huffman Code results in a code that is optimal

and is therefore a code with the highest efficiency.
2. The Huffman procedure is based on the following
observations regarding optimum prefix codes.
a) Symbols that occur more frequently (have a higher
probability of occurrence) will have the shortest code
words.
b) The two symbols that occur least frequently will have the
same length.
c) Code words corresponding to the two lowest probability
symbols differ only in the last bit.

28
 STEPS IN HUFFMAN CODING

There are mainly two major

steps in Huffman Coding, i.e
1. Build a Huffman Tree from
input characters.
2. Traverse the Huffman Tree
and assign codes to
characters. Character Code
a 0
b 111
d 1011
d 100
r 110
29
! 1010
 ASSIGN CODES

1. Start encoding from the last reduction on the

tree.
2. Assign 0 to the first digit of the code words for
all symbols associated with the first probability.
Assign 1 to the second probability.
3. Assign 0 and 1 to the second digit of the two
probabilities that were combined in previous
reduction step while retaining all assignments in
the previous stage
4. Repeat the process until the first column is
reached.

30
 USING THE HUFFMAN CODE IN PRACTICE
• Assume that you a character file that you would like to compress. By parsing
through the list, a computer stablishes that there are 100,000 characters with a
frequency of occurrence as shown below.
Character Frequency
A 45,000
B 13,000
C 12,000
D 16,000
E 9,000
F 5,000
Total 100,000

• Determine a code that encodes the file using as few bits as possible.

SOLUTION 1: USING A HAMMER

A fixed code scheme would require 3 bits per character., i.e 2𝑣 ≥ 6
Therefore using this code we will store 3 x 100,000 = 300kbits
----------
If on the other hand we used a byte to store each character, we would have
required a file of size 8 x 100,000 or 800kbits
31
 HUFFMAN CODE FOR THE EXAMPLE

1. Average Code Length, L = 2.24

2. Bits required = L x 100,000 = 224,000 bits
32
 HOMEWORK

• Determine the Huffman code for the following

codes and their corresponding probabilities.
Character Probability Character Probability

A 0.05 F 0.3 0.3 0.3 0.3 0.4 0.6

B 0.15 C 0.2 0.2 0.2 0.3 0.3

C 0.2 G 0.1 0.1 0.2 0.2 0.3

D 0.05 B 0.15 0.15 0.15 0.2

E 0.15 E 0.15 0.15 0.15

F 0.3 A 0.05 0.1

G 0.1 D 0.05

33
 FIRST, CREATE THE TREE

Character Probability Character Probability

F 0.3 0.3 0.3 0.3 0.4 0.6
A 0.05
C 0.2 0.2 0.2 0.3 0.3
B 0.15
B 0.15 0.15 0.2 0.2 0.3
C 0.2
E 0.15 0.15 0.15 0.2
D 0.05
G 0.1 0.1 0.15
E 0.15

F 0.3 A 0.05 0.1

G 0.1 D 0.05

34
USE ONLINE CALCULATOR TO CROSS-CHECK

35