Viet Nam National University Ho Chi Minh City

University of Science
Faculty of Electronics & Telecommunications

Chapter 7:

Introduction to Information Theory

Dang Le Khoa
Email: [email protected]
 Outline
⚫ Basic concepts
⚫ Information
⚫ Entropy
⚫ Source Encoding

11/12/2024 2
 Basic concepts
⚫ Block diagram of digital communication system

Source Source Channel

Modulator
encoder encoder

Noisy
Channel

Destination Source Channel

Demodulator
decoder decoder

11/12/2024 3
 What is Information Theory?

⚫ Information theory provides a quantitative measure of source

information, the information capacity of a channel
⚫ Dealing with coding as a means of utilizing channel capacity
for information transfer
⚫ Shannon’s coding theorem (Shannon’s noisy channel coding
theorem):
“If the rate of information from a source does not exceed the
capacity of a communication channel, then there exists a
coding technique such that the information can be transmitted
over the channel with an arbitrarily small probability of error,
despite the presence of error”

11/12/2024 4
 Information measure

⚫ Information Theory: how much information

– … is contained in a signal?
– … can a system generate?
– … can a channel transmit?
⚫ Information is the commodity produced by the source for
transfer to some user at the destination
⚫ The less likely the message, the more information it conveys
⚫ How is information mathematically defined?

11/12/2024 5
 Information

⚫ Let xj be an event with p(xj) is the probability of the event

that xj is selected for transmission
⚫ If xj occurred, we have
1
I ( x j ) = log a = − log a p( x j ) (1)
p( x j )
units of information
◼ I(xj) is called self-information
 I(xj)0 for 0 p(xj)1
 I(xj)→0 for p(xj)→1
 I(xj)>I(xi) for p(xj)<p(xi)

11/12/2024 6
 Information

⚫ The base of the logarithm

– 10 →the measure of information is hartley
– e →the measure of information is nat
– 2 →the measure of information is bit

⚫ Examples: A random experiment with 16 equally likely outcomes:

– The information associated with each outcomes is:
I(xj)=-log2(1/16)=log216=4 bits
– The information is greater than one bit, since the probability of each
outcome is much less than ½.

11/12/2024 7
 Entropy and Information rate
⚫ Consider an information source emitting a sequence of symbols from
the set X={x1,x2..,xM}
⚫ Each symbol xi is treated as a message with probability p(xi) and self-
information I(xi)
⚫ This source has an average rate of r symbols/sec
⚫ Discrete memoryless source
⚫ The amount of information produced by the source during an arbitrary
symbol interval is a disrete random variable X.
⚫ The average information per symbol is then given by:
M
H ( X ) = E{I ( x j )} = − p( x j ) log 2 p( x j ) bit/symbol (2)
j =1

⚫ Entropy = information = uncertainty

⚫ If a signal is completely predictable, it has zero entropy and no
information
⚫ Entropy = average number of bits required to transmit the signal
11/12/2024 8
 Example

⚫ Random variable with uniform distribution over 32

outcomes
– H(X) = -  (1/32).log(1/32) = log 32 = 5
– # bits required = log 32 = 5 bits!
– Therefore H(X) = number of bits required to represent a
random event
⚫ How many bits are needed for:
– Outcome of a coin toss

11/12/2024 9
 Entropy

⚫ The value of H(X) for a given source depends upon the

symbol probabilities p(xi) and M
⚫ However,

0  H ( X )  log 2 M (3)

⚫ The lower bound corresponds to no uncertainty

⚫ The upper bound corresponds to maximum uncertainty,
occuring when each symbol are equally likely

11/12/2024 10
 Example

⚫ For a binary source (M=2), p(1)=

and p(0)=1- = .
⚫ From (2), we have the binary
entropy:
⚫ H(X)= -.log -(1-).log(1-)

11/12/2024 11
 Source coding theorem

⚫ Information from a source producing different symbols

could be described by the entropy H(X)
⚫ Source information rate (bit/s):
Rs = rH(X) (bit/s)
– H(X): source entropy (bits/symbol)
– r: symbol rate (symbols/s)
⚫ Assume this source is the input to a channel:
– C: capacity (bits/symbol)
– S: available symbol rate (symbols/s)
– S.C: bits/s
11/12/2024 12
 Source coding theorem (cont’d)
⚫ Shannon’s first theorem (noiseless coding theorem):
– “Given a channel and a source that generates information at
a rate less than the channel capacity, it is possible to encode
the souce output in such a manner that it can be
transmitted through the channel”
⚫ Demonstration of source encoding by an example:

Discrete
Source Binary
binary
encoder channel
source
Source symbol rate C = 1 bit/symbol
r = 3.5 symbols/s S = 2 symbols/s
SC = 2 bits/s
13

11/12/2024
 Example of Source encoding

⚫ Discrete binary source: A(p=0.9), B(p=0.1)

⚫ Source symbol rate (3.5) >channel capacity (2) 
source symbols cannot be transmitted directly
⚫ Check Shannon’s theorem:
– H(X)=-0.1 log20.1-0.9log20.9=0.469bits/symbol
– Rs = rH(X) = 3.5(0.469)=1.642 bits/s < S.C = 2 bits/s
⚫ Transmission is possible by source encoding to
decrease the average symbol rate

11/12/2024 14
 Example of Source encoding(cont’d)

⚫ Codewords assigned to n-symbol groups of source symbols

⚫ Rules:
– Shortest codewords for the most probable group
– Longest codewords for the least probable group
⚫ n -symbol groups of source symbols # n th-order extension of
original source

11/12/2024 15
 First-Order extension

Source P () Codeword [P()].[Number of Code

symbol Symbols]
A 0.9 0 0.9
B 0.1 1 0.1
L=1.0

⚫ Symbol rate of the encoder = symbol rate of source

⚫ Larger than that of the channel can accommodate

11/12/2024 16
 Second-Order extension
Grouping 2 source symbols at a time
The codeword is based on Shannon-Fano Coding
(explained in some slides later)

Source symbol P () Codeword [P()].[Number of Code

Symbols]
AA 0.81 0 0.81
AB 0.09 10 0.18
BA 0.09 110 0.27
BB 0.01 111 0.03
L=1.29

2n L: the average code length

L =  p( xi ) * li p(xi): probability of the i th symbol of the extended source
i =1 17
li : length of the codeword corresponding to the i th symbol

11/12/2024
 Second-Order extension

L 1.29
= = 0.645 code symbols/so urce symbol
n 2

The symbol rate at the encoder output:

L
r = 3.5(0.645) = 2.258 code symbols/sec >2
n

Still greater than the 2 symbols/second of the channel

So, we continue with the third-order extension

11/12/2024 18
 Third-Order extension
Grouping 3 source symbols at a time:

Source P () Codeword [P()].[Number of Code

symbol Symbols]
AAA 0.729 0 0.729
AAB 0.081 100 0.243
ABA 0.081 101 0.243
BAA 0.081 110 0.243
ABB 0.009 11100 0.045
BAB 0.009 11101 0.045
BBA 0.009 11110 0.045
BBB 0.001 11111 0.005
L=1.598
11/12/2024 19
 Third-Order extension

L 1.598
= = 0.533 code symbols/source symbol
n 3

The symbol rate at the encoder output:

L
r = 3.5(0.533) = 1.864 code symbols/se cond
n

This rate is accepted by the channel

11/12/2024 20
 Efficiency of a source code

⚫ Efficiency is a useful measure of goodness of a source

code
Lmin Lmin
eff = = n

 p( x )l
L
i i
i =1

⚫ Entropy of the n th-order extension of a discrete

memoryless source:
H (X n)=n*H (X)
⚫ The efficiency of the extended source:

n.H ( X )
eff =
L
11/12/2024 21
 Behavior of L/n

▪L/n always exceeds

n
1.0
the source entropy
0.8 and converges to the
source entropy for
0.6 large n
0.469 H(X) ▪Decreasing the
0.4
average codeword
length leads to
0.2
increasing decoding
complexity
0 n
0 1 2 3 4

11/12/2024 22
 Shannon-Fano Coding
⚫ Procedure: 3 steps
1. Listing the source symbols in order of decreasing
probability
2. Partitioning the set into 2 subsets as close to equiprobable
as possible. 0’s are assigned to the upper set and 1’s to the
lower set
3. Continue to partition subsets until further partitioning is
not possible
⚫ Example:

11/12/2024 23
 Example of Shannon-Fano Coding
Ui pi 1 2 3 4 5 Codewords
U1 .34 0 0 00
U2 .23 0 1 01
U3 .19 1 0 10
U4 .1 1 1 0 110
U5 .07 1 1 1 0 1110
U6 .06 1 1 1 1 0 11110
U7 .01 1 1 1 1 1 11111
11/12/2024 24
 Shannon-Fano coding
7
L =  pi li = 2.41
i =1
7
H (U ) = − pi log 2 pi = 2.37
i =1

H (U ) 2.37
eff = = = 0.98
L 2.41

▪The code generated is prefix code due to equiprobable

partitioning
▪Not lead to an unique prefix code. Many prefix codes have
the same efficiency.

11/12/2024 25
 Huffman Coding [1][2][3]
⚫ Procedure: 3 steps
1. Listing the source symbols in order of decreasing
probability. The two source symbols of lowest
probabilities are assigned a 0 and a 1.
2. These two source symbols are combined into a new source
symbol with probability equal to the sum of the two
original probabilities. The new probability is placed in the
list in accordance with its value.
3. Repeat until the final probability of new combined symbol
is 1.0.
⚫ Example:

11/12/2024 26
 Examples of Huffman Coding
0
1.0
Ui pi
0 1 Ui Codewords
U1 .34
.58 U1 00
1 U2 10
U2 .23 0
.42
U3 .19 U3 11
0 1
.24 U4 011
U4 .1
U5 .07 0 1 U5 0100
.14
U6 01010
U6 .06 0 1
.07 U7 01011
U7 .01
1
11/12/2024 27
 Huffman Coding: disadvantages
⚫ When source have many symbols (outputs/messages), the code
becomes bulkyHufman code + fixed-length code.
⚫ Still some redundancy and redundancy is large with a small set
of messagesgrouping multiple independent messages
⚫ Grouping make redundancy small but the number of codewords
grows exponentially, code become more complex and delay is
introduced.

11/12/2024 28