MODULE 1

INFORMATION THEORY
PREPARED BY

SEEMA R KARANTH
ASSISTANT PROFESSOR
DEPT. OF ECE
RNSIT

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT

 Information Theory and Coding (18EC54)

Text Book / Reference Book

1. Digital and analog communication systems, K. Sam Shanmugam, John
Wiley India Pvt. Ltd, 1996.
2. Digital communication, Simon Haykin, John Wiley India Pvt. Ltd, 2008.
3. ITC and Cryptography, Ranjan Bose,TMH, II edition, 2007
4. Principles of digital communication, J. Das, S. K. Mullick, P. K.
Chatterjee,Wiley, 1986 – Technology & Engineering
5. Digital Communications – Fundamentals and Applications, Bernard
Sklar, Second Edition, Pearson Education, 2016, ISBN: 9780134724058.
6. Information Theory and Coding, HariBhat, Ganesh Rao, Cengage,
2017.
7. Error Correction Coding by Todd K Moon,Wiley Std. Edition, 2006

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 2

 Information Theory and Coding (18EC54)

Course Outcomes
CO1: Explain and apply the concepts of Entropy, Rate of Information
and Order of a source with respect to Dependent & Independent
Source.
Syllabus
Module 1: Information Theory: Introduction, Measure of
information, Information content of message, Average Information
content of symbols in Long Independent sequences, Average
Information content of symbols in Long dependent sequences, Markov
Statistical Model for Information Sources, Entropy and Information rate
of Markoff Sources

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 3

 Information Theory and Coding (18EC54)

Contents
• Introduction( Text 1 - Section 4.1)
• Measure of Information(Text 1- Section 4.2)
• Information Content of Message (Text 1 - Section 4.2.1)
• Average Information Content of Symbols in Long Independent
Sequences(Text 1 - Section 4.2.2)
• Average Information Content of Symbols in Long Dependent
Sequences (Text 1 - Section 4.2.3)
• Markoff Statistical Model for Information Sources(Text 1 - Section
4.2.4)
• Entropy and Information Rate of Markoff Sources(Text 1 - Section
4.2.5)
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 4
 Information Theory and Coding (18EC54)

Introduction
• Information – Message or Intelligence – Electrical signals
such as voltage, current, power, speech or picture
• Information Source
• Analog Sources - Emits continuous amplitude electrical
signal with respect to time
• Digital Sources – Emits sequence of letters or symbols
• Analog – Digital Information Source
• Sampling – Analog to Discrete
• Quantization – Discrete to Digital
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 5
 Information Theory and Coding (18EC54)

Introduction

Fig: Block Diagram of Information Systems

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 6

 Information Theory and Coding (18EC54)

Introduction
• Information Source - Discrete set of letters / alphabets /
symbols
• Discrete Information Sources
• Source Alphabet / Source Symbols
• Symbol Rate
• Source Alphabet Probability
• Conditional probability of symbols
• Source Encoder - Converts the symbol sequence into a
binary sequence
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 7
 Information Theory and Coding (18EC54)

Introduction
• Binary coding - High efficiency of transmission
• Fixed Length Coding : Symbols with equal probability
• Variable Length Coding: Conditional Probability
• Transmitter – Amplification, Filtering and Modulations
• Channel – Connection between source and destination
• Decoder – Converts binary output into symbol sequence
• Receiver – To identify the symbol sequence and match it
with correct sequence.

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 8

 Information Theory and Coding (18EC54)

Introduction
• Design and analysis of communication systems
• Evaluate the rate at which information is emitted from
the information source
• Evaluate the maximum rate at which reliable
information is transmitted from noisy communication
channel

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 9

 Information Theory and Coding (18EC54)

Measure of Information
• Discrete Information Source – Message with sequence of
symbols.
• Communication system is designed to reproduce information at
the receiver exactly or appropriately.
• If there are large number of messages, amount of information
content of each message will be different.
• Each message conveys some information.
• Amount of information content depends on probability of
occurrence or uncertainity of underlying event rather than
actual event
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 10
 Information Theory and Coding (18EC54)

Measure of Information
Example 1: Example 2: Weather Condition
Event 1: Water is colorless, in Jaipur
odorless and tasteless Event 1: Sun rises in the east
Event 2: Jog falls is in Shimoga Event 2: Hot and sunny day
district Event 3: Cold day
Event 3: MG Road is blocked due Event 4: Possible snowfall
to heavy rain
Event 4: Heavy snowfall in Jaipur

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 11

 Information Theory and Coding (18EC54)

Measure of Information
• Events that are guaranteed to happen should have no
information content.
• Likely events will have some information content.
• Impossible event contains More Information.
• Higher the probability of occurrence, lesser the information
content.
• Information content is inversely proportional to probability
of occurrence.

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 12

 Information Theory and Coding (18EC54)

Measure of Information
• Assumptions
• I(mk) 0 as Pk 1 (Zero Level Talk)
• I(mk) ≥ 0 for 0 ≤ Pk ≤1, Information Content is non
negative
• I(mk) > I(mj) if Pk < Pj
• If two independent messages are received, then the total
information content is the sum of information conveyed by
each of the two messages
I(mk and mj) = I(mk.mj) = I(mk) + I(mj)
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 13
 Information Theory and Coding (18EC54)

Measure of Information
• Let measure of information is logarithmic function
• I(mk) = log(1/Pk)
• Proves I(mk)= 0 as Pk =1
• Proves Ikj = I(mk and mj) = I(mk.mj) = I(mk) + I(mj)

M1 = Earth quake rocks Gujarat State

M2 = It is raining in Bangalore
M1 and M2 = Earth quake rocks Gujarat
State and it is raining in Bangalore

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 14

 Information Theory and Coding (18EC54)

Measure of Information
• Measure of information : Logarithm function
• I(mk) = log(1/pk) Note = Log2X = Log10X / Log102
• If base is natural log, unit is NATS /NEPERS
• If base is 10, unit is HARTLEY/ DECIT
• If base is 2, unit is BIT (Binary Digit)
• If base is 3 , unit is TRINIT/Ternary units
• If base is 4, unit is Quaternary units / Quadits
• Zero Memory Source – Source emitting a sequence of symbols which
are statistically independent (fixed probability). Source has no
memory.

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 15

 Information Theory and Coding (18EC54)

Measure of Information
Example: Rolling a dice Events : {1, 2, 3, 4, 5, 6}
Probability of occurrence :
{P1, P2, P3, P4, P5, P6} = { 1/6, 1/30, 3/10, 1/4, 1/10, 3/20}
Observation: sum of all probability is 1
To find Self Information (Information content of symbols)
I1 = Log(1/P1) = Log2(6) = 2.585 Bits
I2 = Log(1/P2) = Log2(30) = 4.9086 Bits
I3 = Log(1/P3) = Log2(10/3) = 1.736 Bits
I4 = Log(1/P4) = Log2(4) = 2 Bits
I5 = Log(1/P5) = Log2(10) = 3.3219 Bits
I6 = Log(1/P6) = Log2(20/3) = 2.7369 Bits
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 16
 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

Example: Let source emits three symbols A, B, C with
probability 1/2, 1/4, 1/4 such that each message has 20
symbols
Let msg be ABCCABAABCABABAACAAB
A - 10symbols B – 5 symbols C – 5 symbols
Self Information = 10Log2(2)+ 5Log2(4)+ 5Log2(4) = 30bits
Average Information Content = 30bits/20symbols
= 1.5Bits/symbol

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 17

 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

• Average Information Content of symbols depends on
• Instantaneous flow of information
• Statistical dependence of symbols
• Let source emits one of M possible symbols {s1, s2,s3, ....,sM}
• Let probablility of occurance of M symbols be P1, P2.........Pm
• Let source emits messages with one symbol,
1
• Information content of symbol s1 is I1 = 𝐿𝑜𝑔( )
𝑝1
1
• Information content of symbol s2 is I2 = 𝐿𝑜𝑔( )
𝑝2

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 18

 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

1
• Information content of symbol sm is Im = 𝐿𝑜𝑔( )
𝑝𝑚
• Total Information content is Itotal = I1+I2+.....+Im =
𝑚 1
σ𝑖=1 𝐿𝑜𝑔( )
𝑝𝑖
• The source symbols of source is s1, s2, s3, .........., sm
• Probability of occurrence of symbol be P1, P2, P3,..........., Pm
• Let source emits long messages containing N symbols
• Source symbol s1 occurs on average of p1.N times

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 19

 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

• Source symbol s1 occurs on average of p1.N times
• Source symbol s2 occurs on average of p2.N times
• Source symbol sm occurs on average of pm.N times
• In general source symbol si occurs on average of pi.N times
1
• Information content of symbol s1 is I1 = p1N𝐿𝑜𝑔( )
𝑝1
1
• Information content of symbol s2 is I2 = p2N𝐿𝑜𝑔( )
𝑝2

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 20

 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

1
• Information content of symbol sm is Im = pmN𝐿𝑜𝑔( )
𝑝𝑚
𝑚 1
• Total information content is Itotal = σ𝑖=1 𝑝𝑖𝑁𝐿𝑜𝑔( )
𝑝𝑖
• Average information content or Average information per
symbols is defined as total Information content by the
number of symbols in the message

• Average information content per symbols is also called as

Source Entropy (“average uncertainty” or the “average
amount of surprise”)
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 21
 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

• Average information content per symbols is also called as
Source Entropy H(s) and measured as Bits/Symbols
• H(s) is maximum when the symbols are equi-probable
• Symbol rate – Number of symbols emitted from the source
per second and denoted as “rs” and measured as Symbols
per Second
• Information Rate – Product of average information content
per symbol and symbol rate
Rs = H(s)*rs bits/sec (BPS)
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 22
 Information Theory and Coding (18EC54)

Entropy of symbols in Long Independent sequences

Pb: A discrete emits one of six symbols once for every 1msec. The
symbol probabilities are ½, ¼, 1/8, 1/16 and 1/32 respectively. Find the
information content of each symbol, source entropy, and information
rate.
Solution: I1 = 1Bits H(s) = 1.9375 Bits/Symbol
1 I2 = 2 Bits
Im = 𝐿𝑜𝑔( )
𝑝𝑚
I3 = 3 Bits 1 symbol = 1msec = 0.001s
I4 = 4 Bits 1 sec =1000 Symbols
I5 = 5 Bits rs =1000 Symbols/Sec
Rs = H(s)*rs I6 = 5 Bits Rs = 1.9375*1000
= 1937.5 Bits/Sec

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 23

 Information Theory and Coding (18EC54)

Properties of Entropy
• Property 1: The entropy function is continuous for every
independent variable Pk in the interval (0,1), so does the
entropy function. Entropy function vanishes at both Pk = 0 and
Pk = 1
• Property 2: The entropy function is a symmetrical function of its
arguments i.e., H[Pk, (1-Pk)] = H[(1-Pk), Pk]. H(S) remains same
irrespective of locations of probabilities.
• Property 3: The entropy attains a maximum value when all the
source symbols are equi probable.
H(S)max = Log2q where q = number of symbols
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 24
 Information Theory and Coding (18EC54)

Properties of Entropy
• Property 4: Partitioning of symbols into sub symbols will not
decrease the entropy.
H(S’) ≥ H(S)
• Property 5:
Source Efficiency= ηs = H(S)/H(S)max
Source Redundancy = Rηs = 1- ηs
• Extension of Zero Memory Source
H(Sn) = n*H(S)

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 25

 Information Theory and Coding (18EC54)

Source Entropy of dependent source

• Zero Memory Source – The occurrence of symbols at
particular symbols does not alter the probability of occurrences
of an other symbol.
• Dependent Source – The occurrence of symbols depends on
present and past symbols.
• Models with Statistical Structure / Statistical dependence
• Reduces the average information content of symbols.
• To develop statistical model for source emitting symbols in
dependent sequences
• To define entropy and information rate for the source.
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 26
 Information Theory and Coding (18EC54)

Markoff Statistical Model for Information Sources

• Discrete source emits a symbol once every Ts second
belonging to finite set of source symbol on certain
probability depending on particular symbol and
preceding symbol – Stochastic or Random Process.
• Model - Discrete Stationary Markoff Process
• The source is in one of n possible states 1,2,............,n at
the beginning of each symbol interval. The source
changes state during each symbol interval from i to j.
the probability of transition depends on initial state i
and final state j but does not depends on any of the
preceding symbols.
Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 27
 Information Theory and Coding (18EC54)

Markoff Statistical Model for Information Sources

• Transition probabilities remains constant
as the process progresses in time such
that σ𝑛𝑗=1 𝑝𝑖𝑗 = 1
• As the source changes from i to j emits a
symbol. This symbol depends on the
initial state and transition. The probability
of occurrence of a symbol is given as
Pj(K+1) = σ𝑛𝑖=1 𝑃𝑖(𝑘)𝑝𝑖𝑗

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 28

 Information Theory and Coding (18EC54)

Markoff Statistical Model for Information Sources

• Information sources whose outputs can be modelled by
discrete stationary Markoff processes are called Discrete
Stationary Markoff Sources.
• The probability of occurrence of a symbol depends on the
particular symbol and the symbol immediately preceding it.
• Represented in graph form (tree diagram) in which states are
nodes and transition is directed arc (branch).

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 29

 Information Theory and Coding (18EC54)

Entropy and Information Rate of Markoff Sources

• Average information content of the symbols emitted from the
ith state is
𝑛 1
Hi=σ𝑗=1 𝑝𝑖𝑗𝐿𝑜𝑔( ) Bits/Symbol
𝑝𝑖𝑗
• Entropy of the source is a weighted average of the entropy of
the symbols emitted from each state.
𝑛 𝑛 𝑛 1
H=σ𝑖=1 𝑝𝑖𝐻𝑖 = σ𝑖=1 𝑝𝑖 σ𝑗=1 𝑝𝑖𝑗𝐿𝑜𝑔( ) Bits/Symbol
𝑝𝑖𝑗
• The average information rate R is defined as
R = rs*H Bits/Sec

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 30

 Information Theory and Coding (18EC54)

Entropy and Information Rate of Markoff Sources

• Average Information content per symbol for nth order extension
𝑛 1
GN = 1/N σ𝑖=1 𝑝 𝑚𝑖 log( ) = 1/N H(SN)
𝑝 𝑚𝑖
H(S) = Entropy of Adjoint Source
Where S = First order extension of adjoint source
S2 = Second order extension of adjoint source
S3 = Third order extension of adjoint source
SN= Nth order extension of adjoint source
• GN is monotonically decreasing function of N such that

Seema R Karanth, Asst. Professor, Dept. of ECE, RNSIT 31

Information Theory and Coding (18EC54)

Problem 1.1: Find the relationship between HARTLEY, NATS and BITS.
Solution:
I = Log10(1/P) HARTLEY
I = Log2(1/P) BITS
I = Loge(1/P) = ln(1/P) NATS
Relationship between HARTLEY and BITS Relationship between HARTLEY and NATS
1 Hartley = I / Log10(1/P) 1 Hartley = I / Log10(1/P)
1 1
= Log2 ( ) = Loge ( )
𝑃 𝑃
1 1
Log10 ( ) Log10 ( )
𝑃 𝑃
1 1 1 1
= Log( ) Log( ) = Log( ) Log( )
𝑃 𝑃 𝑃 𝑃
Log 2 Log 10 Log e Log 10

= Log 10 = Log2(10) = Log 10 = Loge(10)

Log 2 Log e
= 3.3219Bits = 2.303 NATS
1 Hartley = 3.3219 Bits 1 Hartley = 2.303 NATS

Relationship between BITS and NATS

1 NATS = I / Loge(1/P)
1
= Log2 ( )
𝑃
1
Loge ( )
𝑃
1 1
= Log( ) Log( )
𝑃 𝑃
Log 2 Log e
= Log e = Log2e
Log 2
= 1.443 Bits
1 Hartley = 1.443 Bits

Problem 1.2: Let source emits three symbols A, B, C with probability 1/2, 1/4, 1/4 such that each
message has 20 symbols. Find Self information and average information content.
Solution:
Let a message be ABCCABAABCABABAACAAB
A - 10symbols B – 5 symbols C – 5 symbols
Self-Information:
1
I(mk) = Log2(𝑃𝑘) Bits
1
IA = Log2(𝑃𝐴) = Log2(2) = 1 Bits
1
IB = Log2(𝑃𝐵) = Log2(4) = 2 Bits
1
IC = Log2(𝑃𝐶) = Log2(4) = 2 Bits

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 1

Information Theory and Coding (18EC54)

Total Information of a message:

ITotal = 10IA + 5IB +5IC
= 10Log2(2) + 5Log2(4) + 5Log2(4)
ITotal = 30 Bits

Average Information Content = ITotal / N

= 30Bits/20 Symbols
H(S) = 1.5 Bits/Symbol

Problem 1.3: A discrete source emits one of six symbols once every milli-seconds. The symbol
probabilities are 1/2, 1/4, 1/8, 1/16, 1/32 and 1/32 respectively. Find the Information Content of each
symbol, source entropy and Information rate.
Solution:
Source symbols = 6
Probability = 1/2, 1/4, 1/8, 1/8, 1/16, 1/32, 1/32
Self-Information:
1
I(mk) = Log2(𝑃𝑘) Bits
1
I1 = Log2(𝑃1) = Log2(2) = 1 Bits
1
I2 = Log2(𝑃2) = Log2(4) = 2 Bits
1
I3 = Log2(𝑃3) = Log2(8) = 3 Bits
1
I4 = Log2(𝑃4) = Log2(16) = 4 Bits
1
I5 = Log2(𝑃5) = Log2(32) = 5 Bits
1
I6 = Log2(𝑃6) = Log2(32) = 5 Bits

Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/ Message Symbol
1
H(S) = ∑6𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/2)Log2(2)+(1/4)Log2(4)+(1/8)Log2(8)+(1/16)Log2(16)+(1/32)Log2(32)+(1/32)Log2(32)
= (1/2)*1 + (1/4)*2 +(1/8)*3 +(1/16)*4+(1/32)*5 +(1/32)*5
H(S) = 1.9375 Bits/ Message-Symbol

Symbol Rate:
1symbol = 1msec 1 sec = 1000 symbols
rs = 1000symbols

Information Rate:
Rs = H(S)*rs Bits/Sec
Rs = H(s)*rs
= 1.9375*1000
Rs = 1937.5 Bits/Sec

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 2

Information Theory and Coding (18EC54)

Problem 1.4: Find the entropy of a source and information content of symbols that emits one out of
four symbols A, B, C and D in a statistically independent sequence with probabilities1/2, 1/4, 1/8,
1/8. Also represent Self-Information in NATS and entropy in NATS/Symbol.
Solution:
Source Symbols: A, B, C, D
Probability: 1/2, 1/4, 1/8, 1/8
1 NATS = 1.4426Bits 1 BITS = 0.693NATS

Self-Information:
1
I(mk) = Log2(𝑃𝑘) Bits
1
IA = Log2(𝑃𝐴) = Log2(2) = 1 Bits = 1*0.693 = 0.693 NATS
1
IB = Log2(𝑃𝐵) = Log2(4) = 2 Bits = 2*0.693 = 1.386 NATS
1
IC = Log2(𝑃𝐶) = Log2(8) = 3 Bits = 3*0.693 = 2.079 NATS
1
ID = Log2(𝑃𝐷) = Log2(8) = 3 Bits = 3*0.693 = 2.079 NATS

Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖 ) Bits/Symbol
1
H(S) = ∑4𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/2)Log2(2) + (1/4)Log2(4) + (1/8)Log2(8) + (1/8)Log2(8)
= (1/2)*1 + (1/4)*2 + (1/8)*3 + (1/8)*3
H(S) = 1.75 Bits/Message-Symbol
= 1.75*0.693
H(S) = 1.2127 NATS/ Message-Symbol

Problem 1.5: An analog signal band limited to 4KHz, is sampled at twice the Nyquist rate and then
quantized into 8 levels Q1, Q2, ............., Q8. Of these, 2 levels occur with a probability of 1/4 each,
2 with probability of 1/8 each and the remaining four with a probability of 1/16 each respectively.
Find the information rate associate with the analog signal.
Solution:
Source Symbols (Quantization Levels) = Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8
Probability = 1/4, 1/4, 1/8, 1/8, 1/16, 1/16, 1/16, 1/16
Source Entropy:
1
H(S) =∑8𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/Message-Symbol
H(S) = (1/4)Log2(4) +(1/4)Log2(4) + (1/8)Log2(8) + (1/8)Log2(8) +(1/16)Log2(16) + (1/16)Log2(16)
+ (1/8)Log2(8) + (1/8)Log2(8)
= (1/4)*2 + (1/4)*2 + (1/8)*3 + (1/8)*3 + (1/16)*4 + (1/16)*4 + (1/16)*4 + (1/16)*4
= (2/4)*2 + (2/8)*3 + (4/16)*4
= 2.75 Bits/Symbol

Symbol rate = Sampling Rate

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 3

Information Theory and Coding (18EC54)

= 2*Nyquist Rate
= 2*2*Band-Width
= 4*4K
rs = 16000 Symbols/Sec

Information Rate:
Rs = H(S) * rs Bits/Sec
Rs = 2.75 *16000
Rs = 44000 Bits/Sec

Problem 1.6: The International Morse code uses a sequence of dots and dashes to transmit letters of
the English alphabet. The dash is represented by a current pulse that has a duration of 3 units and the
dot has a duration of 1 unit. The probability of occurrence of a dash is 1/3 of the probability of
occurrence of dot. Calculate the information content of a dot and a dash, average information content
in the dot dash code. Assume that the dot lasts 1msec, which is the same time interval as the pause
between symbols, find the average information rate of transmission.
Solution:
Source Symbols = dot, dash
Given PDash = (1/3)PDot and PDash + PDot = 1
PDash - (1/3)PDot = 0 ....................(1)
PDash + PDot = 1 .....................(2)
Solving (1) and (2)
We get, PDash = 1/4 = 0.25 and PDot = 3/4 = 0.75

Self Information:
1
I(mk) = Log2(𝑃𝑘) Bits
1
IDot = Log2(𝑃𝑑𝑜𝑡) = Log2(4/3) = 0.4150 Bits
1
IDash = Log2(𝑃𝑑𝑎𝑠ℎ) = Log2(4) = 2 Bits
Note: Since IDash > IDot and PDash < PDot , symbol Dash conveys more information than symbol Dot

Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/Message-Symbol
1
H(S) = ∑2𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (3/4) Log2(4/3) +(1/4) Log2(4)
H(S) = 0.8112 Bits/symbols

In 4 symbols, 3 symbols are Dot and 1 symbol is Dash

TDash = 3Units = 3msec TDot = 1Unit =1msec

Duration of 4 symbols = 10msec

1sec = 4/10m = 400symbols

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 4

Information Theory and Coding (18EC54)

Symbol Rate (rs) = 400 Symbols/Sec

Information Rate:
Rs = H(S) *rs
= 0.8112*400
Rs = 324.51 Bits/Sec

Problem 1.7: The output of an information source consists of 128 symbols, 16 if which occur with a
probability of 1/32 and the remaining 112 occur with a probability of 1/224. The source emits 1000
symbols/sec. Assuming that the symbols are chosen independently, find the average information rate
of this source.
Solution:
Source Symbols = 128
Probability = (1/32) for 16 Symbols and (1/224) for 112 symbols
Symbol Rate: 1000 Symbols/Sec
Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/Symbol

1
H(S) = ∑𝑀128
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
1 1
= ∑16 128
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) + ∑𝑖=17 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= 16*(1/32) Log2(32) + 112* (1/224) Log2(224)
H(S) = 6.4036 Bits/Symbol

Information Rate:
Rs = H(S)*rs = Bits/Sec
Rs = 6.4036*1000
Rs = 6403.6 Bits/Sec

Problem 1.8: A pair of dice are tossed simultaneously. The outcome of the first dice is recorded as
x1 and that of second dice as x2. Three events are defined as follows
X = {(x1, x2) such that (x1+x2) is exactly divisible by 3}
Y = {(x1, x2) such that (x1+x2) is an odd number}
Z = {(x1, x2) such that 5 ≤ (x1+x2) ≤ 7}
Which event conveys more information.
Solution:
Possible Outcomes: {(1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (3,1) (3,2) (3,3)
(3,4) (3,5) (3,6) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (6,1) (6,2) (6,3) (6,4)
(6,5) (6,6)} = 36 Outcomes

X = {(1,2) (1,5) (2,1) (2,4) (3,3) (3,6) (4,2) (4,5) (5,1) (5,4) (6,3) (6 6)} = 12
Y = {(1,2) (1,4) (1,6) (2,1) (2,3) (2,5) (3,2) (3,4) (3,6) (4,1) (4,3) (4,5) (5,2) (5,4) (5,6) (6,1) (6,3)
(6,5)} = 18

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 5

Information Theory and Coding (18EC54)

Z= {(1,4) (1,5) (1,6) (2,3) (2,4) (2,5) (3,2) (3,3) (3,4) (4,1) (4,2) (4,3) (5,1) (5,2) (6,1)} = 15

P(X) = 12/36 = 1/3

P(Y) = 18/36 = 1/2
P(Z) = 15/36 = 5/12
Self-Information:
1
I(mk) = Log2(𝑃𝑘) Bits
1
IX = Log2(𝑃(𝑋)) = Log2(3) = 1.5849 Bits
1
IY = Log2(𝑃(𝑌)) = Log2(2) = 1 Bits
1
IZ = Log2(𝑃(𝑧)) = Log2(12/5) = 1.26303 Bits
Note: Event X is conveying more information since Ix > (IY and IZ) and Px < (PY and PZ)

Problem 1.9: A card is drawn from a deck (i) how much information is received, if the card drawn
is a spade(ii) how much information is received, if the card drawn is an ace (iii) how much information
is received, if the card drawn is an ace of spade (iv)is the information obtained in (iii) is sum of
information obtained in (i) and(ii)
Solution: Total Cards = 52 cards
(i) Card drawn is Spade
Number of Spade Cards = 13
Pspade = Number of Spade Cards/ Total Cards = 13/52 = 1/4
1
Information Content of Spade Card ISpade = Log2(PSpade) = Log2(4) = 2 Bits
(ii) Card drawn is an Ace
Number of Ace Cards = 4
PAce = Number of Ace Cards/ Total Cards = 4/52 = 1/13
1
Information Content of Ace Card IAce = Log2(PAce) = Log2(13) = 3.7004 Bits
(iii) Card drawn is an ace of spade
Number of Ace of Spade = 1
PAceSpade = Number of Ace Space Card/ Total Cards = 1/52
1
Information Content of Ace Card IAceSpade = Log2(PAceSpade) = Log2(52) = 5.7004 Bits
(iv) Information obtained in (iii) is the sum of information obtained in (i) and (ii), since drawing
a spades card and drawing of an ace card are two independent events.
IAceSpade = ISpade + IAce

Problem 1.10: A binary source is emitting an independent sequence of 0’s and 1’s with probabilities
p and 1-p respectively. Plot the entropy of this source versus p (0<p<1).
Solution:
Source Symbols = 0,1
Probability = p,1-p such that (p +1 – p =1)
Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/Message-Symbol

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 6

Information Theory and Coding (18EC54)

H(S) = pLog2(1/p) + (1-p)Log2(1/(1-p))

p = 0 ; H(S) = 0Log2(1/0) + 1Log2(1) = 0+0 = 0 Bits/Symbol
p = 0.1; H(S) = 0.1Log2(10/1) + 0.9Log2(10/9) = 0.468 Bits/Symbol
p = 0.2; H(S) = 0.2Log2(10/2) + 0.8Log2(10/8) = 0.722 Bits/Symbol
p = 0.3; H(S) = 0.3Log2(10/3) + 0.7Log2(10/7) = 0.882 Bits/Symbol
p = 0.4; H(S) = 0.4Log2(10/4) + 0.6Log2(10/6) = 0.971 Bits/Symbol
p = 0.5; H(S) = 0.5Log2(10/5) + 0.5Log2(10/5) = 1 Bits/Symbol
p = 0.6; H(S) = 0.6Log2(10/6) + 0.4Log2(10/4) = 0.971 Bits/Symbol
p = 0.7; H(S) = 0.7Log2(10/7) + 0.3Log2(10/3) = 0.882 Bits/Symbol
p = 0.8; H(S) = 0.8Log2(10/8) + 0.2Log2(10/2) = 0.722 Bits/Symbol
p = 0.9; H(S) = 0.9Log2(10/9) + 0.1Log2(10/1) = 0.468 Bits/Symbol
p = 1; H(S) = 0Log2(1/0) + 1Log2(1) = 0. Bits/Symbol

p H(S)
0 0
0.1 0.4689
0.2 0.722
0.3 0.882
0.4 0.971
0.5 1
0.6 0.971
0.7 0.882
0.8 0.722
0.9 0.4689
Fig 1.1: Plot of H(S) versus P
1.0 0

Inference:
The entropy function varies continuously from 0 to 1 as Pk varies from 0 to 1.

Property 1: The entropy function is continuous for every independent variable pk in the interval (0,1)
i.e., if pk varies continuously from 0 to 1, so does the entropy function. Entropy function vanishes at
both Pk = 0 and Pk =1.

Problem 1.11: Calculate the average information content of Sa, Sb and Sc. Let source Sa, Sb, Sc emits
source alphabets S1, S2, S3 with probabilities Pa = {1/2, 3/10, 2/10}, Pb = {3/10, 2/10,1/2} and Pc =
{1/10,1/2, 3/10}
Solution:
Sa = {S1, S2, S3} Pa = {1/2, 3/10, 2/10}
Sb = {S1, S2, S3} Pb = {3/10, 2/10, 1/2}
Sc = {S1, S2, S3} Pc = {2/10, 1/2, 3/10}
Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/Message- Symbol
1
H(Sa) = ∑3𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 7

Information Theory and Coding (18EC54)

= (1/2)Log2(2) + (3/10)Log2(10/3) + (2/10)Log2(10/2)

H(Sa) = 1.4854 Bits/Symbol

1
H(Sb) = ∑3𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (3/10)Log2(10/3) + (2/10)Log2(10/2) + (1/2)Log2(2)
H(Sb) = 1.4854 Bits/Symbol

1
H(Sc) = ∑3𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (2/10)Log2(10/2) + (1/2)Log2(2) + (3/10)Log2(10/3)
H(Sc) = 1.4854 Bits/Symbol

Inference:
The entropy H(S) remains same even though the probabilities of symbols are in different order.

Property 2: Symmetric Property: The entropy function is a symmetrical function of its arguments
i.e., H[Pk,(1-Pk)] = H[(1-Pk), Pk] for all k = 1, 2, ........., q
The value of H(S) remains the same irrespective of the locations of the probabilities

Problem 1.10: Calculate the average information content of Sa, Sb and Sc. Let source Sa, Sb, Sc emits
source alphabets S1, S2, S3 with probabilities Pa = {3/4, 1/4}, Pb = {1/6, 5/6} and Pc = {1/2, 1/2}.
Solution:
Sa = {S1, S2, S3} Pa = {1/2, 1/4, 1/4}
Sb = {S1, S2, S3} Pb = {1/3, 1/2, 1/6}
Sc = {S1, S2, S3} Pc = {1/3, 1/3, 1/3}
Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/ Message Symbol
1
H(Sa) = ∑3𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/2)Log2(2) + (1/4)Log2(4) + (1/4)Log2(4)
H(Sa) = 1.5 Bits/Message-Symbol

1
H(Sb) = ∑3𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/3)Log2(2) + (1/2)Log2(2) + (1/6)Log2(6)
H(Sb) = 1.459 Bits/ Message Symbol

1
H(Sc) = ∑3𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/3)Log2(3) + (1/3)Log2(3) + (1/3)Log2(3)
H(Sc) = 1.584 Bits/Message-Symbol

Inference: H(S) is maximum when symbols are equi-probable

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 8

Information Theory and Coding (18EC54)

Property 3: Extremal Property: The entropy attains a maximum value when all the source symbols
become equi-probable
H(S)max = log2q where q = 1/P = Number of source symbols

Problem 1.12: A source emits one of 4 symbols S1, S2, S3, S4 with probability 1/2, 1/3, 1/12, 1/12.
Calculate average information content of source. If one of the source symbols is sub-divided,
calculate the entropy of source with partitioned symbols.
Solution:
Source Symbols = {S1, S2, S3, S4}
Probability = {P1, P2, P3, P4} = {1/2, 1/3, 1/12, 1/12}
Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/ Message Symbol
1
H(S) = ∑4𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/2)Log2(2) + (1/3)Log2(3) + (1/12)Log2(12) + (1/12)Log2(12)
= 1.6258 Bits/Message-Symbol

Case 1: If S1 is partitioned into three symbols as S11, S12, S13 with probability {1/3, 1/12, 1/12} such
that ∑3𝑖=1 𝑃𝑠1𝑖 = P1

Source Symbols: {S1, S2, S3, S4} = {S11, S12, S13, S2, S3, S4}
Probability: {P1, P2, P3, P4} = {P11, P12, P13, P2, P3, P4}

Entropy of source with split symbols:

1 1
H(S’) = ∑𝑀 𝑀
𝑖=1 𝑃𝑠1𝑖𝑙𝑜𝑔(𝑃𝑠1𝑖 ) + ∑𝑖=2 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/3log3 + 1/12log12 + 1/12log12) + (1/3log3 + 1/12log12 + 1/12log12}
= 2.2516 Bits/Symbols

Case 2: If S4 is partitioned into two symbols as S41, S42, with probability {1/24, 1/24} such that
∑2𝑖=1 𝑃𝑠4𝑖 = P4

Source Symbols: {S1, S2, S3, S4} = {S1, S2, S3, S41, S42}
Probability: {P1, P2, P3, P4} = {P1, P2, P3, P41, P42}

Entropy of source with split symbols:

1 1
H(S’) = ∑𝑀−1 𝑁
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) + ∑𝑖=1 𝑃𝑠4𝑖𝑙𝑜𝑔(𝑃𝑠4𝑖)
= (1/2log2 + 1/3log3 + 1/12log12) + (1/24log24 + 1/24log24)
= 1.7091 Bits/Symbols

Inference: H(S’) is greater than H(S)

Property 4: Additivity - Partitioning of symbols into sub-symbols cannot decrease the entropy.
H(S’) ≥ H(S)

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 9

Information Theory and Coding (18EC54)

Property 5:
Source Efficiency = ηs = H(S) / H(S)Max
Source Redundancy = Rηs = 1 - ηs

Problem 1.13: A data source has 8 symbols that are produced in blocks of four at a rate of 500
blocks/sec. The first symbol in each block is always the same. The remaining three are filled by any
of the 8 symbols with equal probability. Find the entropy rate of the source.
Solution:
A block has 4 symbols. The first symbol in each block is same and other three symbols are filled with
any of the 8 symbols.
Source Symbols = {S1, S2, S3, S4, S5, S6, S7, S8}
Probability = {1/8, 1/8, 1/8, 1/8, 1/8, 1/8, 1/8, 1/8}
I1 = 0 since the first bit is always the same and carries no information.
H1 = 0 Bits/Block

I2 = Log2(8) = 3 Bits
H2 = (1/8)Log28 + (1/8)Log28 + (1/8)Log28 + (1/8)Log28 + (1/8)Log28 + (1/8)Log28 + (1/8)Log28
= Log2(8) = 3 Bits/Block

Hence,
H2 = I2 = H(S)Max = Log2q = Log28 = 3 Bits/Block
H3 = I3 = H(S)Max = Log2q = Log28 = 3 Bits/Block
H4 = I4 = H(S)Max = Log2q = Log28 = 3 Bits/Block
Fig 1.2: A block of data frame
HTotal = H1 + H2 + H3 + H4
=0+3+3+3
HTotal = 9 Bits/Block

Symbol Rate = Block rate = 500 Blocks /Sec

Information Rate:
Rs = HTotal * rs
= 500*9
Rs = 4500 Bits/Sec

Problem 1.14: A black and white TV picture consists of 525 lines of picture information. Assume
that each line consists of 525 picture elements (pixels) and that each element can have 256 brightness
levels. Picture are repeated at the rate of 30 frames/Sec. Calculate average rate of information
conveyed by a TV set to a viewer and efficiency of TV.
Solution:
Source Symbols = B1, B2, ---------------, B256
Probability = {1/256, 1/256, ---------------------, 1/256}
Total number of pixels in the frame = 525*525 = 275625 Pixels
1 Pixel = 256 Brightness Level
2 Pixel = 256*256 Brightness level
3 Pixel = 256*256*256 Brightness level
525*525 Pixels = (256)^(525*525) Brightness level

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 10

Information Theory and Coding (18EC54)

Each pixel value with brightness level variation gives one picture or one frame
Total number of possible frames = (256)(525*525) Frames or (256)(525*525) Pictures

Assuming that all these frames occur with equal probability, the average information content per
frame is
H(S) = I = H(S)Max = Log2q
= Log2(256)(525*525)
= (525*525)Log2(256)
H(S) = 22.05x105 Bits/Frame

Symbol Rate = 30 Frames/sec

Information Rate = Rs = H(S)*rs = 66.15*106 Bits/Sec.
Source Efficiency = H(S)/H(S)Max = 100%

Extension of Zero Memory Source

Let us consider a binary source S emitting symbols S1, S2 with probabilities P1, P2 respectively such
that P1 + P2 = 1.

Fig. 1.4: Extension of Zero Memory Source

1st order binary source: 21 = 2 Symbols

S1 with probability P1
S2 with probability P2
Source entropy: H(S) = P1Log2(1/P1) +P2Log2(1/P2)

2nd order binary source: 22 = 4 symbols

S1S1 with probability P1P1 = P12
S1S2 with probability P1P2
S2S1 with probability P2P1 = P1P2
S2S2 with probability P2P2 = P22

P12 + 2P1P2 + P22 = (P1 + P2)2 = 1

Entropy of 2nd order binary source:

H(S2) = P12Log2(1/P12) + 2P1P2Log2(1/P1P2) + P22Log2(1/P22)
= 2P12Log2(1/P1) + 2P1P2Log2(1/P1P2) + 2P22Log2(1/P2) since (logan = nloga)
= 2P12Log2(1/P1)+2P1P2Log2(1/P1)+2P1P2Log2(1/P2)+2P22Log2(1/P2)since (logxy=logx +logy)
= 2P1Log2(1/P1)(P1 + P2) + 2P2Log2(1/P2)(P1 + P2) since (P1 + P2) = 1

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 11

Information Theory and Coding (18EC54)

= 2(P1Log2(1/P1) + P2Log2(1/P2))
H(S2) = 2*H(S)

3rd order binary source: 23 = 8 symbols

S1S1S1 with probability P1P1P1 = P13
S1S1S2 with probability P1P1P2 = P12P2
S1S2S1 with probability P1P2P1 = P12P2
S1S2S2 with probability P1P2P2 = P1P22
S2S1S1 with probability P2P1P1 = P12P2
S2S1S2 with probability P2P1P2 = P1P22
S2S2S1 with probability P2P2P1 = P1P22
S2S2S2 with probability P2P2P2 = P23

P13 + 3P12P2 + 3P1P22 + P23 =(P1+P2)3 = 1

Entropy of 3rd order binary source:

H(S3) = P13Log2(1/P13) + 3P12P2 Log2(1/P12P2) + 3P1P22Log2(1/P1P22) + P23Log2(1/P23)
= 3P13Log2(1/P1) + 3P12P2Log2(1/P12) + 3P12P2Log2(1/P2) + 3P1P22Log2(1/P1) +
3P1P22 Log2(1/P22) + 3P23Log2(1/P2)
= 3P13Log2(1/P1) + 6P12P2Log2(1/P1) + 3P1P22Log2(1/P1) + 3P12P2Log2(1/P2) + 3P23Log2(1/P2)
6P1P22 Log2(1/P2)
= 3P1 Log2(1/P1) (P12 +2P1P2 +P22) + 3P2 Log2(1/P2) (P12 +2P1P2 +P22)
= 3P1Log2(1/P1) (P1+P2)2 + 3P2Log2(1/P2) (P1+P2)2
= 3(P1Log2(1/P1) + P2Log(1/P2))
3
H(S ) = 3H(S)

Similarly, the entropy of the Nth order source is given as H(Sn) = n*H(S)

Problem 1.15: A source emits one of the four probable message m1, m2, m3 and m4 with
probabilities of 7/16, 5/16, 1/8 and 1/8 respectively. Find the entropy of the source, list all the
elements for the second extension of this source. Hence show that H(S2) = 2H(S). Find the efficiency
of first and second order source.
Solution:
Messages: {m1, m2, m3, m4}
Probability: {P1, P2, P3, P4} = {7/16, 5/16, 1/8, 1/8}

Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/ Message Symbol
1
H(S) = ∑4𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (7/16) Log2(16/7) + (5/16) Log2(16/5) + (1/8) Log2(8) + (1/8) Log2(8)
H(S) = 1.795 Bits/symbols

Maximum Entropy: H(S)Max = Log2q Bits/Message-Symbol q = number of symbols

H(S)Max = Log4
H(S)Max = 2 Bits/Symbol

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 12

Information Theory and Coding (18EC54)

Source Efficiency: ηs = H(S) / H(S)Max

= 1.795/2 = 89.80%

Second Order Extension 42 =16 symbols

Symbols m1m1 m1m2 m1m3 m1m4 m2m1 m2m2 m2m3 m2m4

Probability P1P1 P1P2 P1P3 P1P4 P2P1 P2P2 P2P3 P2P4
49/256 35/256 7/128 7/128 35/256 25/256 5/128 5/128

Symbols m3m1 m3m2 m3m3 m3m4 m4m1 m4m2 m4m3 m4m4

Probability P3P1 P3P2 P3P3 P3P4 P4P1 P4P2 P4P3 P4P4
7/128 5/128 1/64 1/64 7/128 5/128 1/64 1/64

1
H(S2) = ∑16
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (49/256) Log2(256/49) + (2*35/256) Log2 (256/35) + (4*7/128) Log2(128/7) + (25/256) Log2
(256/25) + (4*5/128) Log2(128/5) + (4*1/64) Log264
= 3.5923 Bits/Symbol

Verification: H(S2) = 2H(S) = 2*1.795 = 3.5923 Bits/symbol

Maximum Entropy: H(S2)Max = Log2q Bits/Message-Symbol where q = number of symbols

H(S2)Max = Log 16
H(S2)Max = 4 bits/symbol

Source Efficiency: ηs = H(S2) / H(S2)Max

= 3.5923/4
= 89.8%

Problem 1.16: A source emits one of the source symbols S1, S2 with probabilities of 1/4 and 3/4
respectively. Find the entropy of the source, list all the elements for the second and third order
extension of this source. Hence show that H(S2) = 2H(S) and H(S3) = 3H(S). Find the efficiency of
first, second and third order source.
Solution:
Source Symbols: {S1, S2,}
Probability: {P1, P2} = {1/4, 3/4}
Source Entropy:
1
H(S) =∑𝑀
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖) Bits/ Message Symbol
1
H(S) = ∑2𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/4) Log2(4) + (3/4) Log2(4/3)
H(S) = 0.8112 Bits/symbols

Maximum Entropy: H(S)Max = Log2q Bits/Message-Symbol q = number of symbols

H(S)Max = Log2
H(S)Max = 1 Bits/Symbol

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 13

Information Theory and Coding (18EC54)

Source Efficiency: ηs = H(S) / H(S)Max

= 0.8112/1 = 81.12%

Second Order Extension 22 =4 Symbols

Symbols S1S1 S1S2 S2S1 S2S2
Probability P1P1 P1P2 P2P1 P2P2
1/16 3/16 3/16 9/16

1
H(S2) = ∑4𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/16) Log2(16) + (2*3/16) Log2 (16/3) + (9/16) Log2(16/9)
= 1.6225 Bits/Symbol

Verification: H(S2) = 2H(S) = 2*0.8112 = 1.6224 Bits/symbol

Maximum Entropy: H(S2)Max = Log2q Bits/Message-Symbol where q = number of symbols

H(S2)Max = Log 4
H(S2)Max = 2 bits/symbol

Source Efficiency: ηs = H(S2) / H(S2)Max

= 1.6225/2
= 81.12%

Third Order Extension 23 = 8 Symbols

Symbols S1S1S1 S1S1S2 S1S2S1 S1S2S2 S2S1S1 S2S1S2 S2S2S1 S2S2S2

Probability P1P1P1 P1P1P2 P1P2P1 P1P2P2 P2P1P1 P2P1P2 P2P2P1 P2P2P2
1/64 3/64 3/64 9/64 3/64 9/64 9/64 27/64

1
H(S3) = ∑8𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (1/64) Log2(64) + (3*3/64) Log2 (64/3) + (3*9/64) Log2(64/9) + (27/64) Log2 (64/27)
= 2.4338 Bits/Symbol

Verification: H(S3) = 3H(S) = 3*0.8112= 2.4336 Bits/symbol

Maximum Entropy: H(S3)Max = Log2q Bits/Message-Symbol where q = number of symbols

H(S3)Max = Log 8
H(S3)Max = 3 Bits/symbol

Source Efficiency: ηs = H(S3) / H(S3)Max

= 2.4338/3
= 81.12%

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 14

Information Theory and Coding (18EC54)

Problem 1.17: The state diagram of a Markov Sources is shown in Fig 1.1. a) find the state
probabilities b) find the entropy of source H(S) c) find G1, G2, G3 and verify that G1 ≥ G2 ≥ G3 ≥ H(S).
d)find the efficiency of first, second and third order source.

Fig. 1.5: Markoff Source for Problem 1.17

Solution:

Fig 1.6: Tree Diagram for Problem 1.17

To find state probability:

We know that P(1) +P(2) = 1 ------------------(1)
From Fig: P(1) = 2/3P(1) + 1/4P(2)
1/3P(1) – 1/4P(2) = 0
P(2) = 1/3P(1) – 3/4P(2)
1/3P(1) – 1/4P(2) = 0 ------------------(2)
Solving (1) and (2)

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 15

Information Theory and Coding (18EC54)

We get P1 = P(1) = 3/7 and P2 = P(2) = 4/7

Entropy of State 1: Entropy of State 2:

1 1
H1 = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗) H2 = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗)
= P11Log(1/P11) +P12log(1/P12) = P21Log2(1/P21) + P22Log2(1/P22)
= (2/3)Log2(3/2) + (1/3)Log2(3) = (1/4)Log2(4) + (3/4)Log2(4/3)
= 0.9183 Bits/Symbol = 0.8113 Bits/Symbol

Source Entropy:
H = ∑𝑀𝑖=1 𝐻𝑖𝑃𝑖
= H1P1 + H2P2
= (0.9183* 3/7) + (0.8113*4/7)
= 0.8572 Bits/Symbol

Entropy of Nth order source:

1 1
GN = ∑𝑀 𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( )=
𝑁 𝑃(𝑚𝑖)

First Order Probabilities:

P(X) = 3/7*2/3 = 2/7
P(Y) = 4/7*3/4 = 3/7
P(Z) = 3/7*1/3 + 4/7*1/4 = 1/7 +1/7 = 2/7

Entropy of first order source:

1 1
G1 = 1 ∑3𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔(𝑃(𝑚𝑖))
= P(X)Log2(1/P(X)) + P(Y)Log2(1/P(Y)) + P(Z)Log2(1/P(Z))
= (2/7)Log2(7/2) + (3/7)Log2(7/3) + (2/7)Log2(7/3)
G1 = 1.557 Bits/Symbols such that G1 > H(S)

Efficiency of first order source:

ηs = H(S) / H(S)Max
= 1.557/Log23
= 98.23%

Second Order Probabilities:

P(XX) = 2/7*2/3 = 4/21 P(ZX) = 1/7*2/3 = 2/21
P(XZ) = 2/7 *1/3 = 2/21 P(YZ) = 3/7*1/4 = 3/28
P(ZZ) = 1/7*1/4 + 1/7*1/3 = 1/28 + 1/21 = 1/12 P(YY) = 3/7*3/4 = 9/28
P(ZY) = 1/7*3/4 = 3/28

Entropy of Second order source:

1 1 1
G2 = ∑7𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( ) = H(S2)
2 𝑃(𝑚𝑖) 2

1
H(S2) = ∑7𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= P(XX)Log2(1/P(XX)) + P(XZ)Log2(1/P(XZ))+ P(ZZ)Log2(1/P(ZZ) + P(ZY)Log2(1/P(ZY) +

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 16

Information Theory and Coding (18EC54)

P(ZX)Log2(1/P(ZX)) + P(YZ)Log2(1/P(YZ)) + P(YY)Log2(1/P(YY))

= (4/21)Log2(21/4)+(2/21)Log2(21/2) + (1/12)Log2(12) + (3/28)Log2(28/3) + (2/21)Log2(21/2)
+ (3/28)Log2(28/3) + (9/28)Log2(28/9)
H(S2) = 2.6174 Bits/Message Symbol

1
G2 = 2 H(S2)
1
= 2 * 2.6174
G2 = 1.3087 Bits/Symbol such that G1 > G2 > H(S)

Efficiency of second order source:

ηs = H(S2) / H(S2)Max
= 2.6174/Log27
= 93.23%

3rd order probabilities

P(XXX) =4/21*2/3 = 8/63 P(ZXX) = 2/21*2/3 = 4/63
P(XXZ) = 4/21*1/3 = 4/63 P(ZXZ) = 2/21*1/3 = 2/63
P(XZZ) = 2/21*1/4 = 1/42 P(ZZY) = 1/21*3/4 = 1/28
P(XZY) = 2/21*3/4 =1/14 P(YZX) = 3/28*2/3 = 1/14
P(ZZX) = 1/28*2/3 = 1/42 P(YZZ) = 3/28*1/3 = 1/28
P(ZZZ) = 1/28*1/3 + 1/21*1/4 = 1/42 P(YYZ) = 9/28*1/4 = 9/112
P(ZYZ) = 3/28*1/4 = 3/112 P(YYY) = 9/28*3/4 = 27/112
P(ZYY) = 3/28*3/4 = 9/112

Entropy of 3rd order source:

1 1 1
G3 = 3 ∑15 3
𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔(𝑃(𝑚𝑖)) = 2 H(S )

1
H(S3) = ∑15
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= P(XXX)Log2(1/P(XXX)) + P(XXZ)Log2(1/P(XXZ)) + P(XZZ)Log2(1/P(XZZ)) +
P(XZY)Log2(1/P(XZY)) + P(ZZX)Log2(1/P(ZZX)) + P(ZZZ)Log2(1/P(ZZZ)) +
P(ZYZ)Log2(1/P(ZYZ)) + P(ZYY)Log2(1/P(ZYY)) + P(ZXX)Log2(1/P(ZXX)) +
P(ZXZ)Log2(1/P(ZXZ)) + P(ZZY)Log2(1/P(ZZY)) + P(YZX)Log2(1/P(YZX)) +
P(YZZ)Log2(1/P(YZZ)) + P(YYZ)Log2(1/P(YYZ) + P(YYY)Log2(1/P(YYY))
= (8/63) Log2(63/8) + (2*4/63) Log2(63/4) + (3*1/42) Log2(42) + (2*1/14) Log2(14) +
(3/112) Log2(112/3) + (2*9/112) Log2(112/9) +(2/63) Log2(63/2) + (2*1/28) Log2(28) +
(27/112) Log2(112/27)
= 3.533 Bits/Message Symbol

1
G3 = 3 H(S3)
1
= 3 *3.533
G3 = 1.178 Bits/Symbol such that G1 > G2 > G3 > H(S)

Efficiency of second order source:

ηs = H(S3) / H(S3)Max

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 17

Information Theory and Coding (18EC54)

= 3.533/Log215
= 90.429%

Problem 1.18: The state diagram of a Markoff source is shown in Fig 1.3, determine i) Entropy of
each state ii) entropy of the source iii) find the probabilities of two symbols sequences iv) find G1,
G2 and H. v) efficiency of first, second and third order source.

Fig. 1.7: State Diagram for Problem 1.18

Solution:

Fig. 1.8: Tree Diagram for Problem 1.18

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 18

Information Theory and Coding (18EC54)

To find state probability:

We know that P(1) + P(2) + P(3) = 1 -------------------(1)
From Fig: P(1) = 1/2P(1) + 1/4P(2) + 1/4P(3)
1/2P(1) – 1/4P(2) – 1/4P(3) = 0 -------------------(2)
P(2) = 1/4P(1) + 1/2P(2) + 1/4P(3)
1/4P(1) – 1/2P(2) + 1/4P(3) = 0 -------------------(3)
P(3) = 1/4P(1) + 1/4P(2) + 1/2P(3)
1/4P(1) + 1/4P(2) – 1/2P(3) = 0 -------------------(4)
Solving (1), (2) and (3)
We get P1 = P(1) = 1/3, P2 = P(2) = 1/3 and P3 = P(3) = 1/3

Entropy of State 1:
1
H1 = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔( )
𝑃𝑖𝑗
= P11Log(1/P11) + P12log(1/P12) + P13log(1/P13)
= (1/2)Log2(2) + (1/4)Log2(4) + (1/4)Log2(4)
= 1.5 Bits/Message Symbol

Entropy of State 2:
1
H2 = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗)
= P21Log2(1/P21) + P22Log2(1/P22) + P23log(1/P23)
= (1/4)Log2(4) + (1/2)Log2(2) + (1/4)Log2(4)
= 1.5 Bits/Message Symbol

Entropy of State 3:
1
H2 = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗)
= P31Log2(1/P31) + P32Log2(1/P32) + P33log(1/P33)
= (1/4)Log2(4) + (1/4)Log2(4) + (1/2)Log2(2)
= 1.5 Bits/Message Symbol

Source Entropy:
H = ∑𝑀𝑖=1 𝐻𝑖𝑃𝑖
= H1P1 + H2P2 + H3P3
= (1.5*1/3) + (1.5*1/3) + (1.5*1/3)
= 1.5 Bits/Message Symbol

Entropy of Nth order source:

1 1 1
GN = ∑𝑀 𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( )= H(SN)
𝑁 𝑃(𝑚𝑖) 𝑁
First Order Probabilities:
P(X) = P(X)State1 + P(X)State2 + P(X)State3 = 1/6 +1/12 + 1/12 = 1/3
P(Y) = P(Y)State1 + P(Y)State2 + P(Y)State3 = 1/12 + 1/6 + 1/12 = 1/3
P(Z) = P(Z)State1 + P(Z)State2 + P(Z)State3 = 1/12 + 1/12 + 1/6 = 1/3

Entropy of first order source:

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 19

Information Theory and Coding (18EC54)
1 1
G1 = ∑3𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( )
1 𝑃(𝑚𝑖)
= P(X)Log2(1/P(X)) + P(Y)Log2(1/P(Y)) + P(Z)Log2(1/P(Z))
= (1/3)Log2(3) + (1/3)Log2(3) + (1/3)Log2(3)
G1 = 1.5 Bits/Symbol

Efficiency of first order source:

ηs = H(S) / H(S)Max
= 1.5/Log23
= 100%

Second Order Probabilities:

P(XX) = 1/6*1/2 + 1/12*1/2 + 1/12* 1/2 = 1/6 or P(XX) = = 1/3*1/2 = 1/6
P(XZ) = 1/6*1/4 + 1/12*1/4 + 1/12*1/4 = 1/12 P(XZ) = P(X)*PXZ = 1/3*1/4 = 1/12
P(XY) = 1/6*1/4 + 1/12*1/4 + 1/12*1/4 = 1/12 P(XY) = P(X)*PXY = 1/3*1/4 = 1/12
P(ZX) = 1/12*1/4 + 1/12*1/4 + 1/6*1/4 = 1/12 P(ZX) = P(Z)*PZX = 1/3*1/4 = 1/12
P(ZZ) = 1/12*1/2 + 1/12*1/2 + 1/6*1/2 = 1/6 P(ZZ) = P(Z)*PZZ = 1/3*1/2 = 1/6
P(ZY) = 1/12*1/4 + 1/12*1/4 + 1/6*1/4 = 1/12 P(ZY) = P(Z)*PZY = 1/3*1/4 = 1/12
P(YX) = 1/12*1/4 + 1/6*1/2 + 1/12*1/4 = 1/12 P(YX) = P(Y)*PYX = 1/3*1/4 = 1/12
P(YZ) = 1/12*1/4 + 1/6*1/2 + 1/12*1/4 = 1/12 P(YZ) = P(Y)*PYZ = 1/3*1/4 = 1/12
P(YY) = 1/12*1/2 + 1/12*1/2 + 1/6*1/2 = 1/6 P(YY) = P(Y)*PYY = 1/3*1/2 = 1/6

Entropy of Second order source:

1 1 1
G2 = 2 ∑9𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔(𝑃(𝑚𝑖)) = 2 H(S2)

1
H(S2) = ∑9𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (3*1/6) Log2(6) + (6*1/12) Log2(12)
H(S2) = 3.0849 Bits/Message Symbol

1
G2 = 2 H(S2)
1
= 2 * 2.6174
G2 = 1.5424 Bits/Symbol

Efficiency of second order source:

ηs = H(S2) / H(S2)Max
= 3.0849/Log29
= 97.31%

3rd order probabilities

P(XXX) = P(XX)*PXX = 1/6*1/2 = 1/12 P(ZZZ) = P(ZZ)*PZZ = 1/6*1/2 = 1/12
P(XXZ) = P(XX)*PXZ = 1/6*1/4 = 1/24 P(ZZY) = P(ZZ)*PZX = 1/6*1/4 = 1/24
P(XXY) = P(XX)*PXY = 1/6*1/4 = 1/24 P(ZYA) = P(ZY)*PYX = 1/12*1/4 = 1/48
P(XZX) = P(XZ)*PZX = 1/12*1/4 = 1/48 P(ZYZ) = P(ZY)*PYZ = 1/12*1/4 = 1/48
P(XZZ) = P(XZ)*PZZ = 1/12*1/2 = 1/24 P(ZYY) = P(ZY)*PYY = 1/12*1/2 = 1/24

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 20

Information Theory and Coding (18EC54)

P(XZY) = P(XZ)*PZY = 1/12*1/4 = 1/48 P(YXX) = P(YX)*PXX = 1/12*1/2 = 1/24

P(XYX) = P(XY)*PYX = 1/12*1/4 = 1/48 P(YXZ) = P(YX)*PXZ = 1/12*1/4 = 1/48
P(XYZ) = P(XY)*PYZ = 1/12*1/4 = 1/48 P(YXY) = P(YX)*PXY = 1/12*1/4 = 1/48
P(XYY) = P(XY)*PYY = 1/12*1/2 = 1/24 P(YZX) = P(YZ)*PZX = 1/12*1/4 = 1/48
P(ZXX) = P(ZX)*PXX = 1/12*1/2 = 1/24 P(YZZ) = P(YZ)*PZZ = 1/12*1/2 = 1/24
P(ZXZ) = P(ZX)*PXZ = 1/12*1/2 = 1/24 P(YZY) = P(YZ)*PZY = 1/12*1/4 = 1/48
P(ZXY) = P(ZX)*PXY = 1/12*1/4 = 1/48 P(YYX) = P(YY)*PYX = 1/6*1/4 = 1/24
P(ZZX) = P(ZZ)*PZX = 1/6*1/4 = 1/24 P(YYZ) = P(YY)*PYZ = 1/12*1/4 = 1/48
P(YYY) = P(YY)*PYY = 1/12*1/2 =1/24

Entropy of 3rd order source:

1 1 1
G3 = 3 ∑27 3
𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔(𝑃(𝑚𝑖)) = 2 H(S )

1
H(S2) = ∑27
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (3*1/12) Log2(12) + (12 * 1/24) Log2(24) + (12*1/48) Log2(48)
= 4.585 Bits/Message Symbol

1
G3 = 3 H(S3)
1
= 3 *4.585
G3 = 1.528 Bits/Symbol
Hence G1 > G2 > G3 > H(S)

Efficiency of Third order source:

ηs = H(S3) / H(S3)Max
= 4.585/Log227
= 96.42%

Problem 1.19: For the first order Markoff model shown in Fig. 1.5, find the state probabilities,
entropy of each state and entropy of the source. Find G1, G2, G3 and hence verify G1 > G2 > G3 > H.
Also find efficiency of first, second and third order source

Fig. 1.9: State Diagram for Problem 1.19

Solution:
To find state probability:
We know that P(A) + P(B) + P(C) = 1 -------------------(1)
From Fig: P(A) = 0.2P(A) + 0.2P(C)
0.8P(A) – 0.2P(C) = 0 -------------------(2)
P(B) = 0.2P(B) + 0.3P(C)
0.8P(B) – 0.3P(C) = 0 -------------------(3)
P(C) = 0.5P(C) + 0.8P(B) + 0.8P(A)
0.8P(A) + 0.8P(B) – 0.5P(C) = 0 -------------------(4)

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 21

Information Theory and Coding (18EC54)

Solving (1), (2) and (3)

We get P(A) = 2/13, P(B) = 3/13 and P(C) = 8/13

Fig. 1.10: Tree Diagram for Problem 1.19

Entropy of State A:
1
HA = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗)
= PAALog(1/PAA) + PAClog(1/PAC)
= 0.2Log2(1/0.2) + 0.8Log2(1/0.8)
= 0.7219 Bits/Message Symbol

Entropy of State B:
1
HB = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗)
= PBBLog2(1/PBB) + PBCLog2(1/PBC)
= 0.2Log2(1/0.2) + 0.8Log2(1/0.8)
= 0.7219 Bits/Message Symbol

Entropy of State C:
1
HC = ∑𝑀
𝑗=1 𝑃𝑖𝑗𝑙𝑜𝑔(𝑃𝑖𝑗)
= PCALog2(1/PCA) + PCBLog2(1/PCB) + PCCLog2(1/PCC)

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 22

Information Theory and Coding (18EC54)

= 0.2Log2(1/0.2) + 0.3Log2(1/0.3) + 0.5Log2(1/0.5)

= 1.485 Bits/Message Symbol

Source Entropy:
H = ∑𝑀𝑖=1 𝐻𝑖𝑃𝑖
= HAPA + HBPB + HCPC
= (0.7219*2/13) + (0.7219*3/13) + (1.485*8/13)
= 1.1915 Bits/Message Symbol

Entropy of Nth order source:

1 1
GN = ∑𝑀 𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( )
𝑁 𝑃(𝑚𝑖)
First Order Probabilities:
P(A) = 2/13
P(B) = 3/13
P(C) = 8/13

Entropy of first order source:

1 1
G1 = ∑3𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( )
1 𝑃(𝑚𝑖)
= P(A)Log2(1/P(A)) + P(B)Log2(1/P(B)) + P(C)Log2(1/P(C))
= (2/13)Log2(13/2) + (3/13)Log2(13/3) + (8/13)Log2(13/8)
G1 = 1.3346 Bits/Symbol

Efficiency of First order source:

ηs = H(S) / H(S)Max
= 1.3346/Log23
= 84.20%

Second Order Probabilities:

P(AA) = 2/13*0.2 = 2/65 P(CA) = 8/13*0.2 = 8/65
P(AC) = 2/13*0.8 = 8/65 P(CB) = 8/13*0.3 = 12/65
P(BB) = 3/13*0.2 = 3/65 P(CC) = 8/13*0.5 = 4/13
P(BC) = 3/13*0.8 = 12/65

Entropy of Second order source:

1 1 1
G2 = ∑7𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔( ) = H(S2)
2 𝑃(𝑚𝑖) 2
1
2
H(S ) = ∑7𝑖=1 𝑃𝑖𝑙𝑜𝑔( )
𝑃𝑖
= (2/65) Log2(65/2) + (3/65) Log2(65/3) + (2*8/65) Log2(65/8) + (2*12/65) Log2(65/12) +
(4/13) Log2(13/4)
2
H(S ) = 2.526 Bits/Message Symbol

1
G2 = 2 H(S2)
1
= 2 * 2.526
G2 = 1.2632 Bits/Symbol

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 23

Information Theory and Coding (18EC54)

Efficiency of Second order source:

ηs = H(S2) / H(S2)Max
= 2.526/Log27
= 89.97%

3rd order probabilities

P(AAA) = 2/65*0.2 = 2/325 P(BCC) = 12/65*0.5 = 6/65
P(AAC) = 2/65*0.8 = 8/325 P(CAA) = 8/65*0.2 = 8/325
P(ACA) = 8/65*0.2 = 8/325 P(CAC) = 8/65*0.8 = 32/325
P(ACB) = 8/65*0.3 = 12/325 P(CBB) = 12/65*0.2 = 12/325
P(ACC) = 8/65*0.5 = 4/65 P(CBC) = 12/65*0.8 = 48/325
P(BBB) = 3/65*0.2 = 3/325 P(CCA) = 4/13*0.2 = 4/65
P(BBC) = 3/65*0.8 = 12/325 P(CCB) = 4/13*0.3 = 6/65
P(BCA) = 12/65*0.2 = 12/325 P(CCC) = 4/13*0.5 = 2/13
P(BCB) = 12/65*0.3 = 18/325

Entropy of 3rd order source:

1 1 1
G3 = 3 ∑17 3
𝑖=1 𝑃(𝑚𝑖)𝑙𝑜𝑔(𝑃(𝑚𝑖)) = 2 H(S )

1
H(S3) = ∑17
𝑖=1 𝑃𝑖𝑙𝑜𝑔(𝑃𝑖)
= (2/325) Log2(325/2) +(3/325) Log2(325/3)+ (2*4/65) Log2(65/4) + (3*8/325)Log2(325/8) +
(4*12/325)Log2(325/12)+(18/325)Log2(325/18)+(32/325)Log2(325/32)+(2/13) Log2(13/2)+
(48/325) Log2(325/48) + (2*6/65)Log2(65/6)
= 3.718 Bits/Message Symbol

1
G3 = 3 H(S3)
1
= 3 *3.718
G3 = 1.2.39 Bits/Symbol

Efficiency of Third order source:

ηs = H(S3) / H(S3)Max
= 3.718/Log217
= 90.96%
Hence G1 > G2 > G3 > H

Problem 1.20: Design a system to report the heading of a collection of 400 cars. The heading levels
are heading straight (S), turning left(L) and turning right(R). The information to be transmitted every
second. Construct a model based on the test data given below.
a) On the average during a given reporting interval, 200 cars were heading straight, 100 were turning
left and remaining were turning right.
b) Out of 200 cars that reported heading straight, 100 of them reported going straight during the next
reporting period. 50 of them turning left and remaining turning right during the next period.
c) Out of 100 cars that reported as turning during a signaling period, 50 of them continued their turn
and remaining headed straight during the next reporting period.

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 24

Information Theory and Coding (18EC54)

d) The dynamics of the cars did not allow them to change their heading from left to right or right to
left during subsequent reporting periods.
e) Find the entropy of each state, find the entropy of the system, find the rate of transmission.
Solution:

Fig. 1.11: System Model for Problem 1.19 Fig. 1.12: Tree Diagram for Problem 1.19

PS = P(S) = 200/400 = 1/2

PL = P(L) = 100/400 = 1/4
PR = P(R) = 100/400 = 1/4
P(S|S) = 100/200 = 1/2
P(L|S) = 50/200 = 1/4
P(R|S) = 50/200 = 1/4
P(S|L) = 50/100 = 1/2
P(L|L) = 50/100 = 1/2
P(S|R) = 50/100 = 1/2
P(R|R) = 50/100 = 1/2
P(R|L) = Not Allowed
P(L|R) = Not Allowed

Entropy of Source S:
Hs = (1/2)Log2(2) + (1/4)Log2(4) + (1/4)Log2(4)
= 1.5 Bits/Symbol

Entropy of Source L:
HL = (1/2)Log2(2) + (1/2)Log2(2)
= 1 Bits/Symbol

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 25

Information Theory and Coding (18EC54)

Entropy of Source R:
HR = (1/2)Log2(2) + (1/2)Log2(2)
= 1 Bits/Symbols

Source Entropy:
H = HSPS + HLPL + HRPR
= 1.5*(1/2) + 1*(1/4) + 1*(1/4)
= 1.25 Bits/Symbol
Information Rate:
Rs = rs*H(S) = 1.25*1 = 1.25 Bits/Second

Exercise Problems
1. Find the information content of a message that consists of a digital word 10 digits long in which
each digit may take on one of six possible levels. The probability of sending any of the six levels
is assumed to be equal, and the level in any digit does not depend on the values taken by the
previous digit.
2. Conventional telegraphy uses two symbols, the dash and dot. Assuming that the dash is twice as
long as the dot and half as probable, assume that the dot lasts 10msec, which is the same time
interval as the pause between symbols, find the average symbol rate and the entropy rate.
3. In a facsimile transmission of picture, there are about 3x106 pixels per frame. For a good
reproduction, 16 brightness levels are necessary. Assume all these levels are equally likely to
occur. Find the rate of information transmission if one picture is to be transmitted every 5 minutes.
What is the source efficiency of this facsimile transmission?
4. A zero memory source has a source alphabet S = {S1, S2, S3} with P = {1/2, 1/4, 1/4}. Find the
entropy of this source. Also determine the entropy of its 2nd extension and verify that H(S2) =
2H(S)
5. A source emits one of the source symbols S1, S2 with probabilities of 1/3 and 2/3 respectively.
Find the entropy of the source and show that H(S2) = 2H(S), H(S3) = 3H(S) by listing the symbols
of second and third order extended source along with their probabilities. Find the efficiency of
first, second and third order source.
6. An information system gives an information for about 500 students passing out with Bachelor
degree for every year from Bangalore university. The students can get into one of three fields as
given below
a) Go abroad for higher studies
b) Higher studies in India
c) Jobs in India
Based on the data given below, construct a model for the source and calculate the source entropy.
i. On the average, 250 students are going abroad
ii. Out of 250 going abroad this year,100 were reported going abroad next year, 100 went to
higher studies in India and remaining 50 were joined jobs in India.
iii. Out of 250 remaining in India this year, 125 continued to do so, while 125 went abroad
next year.
7. The state diagram of the first order Mark off Source is shown in Fig. 1.12 (i) find the stationary
distribution (ii) find the entropy of each state and hence the entropy of the source (iii) Find the

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 26

Information Theory and Coding (18EC54)

entropy of the adjoint source and verify that H(S) < H(S) (iv) Also find G1, G2 and G3 and hence
show that G1 > G2 > G3 > H.

Fig 1.12: Markoff Source for Problem 7 Fig 1.13: Markoff Source for Problem 8

8. For the first order Markoff source with a source alphabet S = {A, B, C} shown in Fig. 1.13
(i)Compute the probabilities of states (ii)find H(S) and H(S2) (iii) Find the first and second order
source efficiency.

SEEMA R KARANTH, ASST. PROF, DEPT. OF ECE, RNSIT 27

 MISCELLANEOUS EXAMPLES

Example I.37: An analog signal band limited to 6 KHZ is sampled at thrice the Nyquist rate
and then quantized into 1l levels Q. Q,Q . O fthese, three levels occurwith probability
1
of each, four levels with probability of,, each and the remaining four levels with probability
12
1
each. Find the rate of information associated with the analog signal.
24
Solution : Given sampling rate r
=
3 x
Nyquist rate
=
3 x 2B
=
3 x2x 6K =36 K-samples/sec.
The entropy of the analog signal is given by
11

H(S) 2p,
i 1
logbits/m-sym
Pi
or sample

=
3xlog 6 + 4x 12 log 12 +4x log
24
24
6
= 3.252 bits/sample
Rate of information = (G) [H(S)]

(36 K-samples/sec) (3.252 bits/sanple)

R, =

R,= 117.072 KBPS

Example 1.38 : A fair coin is tossed repeatedly.
Let X =
{event of getting 5 heads out of 8 trials
Y {event of getting 7 heads out of 11 trials}
=
wcn information? Support your answer by numerical computation of
event conveys more

respective amounts of information.

 Information Theory and o.
86
Cot
Z number of heads occurring
Solution: Let,
binomial R.V. with parameters
n = number of trials
head = 0.5
p probabilityof getting
91-p=1 - 0.5 =0.5
have,
From equation (0.47), we
P(Z z) = Cpq"-z

5) Cp"(1 p)"-
-
=

P(X) =
P(X =

= ,C(0.5)(1 -0.5)-5
= 0.21875

PY)= P(Y
=
7) =
p'(1 p)"- -

-7
F
C0.5)'(1 -

0.5)
= 0.16113

1
Self-infornmation of X = I(X) = log P ( X ) l O g 0.,21875

I(X) =
2.193 bits
1
= log 016113
Self-information of Y =I(Y) =
log P(Y) OE
0.16113
I(Y) =
2.634 bits
Y carries more information.
The outcome of first dice is recora
Example 1.39: A pair of dice are tossed simultaneously.
defined as below:
as x, and that of second dice as x,. Three events are
A {(x, x,) such that (x, +x,) is divisable exactly by 4}
B {(x, x,) such that 6 S (x, t+ x) S 8}
C = {(x,, x,) such that x,x, is divisable exactly by 3}
Which events conveys maximum information? Support your answer by numerical computau
Solution : From example 1.20, the sample space S has thirty six combinations given by

0.1) (0,2) (1,3) (1, 4) (1,5)(1,6)||

(2,1) (2, 2) (2,3) (2.4). (2,5) (2,6)
S
(3,1) (3,2) (3,3) (3, 4) (3,5) (3,6)|
(4,1) (4,2) (4,3) (4,4) (4, 5) (4,6)
(5, 1) (5,2) (5,3) (5, 4) (5, 5) (5, 6)
(6,1) (6,2) (6, 3) (6,4) (6, 5) (6,6)
Information Theory

87
The event-A Contains the pairs given by
A =
{(1,3), (2, 2), (2, 6), (3, 1),
9 (3, 5), (4, 4), (5, 3), (6, 2), (6, 6)}
P(A) = 0 . 2 5
36
The event-B contains
B
{(1,5), (1,6), (2, 4), (2, 5),(2, 6), (3, 3), (3,
4,2), (4, 3), (4, 4), (5, 1), (5, 2), 4), (3, 5),
16 (5, 3), (6, 1), (6, 2)}
P(B) -0.444
36
The event C contains
C {(1,3), (1, 6), (2, 3), (2, 6), (3, 1), (3,
2), (3, 3), (3, 4), (3, 5),
(3,6). (4, 3), (4,6), (5, 3), (5, 6), (6,
(6, 5). (6, 6)} 1), (6, 2), (6, 3), (6, 4),
20
PC) =0.556
36
1
Self-information of A =I=log *2 bits
0.25
Self-information of B =I= 1.17 bits
0.444
1
Self-information of C =I=
logsE0.848
0 bits
The
event-A carries the maximum information.
Example 1.40: A voice signal in a PCM system is sampled at 2.5 times the Nyquist rate and
is quantized into 16 levels with the
following probabilities:
PPP, P4 0.08
=

PsPP P=0.065
PPioP Pi2
=
0.055
PiP4Pis Pi6 0.05
Calculate the entropy and information rate of the PCM signal if the bandwidth of
3.5 KHz signal is
16
Solution: Entropy H(S) =2P;log Pi i =

H(S) =
4 x
0.08 log OR +4x 0.065 log
0,065
+4 x 0.055 log 0.055 +4 x 0.05 log
0.05
H(S) = 3.9763 bits/level or sample.
 Information Theory and Cot
88 ding

Given sampling rate =

(2.5) (Nyquist rate)
(2.5) (2B)
= (2.5) (2) (3.5 K)
= 17.5 K-samples/sec.
Information rate R, = r, H(S)
= (17.5 K-samples/sec) (3.9763 bits/sample)
R, 69.585 KBPS
Example 141 : A discrete source emits the messages m, m,, m, and m, with probabilities

7 4 3
15' 15 15 respectively.
and Calculate the information content of each message andthe
average information contents per message. Show that the entropy of the source increases
when the symbol m, is partitioned into three sub-symbols m,j, m,42 and mg3 with respective
1
probabilities 24
4' 60 and
60 *120
Solution
Information content ofeach message:

Self-information in m, = m log (7/15) = 1.0995 bits

Self-information in m, = m log (a11 = 1.9069 bits

(4/15)
Self-information in m, = m = log 1 2.322 bits
(3/15)
1
Self-information in m, =

Im log 1 3.9069 bits

(1/15)
Average information H(S) = Pm, log
i = 1
Pm
HS) log log log log
H(S)= 1.7465 bits/m-symbol
is partitioned into three sub-symbols, then the
When m source S' will have
mbols

six sym
given by m,, m, " a " a 2 and m a With r e s p e c t i v e p r o b a b i l i t i e s o f S

15' 15' 15' 24'

and . The new entropy ol
sourceS' is given by
120
Information Theory
89

H(S) P log
Pi
15 15
log
15 log lo
1
log
24 24 +log
60
60+log
120
120
H(S') =
1.8331 bits/m-symbol
H(S')> H(S) proved
Example 1.42: Consider a source with alphabets m, and m, with respective probabilities of

and. Determine the entropy of source S and the entropy of its third extension. Hence
6
show that H(S) = 3H(S).

Solution: Entropy of source S is given by

2

H(S) 2Pm,log
i=1 Pmi

H(S) = 0.65 bits/message-symbol

3rd Extension:

Symbols(g;)|m,m,m, | m,m,m2 mm,m m,m,m,| mm,m2 m,m,m2 mm2m | m2m2m2

P(,) 125 25 25 25 5 5

216 L 216 21 216 216 216 216 216

Entropy of 3d extended source is given by

H(S) =

2P(G,) o8 p(a,)
i1

12 log 216 +3x 25

216 2 216 1o8 25

+3 log log
216
216
H(S) = 1.95 bits/m-symbol
=
3 x 0.65 bits/message-symbol
H(S) = 3 H(S) proved
 Information Theory and Cot.
oding
90
Markov
source is shown in
i.
fig. entropy
1.15(4). Find entron.

diagram ofa that G, > , ?

1.43 The state hence show
ENample and
OT tne source (ii)
Find G, and G,
1/4
Z

1/3
1.43
for Example
Fig. 1.15: Markov Source
Markov source are given by
Solution: The state equations of the given
1.77)
forstate 1: PC1) PC) +P(2)
2 .(1.78)
for state 2
P(2) PO)+ 3 P(2)
P(1) + P(2) = 1 |
(1.79)
And

From equation (1.77), P(2)

... (1.80)
P2)PCU
4

Using equation (1.80) in (1.79)

3
P(1)+P(1) = 1
4
P(1) = 1

P(1) ;7
3
P(2)-PO)-*
(i) From equation (1.40), the entropy ofthe 1 state is given by
P(2)-
2

H, 2Pj log
Pij
Pi log
Pi +P log P12
4
loB 4log 4
H, =
0,8113 bits/m-sym
Information Theory 91

And H, P log
j=1 P2
1
p1 +P22 108 P22
P2log P21
3
=log3 +log. H,=0.9183 bits/message-symbol
From equation (1.41), the entropy H of the source is given by
H 2P)H
i =1
=
P(1) H, +
P(2) H,

-G(0.8113)+G|0.9183) H=08572 bits/message-symbol

(i) Fig. 1.16 shows the tree corresponding to initial state and the state at the beginning
second symbol interval.
of the
XX

3/7
3/4

4/7 1/4 XZ

1/7

YZ

1/7
1/3

3/7 2/3 YY

2/7
Initial state

States at the end of

2 symbol interval
Fig. 1.16: Tree diagram for Markov source of example 1.43
equation (1.43) with N= 1 (i.e., at the start of the 1 symbol interval equal to initial
state
 Information Theory and Cn
92

H(S)
= H(S)

4 log 7
7 3
0.9852 bits/message-symbol
G, =

conclude that
Comparing H and G, values, we

G, > H

with N =
2 (i.e., at the start of the 2nd symbol intervl
Again from equation (1.43),
we have,

G H(S)

log+
3
log 7+log7+log:
G, = 0.9212 bits/message-symbol

Again comparing values of H, G, and G,, we conclude that,

G,GH