Information Theory and

Coding
Dr. Vidya Kudva
Associate Professor
ECE department
NMAMIT Nitte
Introduction
• Devices dedicated to communication or storage are integral part of day to day life
• Basic goal of a communication system is to transmit some information from
source to destination

• Source – sender of the information

• Sink – receiver that receive the information
• Channel – transmission path/medium of the information between the source and
sink
Introduction

• Information consists of letters, digits, symbols, sequence of letters,

digits, symbols etc.
Information theory versus coding theory
• Information theory is a branch of probability theory
with extensive applications to communication systems.
• It is a study of Communication system plus
Mathematics.
• Study of statistical structure of electrical
communication equipment
• Principally founded by Claude E. Shannon
Model and basic operations of information
processing systems
• Information processing systems can be represented abstractly by the
block diagram as in Figure

• Source: from which the information originates.

• The information source may be many things; for example, a book, music, or
video are all information sources in daily life.
Model and basic operations of information
processing systems

• The source output is processed by an encoder to facilitate the

transmission (or storage) of the information.
• In communication systems, this function is often called a transmitter
• In general, three basic operations can be executed in the encoder:
source coding, channel coding, and modulation
Model and basic operations of information
processing systems
• For source coding, the encoder maps the source
output into digital format.
• The mapping is one-to-one, and the objective is to
eliminate or reduce the redundancy
• Redundancy is that part of the data which can be
removed from the source output without harm to
the information to be transmitted.

• So, source coding provides an efficient

representation of the source output.
Model and basic operations of information
processing systems

• For channel coding, the encoder introduces extra redundant data in a

prescribed fashion so as to combat the noisy environment in which
the information must be transmitted or stored.
Model and basic operations of information
processing systems

• Discrete symbols may not be suitable for transmission over a physical

channel or recording on a digital storage medium.
• Therefore, we need proper modulation to convert the data after
source and channel coding to waveforms that are suitable for
transmission
Model and basic operations of information
processing systems

• Channel: The output of the encoder is then transmitted through

some physical communication channel
• Example:
• wireless radio transmission based on electromagnetic waves,
• telephone communication through copper cables,
• and wired high-speed transmission through fiber optic cables.
Model and basic operations of information
processing systems

• Information conveyed through the channel must be recovered at the

destination and processed to restore its original form.
• This is the task of the decoder.
• In the case of a communication system, this device is often referred
to as the receiver.
 Model and basic operations of information
processing systems

• The signal processing performed by the decoder can be viewed as

the inverse of the function performed by the encoder.

Decoder Demodulato Channel Source

r Decoder Decoder
Model and basic operations of information
processing systems

• The output of the decoder is then presented to the final user, which
we call the information sink.
ELEMENTS OF A DIGITAL COMMUNICATION
SYSTEM
Elements of digital communication system
• Information Source
• Messages can be analog or digital in nature
• Examples of analog signals audio or video
• Output of a teletype machine is an example for digital signal
• Unit-1 covers in detail quantification of information

15
Elements of digital communication system
• Source Encoder/Decoder
• converting the output of either an analog or digital source into a sequence of
binary digits is source encoding
• Decoder converts binary output into a symbol sequence
• Process of encoding of source information and decoding is dealt in Unit-2

16
Elements of digital communication system
• Channel Encoder/Decoder
• Channel encoder introduces controlled amount of redundancy in the binary
information sequence
• This can be used at the decoder to overcome the effects of noise and
interference encountered in the transmission of the signal through the
channel
• This is dealt in detail in Unit-4

17
Elements of digital communication system
• Communication Channel
• The physical medium that is used to send the signal from the transmitter to
the receiver.
• In wireless transmission, the channel may be the atmosphere (free space).
• For telephone channels use physical media, including wire lines, optical fiber
cables, and wireless (microwave radio).
• Different kinds of noises get added in channel
• Unit-3 covers different kind of channels and their representations

18
Elements of digital communication system
• Modulator/Demodulator
• Modulator performs different kinds of modulation for reducing the effect of
noise
• Demodulator recovers the signal from modulated signal

Q1: With neat diagram, explain an information system

19
Course Content Unit-I
Information Theory: Introduction, Measure of
information, Average information content
(entropy) of symbols in long independent
sequences, Information rate, Properties of
entropy, Extension of discrete memory less
(zero-memory) sources, Average information
content (entropy) of symbols in long dependent
sequences, Markoff statistical model for
information source, Entropy and information
rate of Markoff sources.
Course Content Unit-II
Source Coding: Properties of codes- Block
codes, Non-singular codes, Uniquely decodable
codes. Instantaneous codes and Optimal codes,
Prefix of a code, Test for instantaneous
property, Construction of Instantaneous code,
Decision tree, Kraft’s inequality, Source coding
theorem (Shannon’s Noiseless coding theorem),
Shannon’s encoding algorithm, Shannon
Fano Algorithm, Huffman minimum redundancy
code (binary, ternary and quaternary),
Code efficiency and redundancy, Extended
Huffman Coding, Arithmetic Codes, Lempel –
Ziv Algorithm.
Course Content Unit-III
Channels and Mutual Information: Introduction,
Discrete communication channels,
Representation of a channel, Probability
relations- Apriori, Posteriori entropy,
Equivocation, Mutual information, Properties,
Rate of information transmission over a
discrete channel, Capacity of a discrete
memoryless channel, Shannon’s theorem on
channel capacity (Shannon’s second theorem),
Special channels- Symmetric, Binary symmetric,
Binary erasure, Noiseless, Deterministic and
cascaded channels, Estimation of channel
capacity by Muroga’s method, Continuous
channels, Shannon-Hartley theorem and its
implications, Shannon’s limit, Rate Distortion
Theory.
Course Content Unit-IV
Linear Block Codes: Introduction to Fields and
Vector Spaces, Types of errors, Examples,
Methods of controlling errors, Types of codes,
Linear Block Codes- Matrix description,
Encoding circuit, Syndrome and error detection,
Syndrome circuit, Hamming weight, Hamming
distance, Minimum distance of a block code
error detection and correction capabilities
of a linear block code, Single error-correcting
Hamming codes, Table lookup decoding using
standard array, General decoder for a linear
block code.
Binary Cyclic Codes: Algebraic structures of
cyclic codes, Encoding using (n-k) bit shift
register, Syndrome calculation.
Course Content Unit-V
Other Block Codes: BCH codes, RS codes, Golay
codes, Shortened cyclic codes, Burst
error correcting codes, Burst and Random error
correcting codes.
Convolutional Codes: Encoders, Time domain
approach, Transform domain approach,
State diagram, Code tree, Trellis, Sequential
search and Viterbi algorithm, Principle of
Turbo coding.
References
• TEXT BOOKS
• T1 K. Sam Shanmugham, “Digital and Analog Communication Systems", John
Wiley Publications, 1996.
• T2 Simon Haykin, “Digital Communications”, John Wiley Publications, 2003.
• T3 Shu Lin, Daniel J. Costello, “Error Control Coding", 2nd Edition, Pearson
Prentice Hall, 2004.
• REFERENCE BOOKS
• R1 Muralidhar Kulkarni and K S Shivaprakasha, “Information Theory and Coding",
Wiley (India), 2015.
• R2 Glover and Grant, “Digital Communications”, 2nd Edition, Pearson, 2008.
• R3 Abramson, “Information Theory & Coding", McGraw-Hill, 1963.
Course Aim
• To study mathematical theory of few blocks in communication
systems.
Evaluation Scheme
• CIE evaluation

• SEE evaluation
• Semester End Examination (SEE) is a written examination, of three
hours duration of 100 marks, with 50% weightage
Unit-1
Information Theory
Information Source
• Information sources can be classified into two categories based on
the nature of their outputs:
• analog information sources and
• discrete information sources
• Analog information sources, such as a microphone actuated by
speech, or a TV camera scanning a scene, emit one or more
continuous amplitude signals (or functions of time).
• The output of discrete information sources such as a teletype or the
numerical output of a computer consists of a sequence of discrete
symbols or letters.
Information Source
• However, discrete values are preferred in modern communication
systems as they can be transmitted more reliably than analog signals
• An analog information source can be transformed into a discrete
information source through the process of sampling and quantizing.
Information Source
• Sampling: Signal is sampled at equally
spaced intervals of time

• Quantization:quantize the observed value, i.

e. each sample value is approximated by the
nearest level in a finite set of discrete levels.
• By mapping each quantized sample to a
codeword consisting of a prescribed number
of code elements, the information is then
sent as a stream of digits.
Information Source
• symbols 0 and 1 of the binary code
are represented by zero and one
volt, respectively.
• Each sampled value is quantized
into four binary digits (bits) with the
last bit called sign bit indicating
whether the sample value is
positive or negative.
• The remaining three bits are chosen
to represent the absolute value of a
sample in accordance with Table
Information Source
• As a result of the sampling and quantizing
operations, errors are introduced into the digital
signal.
• These errors are nonreversible in that it is not
possible to produce an exact replica of the
original analog signal from its digital
representation.
• However, the errors are under the designer’s
control.
• By proper selection of the sampling rate and
number of the quantization levels, the errors due
to the sampling and quantizing can be made so
small that the difference between the analog
signal and its digital reconstruction is not
discernible by a human observer.
Measure of information
• An information source generates information
• Amount of information conveyed by the source is different for
different symbols it emits
• It rained heavily in Cherrapunji yesterday
• There was a heavy rainfall in Rajasthan last night
Information Measure
Consider two Messages

A Dog Bites a Man High probability Less information

A Man Bites a Dog Less probability High Information

So we can say that

Information α (1/Probability of Occurrence)

 Information Measure
•
 Information Measure
•
 Information Measure
•
 Information Measure
•
 Information Measure
•
 Information Measure
•
Conversion Between Units

Q: List the reasons for choosing logarithmic expression for measuring information
Example
Example
• A Source generates one of four symbols during each interval with
probabilities p1=1/2, p2=1/4, p3= p4=1/8. Find the Information
content of these messages.
•
• Consider a binary system emitting symbols with probabilities 0.7 and
0.3. Find the self-information of each of the symbols in nats and
Hartley.
• A man was informed that when a pair of dice was rolled the result is
7. What is the amount of information conveyed?
Average information content (entropy) of symbols
in long independent(zero memory) sequences
• A zero memory source or a discrete memoryless source is the one in
which the emission of the current symbol is not dependent on the
emissions of the previous symbols
• Suppose a source emits M possible symbols s1, s2, …..sM having
Probability of occurrence p1,p2,…….pM
Average information content (entropy) of symbols
in long independent(zero memory) sequences
For a long message having symbols N (>>M)

s1 will occur p1N times,

s2 will occur p2N times so on…….

• Since s1 occurs p1N times so information Contribution by s1 is

I1= p1N log2(1/p1).

• Similarly information Contribution by s2 is

I2=p2N log2(1/p2). And So on…….

 Average information content (entropy) of symbols
in long independent(zero memory) sequences
•
 Average information content (entropy) of symbols
in long independent(zero memory) sequences
•

It means that in long message we can expect H bit of information per symbol. Another name of H(s) is entropy.
 Average Rate of Information
• Average rate of the Information can be defined for an information
system if the symbol rate of the system is known.

Q:Derive the expression for average information content/entropy of a zero memory/discrete memoryless source
• A discrete memoryless source emits five symbols in every 2ms.The
symbol probabilities are {0.5, 0.25, 0.125, 0.0625, 0.0625}. Find the
average information rate of the source.
• The output of an information source contains 160 symbols, 128 of
which occur with a probability of 1/256 and remaining with a
probability of 1/64 each. Find the average information rate of the
source if the source emits 10000sym/s.
•
• A binary source is emitting an independent sequence of ‘0’ and ‘1’ s
with probability ‘p’ and ‘1-p’ respectively. Plot the entropy of the
source versus p
 Properties of Entropy
•
Properties of Entropy
•

Derive the expression for the maximum limit on the entropy of a source
Properties of Entropy
Source efficiency( )
It is defined as the ratio of average information conveyed
by the source to that of the maximum average
information
• Consider a system emitting one of the three symbols A, B and C with
respective probabilities 0.7, 0.15 and 0.15. Calculate its efficiency
and redundancy.
• A certain digital frame consists of 15 fields. The first and the last field
of each frame are same. The remaining 13 fields can be filled by any
of the 16 symbols with equal probability. Find the average
information conveyed by the frame. Also find the average rate of
information if 100 frames are transmitted every second.
• A data frame consists of ten fields. First field in each frame is the
same for some synchronization purpose. However the remaining
fields can be filled by any of 32 symbols with equal probability. Find
the average rate of information if 500 frames are transmitted in
every second.
 Extension of zero memory source
•
Extension of zero memory source
Consider the second order extension of the source S. This source is
denoted as S2 . Number of symbols in S2 is given by
(number of symbols in basic source)extension
Hence second order extension of S will have 22= 4 combinations
Extension of zero memory source
Corresponding probabilities are given by
Extension of zero memory source
Therefore entropy of second order extension of the source H(S2) is
given by
Extension of zero memory source
Extension of zero memory source

• Similarly we can show that

 Average information content of long dependent
sequences
Independent Sequences: A zero memory source or a discrete
memoryless source is the one in which the emission of the current
symbol is not dependent on the emission of previous symbols
Most of the practical sources we consider in real time are with
memory/ dependent sources where emission of the current symbol
is dependent on the previous symbols.
• Source emitting English symbol :
– After Q next letter is U with 100% probability
– After a consonant, probability of occurrence of vowel is more. And vice versa.
• Statistical dependence reduces amount of information coming out
of source.
 Average information content of long dependent
sequences
One of the most popular ways of representing dependence in the
data is through the use of Markov/Markoff models
-Let { xn } be a sequence of observations
Assume that each output symbol depends on previous k ones

It is called a kth order Discrete Markov Chain (DMC)

The values taken on by the set sn−1 ,…, sn−k are called the states of the
process. If the size of the source alphabet is l, then the number of
states is lk
Average information content of long dependent
sequences
-The first order DMC (the knowledge of 1 symbol in the past is enough)

-Theentropy of a finite state process with states Si is simply the

average value of the entropy at each state
 Average information content of long dependent
sequences
• A system with memory can be represented using state diagram
• State diagram represents all possible states along with transition probabilities
• Symbols emitted in each of the transitions are depicted in state diagram
• From state diagram a tree diagram can be constructed
• Probabilities of message length of L can be determined using tree diagram
For the first order Markov source shown in Figure, draw the tree
diagram representing the states at the end of second symbol
interval and find the corresponding probabilities .
Assume P(1)=P(2)=1/2
 Entropy and Information rate

•
 Average Information Rate per symbol

•
 Example
•
• To write state probabilities, we have to first write state equations.
• State equations can be written by considering all incoming
probabilities of the state.
• Consider the equation for state 1

………………..(1)
• To write state probabilities, we have to first write state equations.
• State equations can be written by considering all incoming
probabilities of the state.
• Consider the equation for state 2

………………..(2)
• Also, sum of all state probabilities is equal to 1

………………..(3)
•
•
•
•
•
•
 Homework
•
• Design an information system, which gives the information every
year, for about 200 students passing out from a certain university.
The students can get into one of the three fields as given below.
Go abroad for higher studies A
Join MBA or Civil Services B
Join Industries in India C
Based on the data given below, construct the model for the source and
find the source entropy.
❖ On the average 100 students are going abroad.
❖ Out of 100 going abroad this year, 50 were reported to going abroad next
year, while 25 each went to MBA and Civil Services or joined Industries in
India.
❖ Out of 100 remaining in India this year, 50 continued to do so, while 50 went
abroad next year.
❖ Those joining MBA and Civil Services or Industry could not swap the two
fields next year.
• Design a system to report the heading of collection of 400 cars. The
heading levels are heading straight (S), turning left (L) and turning
right (R). This information is to be transmitted every second.
Construct a model based on the test data given below.
On an average during reporting interval, 200 cars were heading straight, 100 were
turning left and remaining were turning right.
Out of 200 cars that reported heading straight 100 of them reported going straight
during next reporting period. 50 of them turning left and remaining turning right during
the next period.
Out of 100 cars that reported as turning during a signaling period, 50 of them continued
the turn and remaining headed straight during next signaling interval.
The dynamics of the cars did not allow them to change their heading from left to right or
right to left during subsequent reporting periods.
• Find the entropies of each states, source entropy and rate of
information.