1. Five symbols of the alphabet of DMS and their probabilities are given below.

S={So,S1,S2,S3,S4} P(S) = {0.1, 0.1, 0.2, 0.2, 0.4}.Code the symbols using Huffman coding.
Find the efficiency of the code

2. Find the Shannon –fanno code for the following seven messages with probabilities indicated.
S={S1,S2,S3,S4,S5, S6,S7},P(S)= {0.05,0.15,0.2,0.05,0.15,0.3,0.1}.( Nov’15)

Step1: List the source symbols (messages) in the order of decreasing probability.
Step2: Partition the set of symbols into two sets that are as closes as to being equiprobable as
possible.
Step3: Assign 0 to each message in the upper set and 1 to each message in the lower set.
Step4: Continue this process each time partitioning the sets with as nearly equal probabilities
until only one message in the partition.
 Shannon – Fano Coding:

Message Probability I II III IV Code word No.Of (lk)

for message Bits/Messages
S1 0.3 0 0 00 2
S2 0.2 0 1 01 2
S3 0.15 1 0 0 100 3
S4 0.15 1 0 1 101 3
S5 0.1 1 1 0 110 3
S6 0.05 1 1 1 0 1110 4
S7 0.05 1 1 1 1 1111 4
Average information per message (entropy) is given as,
𝑀
1
𝐻 = ∑ 𝑝𝑘 𝑙𝑜𝑔2 ( )
𝑝𝑘
𝑘=1

1 1 1
𝐻 = 𝑝1 𝑙𝑜𝑔2 ( ) + 𝑝2 𝑙𝑜𝑔2 ( ) + ⋯ + 𝑝7 𝑙𝑜𝑔2 ( )
𝑝1 𝑝3 𝑝7

1 1 1 1 1
= 0.3𝑙𝑜𝑔2 ( ) + 0.2𝑙𝑜𝑔2 ( ) + 0.15𝑙𝑜𝑔2 ( ) + 0.15𝑙𝑜𝑔2 ( ) + 0.1𝑙𝑜𝑔2 ( )
0.3 0.2 0.15 0.5 0.1
1 1
+ 0.05𝑙𝑜𝑔2 ( ) + 0.05𝑙𝑜𝑔2 ( )
0.05 0.05

= (0.3 × 1.73) + (0.2 × 2.32) + (0.15 × 2.74) + (0.15 × 2.74) +

(0.1 × 3.32) + (0.05 × 4.32) + (0.05 × 4.32) 𝑙𝑜𝑔10 𝑥
𝑙𝑜𝑔2 𝑥 =
𝑙𝑜𝑔10 2
= 0.59 + 0.464 + 0.411 + 0.411 + 0.332 + 0.216 + 0.216
= 3.32 × 𝑙𝑜𝑔10 𝑥
= 2.6 bits of information/message.

L = Average number of binary digits per message

𝑀

= ∑ 𝑝𝑘 𝑙𝑘
𝑘=1

= (0.3 × 2) + (0.2 × 2) + (0.15 × 3) + (0.15 × 3) + (0.1 × 3) + (0.05 × 4) + (0.05 × 4)

= 2.6 𝑏𝑖𝑡𝑠/𝑚𝑒𝑠𝑠𝑎𝑔𝑒𝑠

𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 𝑝𝑒𝑟 𝑚𝑒𝑠𝑠𝑎𝑔𝑒

Coding Efficiency(ɳ)=
𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑏𝑖𝑛𝑎𝑟𝑦 𝑑𝑖𝑔𝑖𝑡𝑠 𝑝𝑒𝑟 𝑚𝑒𝑠𝑠𝑎𝑔𝑒

𝐻 2.6
= × 100 = × 100 = 100%
𝐿 2.6

3. Construct a convolution encoder whose constraint length is 3 and has 3 modulo- 2 adders
and an output multiplexer. The generator sequences of the encoder
g(1)=(1,0,1),g(2)=(1,1,0),g(3)=(1,1,1). Draw the block diagram of the encoder. Find the encoder
output produced by the message sequence 10111
4. Derive the expression for mutual information and channel capacity.

Channel capacity theorem:

When Shannon’s theorem of channel capacity is applied specifically to a channel in

which noise is Gaussian is known as Shannon’s Hartley (or) Information capacity
theorem.

The channel capacity of Gaussian channel is

where, B = channel bandwidth S = signal power N = noise power

 Bandwidth - SNR Tradeoff Codes

Channel capacity of the Gaussian channel is given as,

Channel capacity depends on two factors (i.e.,) (i) bandwidth and (ii) signal to noise ration
(SNR)

1. Infinite capacity for noiseless channel

Consider a channel with no noise (N = 0)

2. Limited capacity for infinite bandwidth Consider a bandwidth ‘B = ∞’ that is increased

bandwidth, noise power (N) also increases. Then noise power is given by

3. Maximum value by increasing bandwidth is given as

The tradeoff between bandwidth and SNR is 1. To decrease the bandwidth, the signal
power has to be increased. 2. Similarly to decrease the signal power, the bandwidth
must be increased.

(ii) What are the types of error control coding ? Describe the working of viterbi decoding
algorithm.
Types of Codes:
The codes are mainly classified as block codes and convolutional codes.
Block codes:
These codes consists of ‗n‘ number of bits in one block re codeword. This codeword
consists of ‗k‘ message bits and (n-k) redundant bits. Such codes are called (n, k) block
codes.
Convolutional Codes:
The coding operation is discrete time convolution of input sequence with the
impulse response of the encoder. The convolutional encoder accepts the message bits
continuously and generates the encoded sequence continuously.
The codes can also be classified as Linear and non linear Codes.
Linear Codes:
If the two code words of the linear code are added by modulo-2 arithmetic, then it
produces third codeword in the code.
This is very important property of the codes, since other code words can be obtained
by addition of existing code words.
Nonlinear Code:
Addition of the nonlinear code words does not necessarily produce third codeword.
Examples of Error Control Coding:
Principle of redundancy:
Let us consider the error control coding scheme which transmits 000 to transmit symbol
‗0‘ and 111 to transmit symbol ‗1‘. Here note that there are two redundant bits in every
message (symbol) being transmitted. The decoder checks the received triplets and takes
the decision in favour of majority of the bits. For example, if the triplet is 110, then
there are two 1‘s. Hence decision is taken in favour of 1. Here note that there is certainly
error introduced in the last bit. Similarly if the received triplet is 001 or 100 or 010 then
the decision is taken in favour of symbol ‗0‘. The message symbol is received correctly
if no more than one bit in each triplet is in error. If the message would have been
transmitted without encoding then it is difficult to recover the original transmitted
symbols. Thus the redundancy in the transmitted message reduces probability of error at
the receiver.
Important Aspects of Error Control Coding:
The redundancy bits in the message are called Check bits. Errors can be detected
an corrected with the help of these bits. It is not possible to detect and correct all the
error in the message. Errors up to certain limit can only be detected and corrected. The
check bits reduce the data rate through the channel.
Methods of controlling Errors:
There are two main methods used for error control coding. Forward acting error
correction, Error detection with transmission.
Forward acting error correction:In this method the errors are detected and corrected
by proper coding techniques at the receiver (decoder). The check bits or redundant
bits are used by receiver to detect and correct errors.
Error detection with transmission:
In this method, the decoder checks the input sequence. When it detect any error, it discard
that part of the sequence and requests the transmitted for retransmission. The
transmitter then again transmits the part of the sequence in which error was detected.
Here note that the decoder does not correct the errors. It just detects the error and
sends requests to transmitter. This method has lower probability of error but it is slow.
Important terms in error control coding:
Code word: The encoded block of ‗n‘ bits is called a code word
Block length: The number of bits ‗n‘ after coding is called the block length of the code.
Code rate: The ratio of message bits (k) and the encoder output bits (n) is called as
code rate. It is given as, r=[k/n], 0<r<1.
Channel data rate:
It is the bit rate at the otput of the encoder. I f the bit rate at the input of encoder is Rs,
then channel data rate is given by,
 R0=(n/k) Rs
Code efficiency:
η= [message bits in a block/ transmitted bits for the
block]
weight of the code:
the number of non-zero elements in the transmitted code vector is called as vector
weight. It is denoted by w(A) where A – code vector.
Hamming distance:
The number of elements in which the two code vectors differ is called as hamming
distance. A=[111] ,B=[010] , D= 2
Efficiency of transmission:
Transmission efficiency can be achieved through
1. Multiplexing2.Compression --- Data compression and video compression
Multiplexing: Several data sources share a common transmission medium
simultaneously. Line sharing saves transmission costs.
Data compression:
-Reduces the size of data files to move more information with fewer bits.
-Used for transmission and for storage.
- Combines multiplexing to increase efficiency.
- It works on the principle of eliminating redundancy.
- Codes are substituted for compressed portion of data.
Lossless- reconstituted data is identical to original (ZIP, GIF)
Lossy- reconstituted data is only perceptually equivalent (JPEG, MPEG)
Video compression:
Three common standards are : 1. M-JPEG 2. TUT H.261 3. MPEG
MPEG processing steps:
Preliminary scaling and color conversion
Color sub-sampling
Discrete cosine transformation
Quantization
Run length encoding
Huffman coding

The Viterbi Algorithem:

Assume for simplicity a convolutional code with k=1, and up to 2k = 2 branches can enter
each state in trellis diagramAssume optimal path passes S. Metric comparison is done by
adding the metric of S into S1 and S2. At the survivor path the accumulated metric is
naturally smaller

For this reason the non-survived path can be discarded -> all path alternatives need not to
be considered
The maximum likelihood path:

The decoded ML code sequence is 11 10 10 11 00 00 00 whose Hamming distance to the

received sequence is 4 and the respective decoded sequence is 1 1 0 0 0 0 0 . Note that this is
the minimum distance path. (Black circles denote the deleted branches, dashed lines: '1' was
applied)

5.Write in detail the procedure of Shannon-fano coding scheme .(May’15)

Coding the message with different number of bits, according to their probability.
Definition Shannon Fano algorithm is one where less number of bits are used for highly
probable messages and more number of bits for rare occurrence.
Algorithm (or) procedure
1. List the source symbols or message in the descending(decreasing) order of their probability.
2. Partition the set into two sets that are equi-probable or close as equiprobable.
Assume 0 → to upper partition set 1 → lower partition set
3.Those partitions are further sub-divided into new partitioning sets are worth nearly equi-
probabilities and further partitioning is not possible.
Partitioning process is stopped when single message in a set was found.
4.Calculate code length(L),Entropy(H),Efficiency(η),code redundancy(r) and code variance
(σ2).

Solved Problem:
The discrete memoryless source has 6-symbols 0.25, 0.2, 0.12, 0.08, 0.3, 0.05.
Construct Shannon Fano code.
 1)Entropy (H)

(2)average code word length:

(3) efficiency (η):

(4) redundancy (r):

r=1−η
r = 1 − 0.99
r = 0.01
(5) variance (σ2):
6.Define entropy. Explain the properties of entropy.
The entropy of a source is a fundamental quantity in Information theory.
Consider a discrete random variable S , which may represent S the set of possible symbols to
be transmitted at a particular time, taking possible values , s 1,s 2,..., s N , with respective
probabilities p S (s 1), p S (s 2),..., p S (s N)
The entropy H ( S ) of S is the expected (or mean or average) value of the information obtained
by learning the outcome of S

This is the maximum attainable value.

Entropy
The minimum average number of binary digits needed to specify a source output (message)
uniquely is called “SOURCE ENTROPY”

When we observe the possibilities of the occurrence of an event, how surprising or uncertain
it would be, it means that we are trying to have an idea on the average content of the
information from the source of the event.

Entropy can be defined as a measure of the average information content per source
symbol. Claude Shannon, the “father of the Information Theory”, provided a formula for it
as H=−∑pi logb⁡pi

Where pi is the probability of the occurrence of character number i from a given stream of
characters and b is the base of the algorithm used. Hence, this is also called as Shannon’s
Entropy. The amount of uncertainty remaining about the channel input after observing the
channel output, is called as Conditional Entropy. It is denoted by H(x∣y)
Properties of entropy
A: For a source X with M different outputs: log 2M  H(X)  0
Entropy: Proof of A log2M=ln M log2 e
ln x= y=>x=ey
log2x= y log2e= ln x log2e
B: For a source X „related“ to a source Y: H(X)  H(X|Y)

Entropy: Proof of B

Joint Entropy: H(X,Y) = H(X) + H(Y|X)

also H(X,Y) = H(Y) + H(X|Y)
from this: H(X) – H(X|Y) = H(Y) – H(Y|X)

7.Five source messages are probable to appear as m1 =0.4, m2= 0.15, m3 = 0.15,m4 = 0.15, and
m5 = 0.15. Determine the coding efficiency for 1) Shannon-Fano coding 2) Huffman coding
(Nov’17)
Algorithm:

Step1: List the source symbols (messages) in the order of decreasing probability.
Step2: Partition the set of symbols into two sets that are as closes as to being equiprobable as
possible.
Step3: Assign 0 to each message in the upper set and 1 to each message in the lower set.
Step4: Continue this process each time partitioning the sets with as nearly equal probabilities until
only one message in the partition.
Shannon- Fano Coding:

Messages Probability I II III Code word No.Of (lk)

for message Bits/Messages
m1 0.4 0 0 00 2
m2 0.15 0 1 01 2
m3 0.15 1 0 10 2
m4 0.15 1 1 0 110 3
m5 0.15 1 1 1 111 3
Average information per message (entropy) is given as, 𝑙𝑜𝑔10 𝑥
𝑙𝑜𝑔2 𝑥 =
𝑀 𝑙𝑜𝑔10 2
1
𝐻 = ∑ 𝑝𝑘 𝑙𝑜𝑔2 ( )
𝑝𝑘 = 3.32 × 𝑙𝑜𝑔10 𝑥
𝑘=1

1 1 1
𝐻 = 𝑝1 𝑙𝑜𝑔2 ( ) + 𝑝2 𝑙𝑜𝑔2 ( ) + ⋯ + 𝑝5 𝑙𝑜𝑔2 ( )
𝑝1 𝑝3 𝑝5

1 1 1 1 1
= 0.4𝑙𝑜𝑔2 ( ) + 0.15𝑙𝑜𝑔2 ( ) + 0.15𝑙𝑜𝑔2 ( ) + 0.15𝑙𝑜𝑔2 ( ) + 0.15𝑙𝑜𝑔2 ( )
0.4 0.15 0.15 0.15 0.15

= (0.4 × 1.32) + (0.15 × 2.74) + (0.15 × 2.74) + (0.15 × 2.74) + (0.15 × 2.74)

= 0.528 + 0.411 + 0.411 + 0.411 + 0.411

= 2.172 bits of information/message.

L = Average number of binary digits per message

𝑀

= ∑ 𝑝𝑘 𝑙𝑘
𝑘=1

= (0.4 × 2) + (0.15 × 2) + (0.15 × 2) + (0.15 × 3) + (0.15 × 3)

= 2.3 𝑏𝑖𝑡𝑠/𝑚𝑒𝑠𝑠𝑎𝑔𝑒𝑠

𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 𝑝𝑒𝑟 𝑚𝑒𝑠𝑠𝑎𝑔𝑒

Coding Efficiency(ɳ)=
𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑏𝑖𝑛𝑎𝑟𝑦 𝑑𝑖𝑔𝑖𝑡𝑠 𝑝𝑒𝑟 𝑚𝑒𝑠𝑠𝑎𝑔𝑒

𝐻
= × 100
𝐿
2.172
= × 100 = 94.43%
2.3

2) Huffman coding:

Codeword No.Of (lk)

Messages Step1 Step2 Step3 Step4
Bits/Messages
m1 0.4 0.4 0.4 0.6 0 1 1

m2 0.15 0.3 0.3 0 0.4 1 000 3

m3 0.15 0.15 0.3 1 100 3

0
m4 0.15 0.15 010 3
0 1
110 3
m5 0.15 1
Average information per message (entropy) is given as, 𝑙𝑜𝑔10 𝑥
𝑙𝑜𝑔2 𝑥 =
𝑀 𝑙𝑜𝑔10 2
1
𝐻 = ∑ 𝑝𝑘 𝑙𝑜𝑔2 ( )
𝑝𝑘 = 3.32 × 𝑙𝑜𝑔10 𝑥
𝑘=1

L = Average number of binary digits per message

𝑀

= ∑ 𝑝𝑘 𝑙𝑘
𝑘=1

𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 𝑝𝑒𝑟 𝑚𝑒𝑠𝑠𝑎𝑔𝑒

Coding Efficiency(ɳ)=
𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑏𝑖𝑛𝑎𝑟𝑦 𝑑𝑖𝑔𝑖𝑡𝑠 𝑝𝑒𝑟 𝑚𝑒𝑠𝑠𝑎𝑔𝑒

𝐻
= × 100
𝐿
8.Write short notes on (1)Linear Block Codes(2)Viterbi Algorithm (Nov’14)
1) Linear Block Codes

In coding theory, the linear block code generally referred as a error correcting code for which the
obtained resultant codeword is the linear combination of any two codewords. In simple words, the linear
block code possesss the linearity property that is the sum of any two codewords is also a codeword.
These linear block codes are divided in to the block codes and convolutional codes, eventhough turbo
codes is the combination of these two types. More efficient encoding and decodig algorithms are
provided by the linear codes when compared with the other codes.

2) Viterbi Decoding Algorithm

Maximum Likelihood (ML) decoding rule

Viterbi Decoding Algorithm

An efficient search algorithm Performing ML decoding rule.
Reducing the computational complexity.
Basic concept
 Generate the code trellis at the decoder
 The decoder penetrates through the code trellis level by level in search for the transmitted code
sequence
 At each level of the trellis, the decoder computes and compares the metrics of all the partial paths
entering a node
 The decoder stores the partial path with the larger metric and eliminates all the other partial paths.
The stored partial path is called the survivor

Use of the Viterbi Algorithm

• Low Signal to noise ratio ratio
• Radio link
• Satellite connection
• Digital TV (MPEG-2/DVB)
• Geostationary satellite networks (VSAT)
• Speech recognition
• Magnetic recoring
• Direct broadcast satellite systems (DBS)
Diagrams help to understand the algorithm
• State diagram
• Trellis diagram

Advantages
• Self-Correction of the Code
• Minimization of the Transmitting Energy
• Minimization of the Bandwidth
9.The generator polynomial of a (7, 4) cyclic code is given by G(D) =1 + D + D2.compute all the non-
systematic codewords. (May’16)
Solution:
11.Explain the concept of code generation and decoding of correlation codes. (Nov’16)
12. a)State and prove Shannon’s Noiseless coding theorem. [APR ‘18]
(b) Describe the concept of source coding theorem.
(b) Describe the concept of source coding theorem.

1. Source symbols encoded in binary

2. The average code length must be reduced
3. Remove redundancy ⇒ reduces bit-rate
Consider a discrete memoryless source on the alphabet

Let the corresponding probabilities be

and codelengths be

Then, the average code length(average number of bits per symbol) of the source is defined as

If L min is the minimum possible value of 𝐿̅, then the coding efficiency of the source is given
by η.