' $

Channel Coding
Channel Coding is done to ensure that the signal transmitted is
recovered with very low probability of error at the destination.

Let X and Y be the random variables of symbols at the source and

destination respectively. The description of the channel is shown in
the Figure. 1

x y0
0
x y
1 1
Source, X Destination, Y
X Channel Y

x y
J−1 K−1

Figure 1: Description of a channel

& %
' $
The channel is described by a set of transition probabilities

P (Y = yk |X = xj ) = p(yk |xj ), ∀j, k

such that

K−1
X
p(yk |xj ) = 1, ∀j.
k=0

The joint probability is now given by

p(xj , yk ) = P (X = xj , Y = yk )
= P (Y = yk |X = xj )P (X = xj )
= p(yk |xj )p(xj )
& %
' $

• Binary Symmetric Channel

– A discrete memoryless channel with J = K = 2.
– The Channel has two input symbols(x0 = 0, x1 = 1) and
two output symbols(y0 = 0, y1 = 1).
– The channel is symmetric because the probability of
receiving a 1 if a 0 is sent is same as the probability of
receiving a 0 if a 1 is sent.
– The conditional probability of error is denoted by p. A
binary symmetric channel is shown in Figure. 2 and its
transition probability matrix is given by
 
1−p p
 
p 1−p

& %
' $
1−p
x =0 y =0
0 0

x =1 y =1
1 1
1−p

Figure 2: Binary Symmetric Channel

• Mutual Information

If the output Y is the noisy version of the channel input X and

H(X ) is the uncertainity associated with X, then the
uncertainity about X after observing Y , H(X |Y) is given by
& %
' $

K−1
X
H(X |Y) = H(X |Y = yk )p(yk ) (1)
k=0
K−1
" #
X J−1
X 1
= p(xj |yk )p(yk )log2 (2)
p(xj |yk )
k=0 j=0
K−1
" #
X J−1
X 1
= p(xj , yk )log2 (3)
p(xj |yk )
k=0 j=0

The quantity H(X |Y) is called Conditional Entropy. It is the

amount of uncertainity about the channel input after the
channel output is observed. Since H(X ) is the uncertainity in
channel input before observing the output, H(X ) − H(X |Y)
represents the uncertainity in channel input that is resolved by
observing the channel output. This uncertainity measure is

& %
termed as Mutual Information of the channel and is denoted by
' $
I(X ; Y).

I(X ; Y) = H(X ) − H(X |Y) (4)

Similarly,

I(Y; X ) = H(Y) − H(Y|X ) (5)

• Properties of Mutual Information

– Property 1:
The mutual information of a channel is symmetric, that is

I(X ; Y) = I(Y; X ) (6)

& %
' $
Proof:

J−1
" #
X 1
H(X ) = p(xj ) log2 (7)
j=0
p(xj )
J−1
" # K−1
X 1 X
= p(xj ) log2 p(yk |xj ) (8)
j=0
p(xj )
k=0
J−1 K−1
" #
XX 1
= p(yk |xj )p(xj ) log2 (9)
j=0 k=0
p(xj )
J−1 K−1
" #
XX 1
= p(xj , yk ) log2 (10)
j=0
p(xj )
k=0

Substituting Eq.3 and Eq.10 in Eq.4 and then combining,

& %
we obtain
' $

J−1
" #
X K−1
X p(xj |yk )
I(X ; Y) = p(xj , yk ) log2 (11)
j=0 k=0
p(xj )

From Bayes’ rule for conditional probabilities, we have

p(xj |yk ) p(yk |xj )

= (12)
p(xj ) p(yk )
Hence, from Eq.11 and Eq.12

J−1
" #
X K−1
X p(yk |xj )
I(X ; Y) = p(xj , yk ) log2 = I(Y; X ) (13)
j=0 k=0
p(yk )

– Property 2:

& %
The mutual is always non-negative, that is
' $

I(X ; Y) ≥ 0

Proof:
We know,

p(xj , yk )
p(xj |yk ) = (14)
p(yk )
Substituting Eq. 14 in Eq. 13, we get

J−1
" #
X K−1
X p(xj , yk )
I(X ; Y) = p(xj , yk ) log2 = I(Y; X )
j=0 k=0
p(xj )p(yk )
(15)
Using the following fundamental inequality which we

& %
derived discussing the properties of Entropy,
' $

K−1
!
X qk
pk log2 ≤0
pk
k=0

Drawing the similarities between the right hand side of the

above inequality and the left hand side of Eq. 13, we can
conclude that

I(X ; Y) ≥ 0

– Property 3:

The mutual information of a channel is related to the joint

entropy of the channel input and channel output by

& %
I(X ; Y) = H(X ) + H(Y) − H(X , Y)
' $
where, the joint entropy H(X , Y) is defined as

J−1
!
X K−1
X 1
H(X , Y) = p(xj , yk ) log2
j=0 k=0
p(xj , yk )

Proof:

J−1
!
X K−1
X p(xj )p(yk )
H(X , Y) = p(xj , yk ) log2 + (16)
j=0 k=0
p(xj , yk )
J−1
!
X K−1
X 1
p(xj , yk ) log2 (17)
j=0 k=0
p(xj )p(yk )
J−1
!
X K−1
X 1
= I(X ; Y) + p(xj , yk ) log2 (18)
j=0 k=0
p(xj )p(yk )

& %
But,
' $

J−1
!
X K−1
X 1
p(xj , yk ) log2 (19)
j=0 k=0
p(xj )p(yk )
J−1
! K−1
X 1 X
= log2 p(xj , yk )+ (20)
j=0
p(xj )
k=0
K−1
! J−1
X 1 X
log2 p(xj , yk ) (21)
p(yk ) j=0
k=0
J−1
! K−1 !
X 1 X 1
= p(xj ) log2 + p(yk ) log2 (22)
j=0
p(xj ) p(yk )
k=0
= H(X ) + H(Y) (23)

Therefore, from Eq. 18 and Eq. 23, we have

H(X , Y) = −I(X ; Y) + H(X ) + H(Y)

& %
' $

Problems
The following problems may be given as exercises.
1. Show that the mutual information is zero for a deterministic
channel.
2. Prove that I(X ; Y) = min(H(Y), H(X ))
3. Prove that I(X ; Y) = min(log(|Y|), log(|X |))

& %