TE361: Information Theory

Channel Coding

“If you can’t explain it simply, you don’t understand it well enough”
- Albert Einstein
Channel Coding
 Introduction
• In practice
– Most media are not perfect – noisy channels:
• Wireless link
– satellite
• Modem line
• Hard disk

• Can we
– recover the original message (without errors) after passing
through a noisy channel?
• How
– can we communicate in reliable fashion over noisy channels?
 Introduction
• The communication model
– we are using consists of a source that generates discrete symbols/
digital information
• This information is sent to a destination through a CHANNEL
• The channel
– can be associated with noise so we have two cases
1. Noiseless case
– The channel in this case transmits symbols without causing any
errors
2. Noisy case
– The channel in this case introduces noise that causes errors in
the received symbols at the destination
» Noise can
• be defined as any unwanted signal or effect in addition
to the desired signal
» To reduce the errors due to noise we would add systematic
redundancy to the information to be sent
• This done through CHANNEL CODING
 Introduction
• Assumptions
1. That the information generated at the source is ‘source-coded’
or compressed into a string of binary digits
2. Discrete Memoryless Channel (DMC)
• Each symbol is sent over the channel independently of the previous
symbols sent
• the behavior of the channel and the effect of the noise at time t will
not depend on the behavior of the channel or the effect of the noise
at any previous time
 Introduction
• We defined
– information rate R and learnt that arbitrarily reliable
communication is possible for any R < C
• In this lecture
– we will study noisy channel models and compute the channel
capacity for such channels
– We will design and analyze block codes
 Introduction
• Shannon put forth a stochastic model of the channel
– Input alphabet A
– Output alphabet B
– Probability transition matrix P(b/a)
• that expresses the probability of observing the output symbol b given
that the symbol a was sent
• describes the transmission behavior of the channel

• The input and output symbols

– Arise from two random variables and the channel models the
joint relationship between the variables
• Focus
– is on measuring the information carrying capacity of DMC in the
presence of noise
 Discrete Memoryless Channel
• The information carrying capacity
– is defined as the maximum mutual information

C  max I ( A ; B)
p( x )

• The mutual information

– I(A; B) of a channel with input A and output B
• measures the amount of information a channel is able to convey
about the source
 Noiseless Channel
• If the channel is noiseless
– then the mutual information is equal to the information content of
the source A
– That is
I ( A; B)  H ( A)

– For noiseless binary channel, C = 1 bit

 Noisy Channel
• In the presence
– of noise there is an uncertainty H(A/B), that reduces the mutual
information to

I ( A; B)  H ( A)  H ( A / B)

– Here
• H(A) is source entropy
• H(A/B) is the average information lost per symbol
 Discrete Memoryless Channel
• We need
– a mathematical model of data/information transmission over
unreliable communication channels
• Popular DMC models include
– The Binary Symmetric Channel (BSC)
– The Binary Erasure Channel (BEC)
 Binary Symmetric Channel (BSC)
• This is a channel
– when you input a bit, 0 or 1, with
• probability (1 - p) it passes through the channel intact
• probability p it gets flipped to the other parity

• That is
– the probability of error or bit error rate (BER) is p

P y  1 / x  0  P y  0 / y 1  p

– the channel experienced bit inversion

– the parameter p fully defines the behavior of the channel
 Binary Symmetric Channel (BSC)

 P( y  0 / x  0) P( y  1 / x  0 
P 
 P ( y  0 / x  1) P ( y  1 / x  1) 

1  p p 
P 
 p 1  p 
• The mutual information is
I ( X ; Y )  H  X   H  X / Y   H (Y )  H (Y / X )

C  1  H ( p)
– where H(p) is the binary entropy function
 Binary Erasure Channel (BEC)
• Another effect
– that noise may have is to prevent the receiver from deciding
whether the symbol was a 0 or 1 (loss of signal)
• In this case
– the output alphabet includes an additional symbol “e”
• called the erasure symbol that denotes a bit that was not able to be
detected
• For a
– binary input {0, 1} the output alphabet consists of three symbols
{0, e, 1}
• This information channel
– is called a Binary Erasure Channel
– experienced bit lost
– BEC does not model/capture the effect of bit inversion
 Binary Erasure Channel (BEC)
• The probability of error or erasure probability is α
• That is
P y  e / x  0  P y  e / x 1  

 P( y  0 / x  0) P( y  e / x  0) P( y  1 / x  0)
P 
 P ( y  0 / x  1) P ( y  e / x  1) P ( y  1 / x  1) 

1    0 
P
0  1   
 Binary Erasure Channel (BEC)
• The BEC capacity is
I ( X ; Y )  H  X   H  X / Y   H (Y )  H (Y / X )

C 1  

• BEC
– Important model for wireless, mobile and satellite communication
channels
 Discrete Memoryless Channel
• To fully specify
– the behavior of an information channel it is necessary to specify
• the characteristics of the input
• as well as the channel matrix
• We will assume
– that the input characteristics are described by a probability
distribution over the input alphabet with
• p(xi) denoting the probability of symbol xi being input to the channel
• If the channel
– is fully specified then the output can be calculated by

P y j   p y j / x i  px i 
r

i 0
 Discrete Memoryless Channel
• Note well
– py j / xi  is forward probabilities
– px / y  is backward probabilities
i j

• The backward probabilities

– can be calculated by using Bayes’ rule

P xi , y j  P  y j / xi  Pxi 
p xi / y j   
P y j  Py j 
 Problem
• Consider
– the binary information channel fully specified by
P  x  0  2 / 3 3 / 4 1 / 4 
P  
Px  1  1 / 3
and
1 / 8 7 / 8

– Represented as

– Calculate the output probabilities

 Solution
• The output probabilities are calculated using

P y j   p y j / x i  px i 
r

i 0

• The probability that the output is 0 is

P y  0  P y  0 / x  0 Px  0  P y  0 / x  1 Px  1
 3  2   1  1  13
        
 4  3   8  3  24

• The probability that the output is 1 is

P y  1  1  P y  0 
11
24
 Solution
• The backward probabilities are:
P( y  0 / x  0) P( x  0)
P  x  0 / y  0 
P( y  0)
(3 / 4)( 2 / 3) 12
 
13 / 24 13

P  x  1 / y  0  1  P  x  0 / y  0  1 
12 1

13 13
P( y  1 / x  1) P( x  1)
P x  1 / y  1 
P( y  1)
(7 / 8)(1 / 3) 7
 
11 / 24 11

Px  0 / y  1  1  Px  1 / y  1  1 
7 4

11 11
 Solution
• The average uncertainty
– we have about the input after the channel out yj is observed is
given by

H X / y j   P( x / y j ) log 2
1
xX p( x / y j )

– Let say we observe an output of y = 0

• then

H  X / y  0  Px  0 / y  0log 2  Px  1 / y  0log 2

1 1
p( x  0 / y  0) p( x  1 / y  0)
 13  1
log 2    log 2 13  0.387
12

13  12  13
 Solution
– Let say we observe an output of y = 1
• Then

H  X / y  1  P x  0 / y  1 log 2  P x  1 / y  1 log 2
1 1
p( x  0 / y  1) p( x  1 / y  1)
4  11  7  11 
 log 2    log 2    0.946
11  4  11 7

• The conditional entropy

– is given by
H X / Y    P( y ) H ( X / y )
j j

 P( y  0) H ( X / y  0)  P( y  1) H ( X / y  1)
 13   11 
  0.387    0.946
 24   24 
 0.643
– Thus
• the average information lost per symbol is through the channel is 0.643
 Solution
• The source entropy is

H  X    p( x ) log 2
1
p( x )
1 1
 p( x  0) log 2  p( x  1) log 2
p( x  0) p( x  1)
2  3 1  3
 log 2    log 2    0.918
3 2 3 1

• The capacity
– of the channel in the presence of noise is

C  H ( X )  H ( X /Y )
 0.918  0.643  0.275 bits / channel use
 Channel Coding
• If the data
– at the output of a communication system has errors that are too
frequent for the desired use then
• channel coding is needed to reduce the error rate and improve
– the reliability of the transmission
• In 1948
– Shannon demonstrated that by proper encoding of information
• Errors induced by a noisy channel can be reduced to any desired
level without sacrificing the rate of information transfer
• Channel coding
– is to represent the source information in a manner that minimizes
the error probability in decoding into symbols
– is used by the receiver to detect or correct some (or all) of the
errors introduced by the channel
 Channel Coding
• The central concept
– in detecting or correcting errors is redundancy
• We need to send some extra bits with the data. The redundant bits
allow the receiver to detect or correct corrupted bits
– These redundant bits are added by the sender and removed by
the receiver
• The operation of channel encoder
 Channel Coding
• Redundancy is achieved through various coding schemes
– The sender adds redundant bits through a process that creates a
• relationship between the redundant bits and the actual data
bits
– The receiver checks the relationship between two sets of bits to
detect errors
• The ratio of
– redundant bits to data bits and the robustness of the process are
important factors in any coding scheme
• We can divide coding schemes into the following categories
– Block codes
– Convolutional codes
– Turbo codes
 Channel Coding
• The primary objective
– of coding is that the decoder can determine if the received word
is a valid codeword or it is a codeword which has been corrupted
by noise
– Ideally the decoder should be able to decide which codeword
was sent even if the transmitted codeword was corrupted by
noise

• Error control
– Detection (Automatic Repeat reQuest -ARQ)
• When a receiver detects error in a block of data
– It requests that the data be retransmitted
– Correction (Forward Error Correction – FEC)
• The transmitted data is encoded so that error cause by channel
noise can be detected and corrected
 Block Codes
• The input
– to the block coder is a stream of information bits
• The coder
– segments this bit stream into blocks of k information bits called
datawords and
• for each block it calculates a number of r check bits or it picks the r
check bits from a tabulated set of values
• The coder
– then transmits the codeword of n = k + r channel bits this is
called (n , k) block code
• With
– k bits we can create a combination of 2k datawords
– n bits we can create a combination of 2n codewords
Block Codes
 Block Codes
• Since n > k
– The number of possible codewords is larger than the number of
possible datawords
– We have 2n – 2k codewords that are not used (invalid or illegal)
• The trick
– in error detection is the existence of these invalid codewords
• If the receiver receives an invalid codeword this indicates that the
data was corrupted during transmission
• Example of block codes
– Hamming codes
– Cyclic codes
 Error Detection
• How can errors be detected using block codes?
– The receiver has (or can find) the list of invalid codewords
– The original codeword has changed to an invalid codeword

• An error-detecting code can only detect the types of

errors for which it is designed
– Other types of errors may remain undetected

• The ability
– of a block code to detect/correct errors is a function of the code
distance
 Hamming Distance
• It is one of the central concepts in coding for error control
• The Hamming distance between two words (of the same size)
– Is the number of differences between the corresponding bits
• The hamming distance between two words x and y is denoted by
d(x, y)
• The hamming distance can easily be found if we apply the exclusive
OR (XOR) operation on the two words and count the number of 1s
in the result
• For the
– two codewords 000 and 011
• The hamming distance d(000, 011) = 2
– two codewords 10101, 11110
• The hamming distance d(10101, 11110) = 3
 Hamming Distance
• The hamming distance
– between the received codeword and the sent codeword is the
number of bits that are corrupted during transmission
• For example
– If the codeword 00000 is sent and 01101 is received
• The hamming distance between the two codewords is
– d(00000, 01101) = 3
» That is if the hamming distance between the sent and the
received codeword is not zero the codeword has been
corrupted during transmission
 Minimum Hamming Distance
• In a set of codewords
– The minimum Hamming distance is the smallest Hamming
distance between all possible pairs of codewords and is denoted
by dmin
• example
– For the set of codewords: 00000, 01011, 10101, 11110
• We have
d(00000, 01011) = 3 d(00000, 10101)= 3
d(00000, 11110) = 4 d(01011, 10101) = 4
d(01011, 11110) = 3 d(10101, 111101) = 3
• The minimum Hamming distance dmin = 3
• The minimum Hamming distance
– is a measure used in designing codes
 Minimum Hamming Distance
• The minimum distance
– of a code determines both its error-detecting ability and
error-correcting ability
• Suppose
– That a block code is used for error detection only
• Let c be the transmitted codeword and let r be the received
codeword
– If r is not a codeword, a detectable error has occurred
– If r is a codeword but not the transmitted codeword, an error
has occurred that cannot be detected
• If d(c, r) < dmin
– Then the received codeword cannot be an incorrect codeword
 Minimum Hamming Distance
• Error detection
– To guarantee the detection of up to s errors in all cases
• The minimum Hamming distance in a block code must be

d min  s  1
– The guaranteed error-detecting ability is

s  d min  1
 Minimum Hamming Distance
• Let us assume
– That the sent codeword x is at the center of a circle with radius s
• All received codewords that are created by 0 to s errors are points
inside the circle or on the perimeter
• All other valid codewords must be outside the circle
 Minimum Hamming Distance
• Error correction
– Retransmission of corrupted and lost packets is not useful for
real-time multimedia transmission because of unacceptable
delay
– We need to correct the error or reproduce the packet
– To guarantee correction up to t errors in all cases the minimum
Hamming distance in a block code must be

d min  2t  1
 Hamming Codes
• In a linear block code
– The exclusive OR of any two valid codewords creates another
valid codeword

• Originally designed with dmin = 3

– Which means that we can detect up to 2 errors and can correct a
single error
• We will discuss
– a Hamming code C(7, 4) with dmin = 3
 Hamming (7, 4)
• 7 bits total
– 4 data bits
– 3 parity bits
• Can find and correct 1 bit errors
or
• Can find but not correct 2 bit errors
 Hamming Codes

• Check bits in power-of-two positions

 Hamming Codes
• A linear block code
• Consider a (7,4) Hamming code
• Let d1 d2 d3 d4 be information symbols
• Let p1p2 p3 be check symbols
• The parity equations:
p1 = d1  d2  d4
p2 = d1  d3  d4
p3 = d2  d3  d4
 Hamming Codes
a b c abc

T T T T

T T F F

T F T F

T F F T

F T T F

F T F T

F F T T

F F F F
 Example
Byte 1011 0001
Two data blocks, 1011 and 0001.
Expand the first block to 7 bits: _ _ 1 _ 0 1 1.
Bit position 1 is 0
Bit position 2 is 1
bit position 4 is 0
Our 7 bit block is: 0 1 1 0 0 1 1

Repeat for right block giving 1 1 0 1 0 0 1

 Hamming Codes –
• Properties of the code
– If there is no error, all parity equations will be
satisfied
– Denote the outcomes of these equation checks as
c1, c2, c3
– If there is exactly one error, then c1, c2, c3 point to
the error
 Detecting/Correcting Error(s)
0110111
• Re-Check each parity bit
 Detecting/Correcting Error(s)
0110111
• Re-Check each parity bit

– Bits positions 1 and 4 are incorrect

• 1 + 4 = 5, so the error occurred in bit position 5

 Hamming Codes – by example

• Can write the equations as follows (easy to

remember)
p1 p2 d1 p3 d2 d3 d4
- - 1 - 1 0 1
- - 0 - 1 1 1
1 2 3 4 5 6 7

• This encodes
– a 4-bit information word into a 7-bit codeword
 Hamming Codes – by example

• Can write the equations as follows (easy to

remember)
p1 p2 i1 p3 i2 i3 i4
1 0 1 0 1 0 1
0 0 0 1 1 1 1
1 2 3 4 5 6 7
This encodes a 4-bit information word into a 7-bit
codeword
 Parity Checking
Single Bit Parity:
Detect single bit errors

• Even (Odd) Parity scheme

– Sender includes one additional bit and chooses its
value such that the total number of 1s in the d+1 bits
(original info +parity bit) is even (odd)
 Parity Checking
Two Dimensional Bit Parity:
Detect and correct single bit errors
• Two dimensional

0 0
 example
• Suppose the information content of a packet is the bit pattern 1110
1011 1001 1101and an even parity scheme is being used.
– What would the value of the field containing the parity bits be for the
case of a two-dimensional parity scheme?
 Solution
11101
10111
10010
11011
00011
 Cyclic Redundancy Check
• Consider the d-bit piece of data, D, that the sending node wants to
send to the receiver
• The sender and receiver
– 1st agree on an r+1 bit pattern (generator), G
• goal:
– choose r additional bits, R, and append them to D such that
– d + r bit pattern is exactly divisible by G using modulo 2 arithmetic

– Error checking
• receiver knows G, divides d + r received bits by G.
• If remainder is non-zero : error is detected
– can detect all burst errors less than r+1 bits
• widely used in practice (Ethernet, 802.11 WiFi, ATM)
Cyclic Redundancy Check
CRC
 Modulo-2 Arithmetic
• Modulo 2 arithmetic is
– performed digit by digit on binary numbers
• Each digit is considered independently from its neighbors.
– numbers are not carried or borrowed
• Addition/Subtraction
– Modulo 2 addition/subtraction is
• performed using an exclusive OR (XOR) operation on the
corresponding binary digits of each operand
• Addition Subtraction
 Modulo-2 Arithmetic
• Division
– Modulo 2 division can be performed in a manner similar to
arithmetic long division
• Subtract the denominator (the bottom number) from the leading
parts of the numerator (the top number).
 Example
• Consider the 4-bit generator, G (1001) and
suppose that D has the value 101110. What is
the value of R?
 A Sample CRC Calculation

• Transmitted message (codeword) = 101110011

 Problem
• Consider the 4-bit generator, G (1001) and suppose
that D has the value 10101010. What is the value of
R?
 Solution
10111101
• 1001 10101010000
1001
111
000
1110
1001
1111
1001
1100
1001
1010
1001
110
000
1100
1001
101
The 11 bits transmitted are 10101010101
 Problem
• Consider the 5-bit generator, G = 10011, and suppose
that D has the value 1010101010. What is the value of R
 Solution
101101 1100
• 10011 1010101010 0000
10011
1100
0000
11001
10011
10100
10011
1111
0000
11110
……..
0100
 Problem
• Given a standard generator polynomial of 10011 and a
message sequence of 1101011111
– determine the remainder that forms that CRC
– By the CRC method of error detection, describe the information
that is transmitted to the receiver and explain how this
information is used to detect bit errors that may have occurred in
transmission
– Suppose the seventh bit from the left of the transmitted message
is inverted due to noise on the transmission link. What is the
result of the receiver’s CRC calculation? How does the receiver
know an error has occurred?