Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Information Theory
Degree in Data Science and Engineering
Lesson 6: Channel coding

Jordi Quer, Josep Vidal

Mathematics Department, Signal Theory and Communications Department

{jordi.quer, josep.vidal}@upc.edu

2019/20 - Q1

1/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Why channel coding?

Let us return to the example of the HDD and compare repetition coding and
Shannon channel coding.
For the BSC with p = 0.1, the channel capacity is

C = 1 − H(0.1) = 0.469 bits/tr

This implies an average of 1/R = 2.132 transmissions/bit for zero error

transmission with Shannon’s channel codes.
Compare it to the 68 transmissions/bit required for the repetition code with a
bit error probability of 10−15 .

In spite of its great performance, Shannon’s coding is impractical because...

Codewords are infinite length, and
Even if making them shorter (at the expenses of some residual error),
decoding implies exhaustive evaluation of joint typicality with a large
number of codewords.

2/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Why channel coding?

We need to design error correction codes that:

Achieve low probability of error without much rate loss, and
Have certain internal structure allowing low decoding complexity.

Two typical practical ways to combat errors are:

ARQ (Automatic Repeat Request): Detect errors and request
retransmission, requires a feedback channel and some delay is incurred.
FEC (Forward Error Correction): Correct errors at the receiver side by
introducing redundancy at the transmitter side. It is quicker, does neither
require feedback nor buffering at the transmitter, but entails additional
complexity at the receiver.

They complement each other in communication systems, we will focus on FEC.

3/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Why channel coding?

Channel codes are an intrinsic part of all communication systems...

Deep space communications: the Voyager 1 spacecraft transmitted
images of Jupiter and Saturn using convolutional codes concatenated
with Golay codes. Voyager 2 used Reed-Solomon codes, and later
missions use LDPC codes.
Cellular communication systems: 3G and 4G use convolutional
concatenated turbo codes, and 5G use LDPC codes and Polar codes.
Satellite broadcasting: DVB use LDPC and BCH codes,
CD, DVD and HDD: used interleaved Reed-Solomon codes. LDPC codes
in modern HDD.
Error-correcting memory controllers. In deep space missions, RAM may
be affected by cosmic rays. Hamming codes with triple modular
redundancy are used.
QRcodes (2D bar codes) Use Reed-Solomon codes and 4 different levels
of protection.

4/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Channel codes

A (n, k) channel code C is a mapping that assigns the k-length sequence of

symbols belonging to an alphabet X , to a codeword (a vector of n symbols)
belonging to a possibly different alphabet (we adopt the same though).

C : Xk → Xn
c = C(xk )

The codeword generated at the transmitter is sometimes received in error

r = c + e after having passed through a noisy channel.
Reconstructing c from the observed r requires introducing redundancy, by
taking n > k.

The rate of the code is defined as Rc = k/n.

5/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Fields

Algebraic encoding and decoding entails arithmetic operations defined in a

field, F, consisting of a set X together with two operations:

Addition
Assuming a, b, c ∈ X ...
The set is closed under addition: a + b ∈ F
Associativity: a + (b + c) = (a + b) + c
Commutativity: a + b = b + a
Additive identity element: a + 0 = a
Additive inverse element: −a ∈ F such that a + (−a) = 0

6/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Fields

Multiplication
Assuming a, b, c ∈ X ...
The set is closed under multiplication: ab ∈ X
Associativity: a(bc) = (ab)c
Commutativity: ab = ba
Distributive over the addition: a(b + c) = ab + ac
Multiplicative identity element: 1a = a
Inverse element: ∀a 6= 0, ∃ a−1 such that a−1 a = 1

A finite field with q = |X | elements is called a Galois field, GF (q), which can
only be defined if q is a prime number or an integer power of a prime. Very
much used fields in coding are GF (2n ), in particular n = 8.
Only X = {0, 1} will be considered in the sequel.

7/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Linear block codes

There are q n vectors in the vector space Vn of dimension n in the field GF (q).
A subset of this space is called a vector subspace S, if two conditions are met:
The all-zeros vector is in S, and
The linear combination of two vectors is in S, i.e. if c1 , c2 ∈ S and
α, β ∈ X , then αc1 + βc2 ∈ S

Example. If n = 4 and q = 2, there are q n = 16 vectors in the vector space V4

of dimension 4. Two linearly independent codewords, e.g.

c1 = 0101 c2 = 0110

form a basis of the 2-dimensional subspace S. 

The scalar product of two vectors is denoted as ci cTj and defined as the sum of
the products of its elements. Two vectors are orthogonal if its scalar product is
zero.

8/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Linear block codes

A collection of vectors orthogonal to S is a base of the null space (or kernel) Sc

of S. If the dimension of S is k, the dimension of Sc is n − k.

Example. The codewords

c3 = 1000 c4 = 1111

are orthogonal to c1 , c2 and generate Sc . 

The weight of a codeword, w(cj ), is the number of non-zero elements.

The Hamming distance of two codewords ci , cj , d(ci , cj ), is the number of

elements in which they differ. Four relevant properties are:
1 Triangular inequality: d(ci , cj ) ≤ d(ci , ck ) + d(ck , cj )
2 Non-negativity: 0 ≤ d(ci , cj ) ≤ n, with d(ci , cj ) = 0 if ci = cj
3 Symmetry: d(ci , cj ) = d(cj , ci )
4 d(ci , cj ) = w(ci − cj ), from the definition of the Hamming distance.

9/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Linear block codes

The minimum distance of a code C is

dmin = min{w(ci − cj ) : ci , cj ∈ C, ci 6= cj }
= min{w(ck ) : ck ∈ C, ck 6= 0}
, wmin
since the linearity of the code implies that a sum of codewords is a codeword.

The key idea behind block codes is: take a number of vectors in Vn (the
codebook) that form a subspace. If due to channel errors any element of the
codeword is changed, we can still recover the codeword as long as it is not
identified as another codeword.

10/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Linear block codes

The codebook (vectors ci ) contain a

fraction of the total set of vectors in Vn
(all black dots), and each codeword can
be considered as the center of a sphere
of a certain radius t.

Two contradicting goals arise in the

design:
1 High efficiency (large Rc , close
to channel capacity), by using as
many codewords as possible in
Vn ,
2 High error protection by placing
codewords as far as possible from
each other, so that corrupted
codewords can still be correctly
decoded with high probability.

11/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

The generator matrix

As an example, let us take a (n, k) = (127, 92) code. From the 2127 possible
codewords, only 292 ' 5 × 1027 are selected, but using them still implies a huge
lookup table! Fortunatelly we can generate codewords on the fly from the
generator matrix.

Define a basis of a k-dimensional space, {g1 , g2 , . . . , gk }, and arrange them

into a matrix:
   
g1 g11 g12 . . . g1n
 g2   g21 g22 . . . g2n 
G= . = .
   
.. .. 
 ..   .. . . 
gk gk1 gk2 . . . gkn

Given k input symbols xm , the associated codeword is generated from the

vector-matrix product in F:
cm = xm G
That is, codewords are generated as linear combinations of the rows of G.

12/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

The generator matrix

By linearly combining rows and permuting columns (Gauss-Jordan elimination),

matrix G can be reduced to its systematic form:
 
p11 p12 . . . p1(n−k) 1 0 0 ... 0
 p21 p22 . . . p2(n−k) 0 1 0 ... 0 
G = [P|Ik ] =  .
 
.. ..
 ..

. . 
pk1 pk2 . . . pk(n−k) 0 0 0 ... 1

As a result, the codeword is split between parity check symbols and systematic
symbols:
cm = [cp |xm ]

13/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

The parity check matrix

Associated with the (n, k) code, we can define a (n, n − k) linear dual code,
whose generator matrix is H. The n − k rows of H form a base of the null
space of the rows of G. Any codeword belonging to the code is orthogonal to
any codeword generated with the dual code.
Therefore:
GHT = 0
Solving the linear equation, it follows that

H = [In−k | − PT ]

Hence, for any codeword cm HT = 0, and hence we have an algebraic tool that
allows detecting errors.

14/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

The parity check matrix

Example. A (5, 2) linear block code has a generator matrix:

 
1 0 1 1 0
G=
0 1 1 0 1

The minimum distance is dmin = 3 and its rate is R = 2/5. The parity check
matrix is  
1 0 0 1 0
H= 0 1 0 0 1 
0 0 1 1 1
It can be verified that GHT = 0. 

Any codeword is orthogonal to HT , in particular the minimum distance

codeword: cdmin HT = 0. Since no other codeword has lower weight, the
number of linearly independent rows of HT must be at least dmin − 1. Since
the columns of HT generate the null space, rank(H) = n − k. Hence,

dmin ≤ n − k + 1

15/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Optimum decoding
Assume that a transmitted codeword c is received in error r = c + e after
having passed through a noisy channel. The decoder operates on r and returns
ĉ = g(r). An error event E occurs with probability Pr(E|r) = Pr(ĉ 6= c|r), and
the total error probability is
X
Pr(E) = Pr(E|r) Pr(r).
r

Minimizing the error is equivalent to

minimize Pr(E|r)
c

or also
maximize Pr(ĉ = c|r) ⇔ maximize Pr(r|ĉ = c) Pr(c)
c c

that is, ĉ is the most likely codeword given r (maximum a posteriori decoder).
If the channel is memoryless, all errors in the elements of the received codeword
are independent so we can write
n
Y
Pr(r|c) = Pr(ri |ci )
i=1
16/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Optimum decoding
Let us assume all codewords are equally likely. Since the logarithm is a
monotonous increasing function we can use it in the optimization:
n
X
log Pr(r|c) = log Pr(ri |ci )
i=1

= d(r, c) log p + (n − d(r, c)) log(1 − p)

p
= d(r, c) log + n log(1 − p)
1−p
where d(r, c) is the Hamming distance between the received vector and a
p
possible decoded vector. Since log 1−p is negative for p < 21 the minimum
error criterion finally results in

ĉ = argmin d(r, c)
c

Check the figure in slide 11: any black dot corresponding to a possible received
vector will be decoded into the closest codeword. Therefore, optimum decoding
entails comparison of r with all possible 2k codewords! For long codes with
large codebooks the complexity may be prohibitive.
17/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

Fortunatelly, the algebraic structure of linear block codes allow simple decoding
using the parity check matrix H.

We may express r = c + e, where e is an arbitrary error vector in X n . Applying

the parity check matrix:

rHT = (c + e)HT = cHT + eHT = eHT = s

s is a (n − k) dimensional vector called the syndrome. Note that there are

2n − 1 possible error vectors e and only 2n−k syndromes, so different error
vectors may result in the same syndrome.

Error correction is possible if using the standard array D.

18/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

Construction of the standard array D: include in the first row all possible
codewords ci (they form a vector space). In the second row, put all vectors
resulting from the sum of a coset leader e2 (a vector that is not a codeword)
and the vectors in the first row. For the third row, take as coset leader e3 a
vector that did not appear in the previous rows and form the third row by
summing it to the vectors of the first row. Follow in this way until 2n−k rows
are completed:
 
c1 c2 c3 ... c 2k
 e2 c2 + e2 c3 + e2 ... c 2k + e 2 
 
 e3 c 2 + e3 c3 + e3 . . . c 2k + e 3

 
 .. .. .. .. 
 . . . . 
e2n−k c2 + e2n−k c3 + e2n−k ... c2k + e2n−k

Row j (called the j-th coset) contains all possible received vectors resulting
from a particular error pattern ej , and forms an affine space.

Built in this way, D contains all possible vectors of Vn .

19/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

Using the standard array D...

 
c1 c2 c3 ... c 2k
 e2 c2 + e2 c3 + e2 ... c 2k + e 2 
 
 e3 c2 + e3 c3 + e3 ... c 2k + e 3 
 
 .. .. .. .. 
 . . . . 
e2n−k c2 + e2n−k c3 + e2n−k ... c2k + e2n−k

...how to correct errors?

i. Compute the syndrome on the received vector s = rHT
ii. Identify the coset leader with the same syndrome, em
iii. Determine the transmitted codeword as ĉ = r − em

20/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

Theorem (Error detection)

A linear code can detect all vector errors of weight dmin − 1 or less.

Proof. If the minimum distance of the code is dmin , any two codewords differ
in at least dmin elements. Therefore, no error vector of weight dmin − 1 or less
can convert one codeword into another. In that case, the received vector will
generate a non-zero syndrome. 

The code cannot detect all error vectors of weight dmin . In practice, the code
can detect many error vectors of weight dmin and more. The code can detect
2n − 2k error vector.

21/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

Theorem (6.0)
All the 2k vectors of a coset have the same syndrome. The syndromes for
different cosets are different.

Proof. By construction of D (see slide 19) every vector in a coset has the same
syndrome
(el + ci )HT = el HT + ci HT = el HT
which is the syndrome of the coset leader.
For the second part of the theorem, let ej and el be the coset leaders of the
jth and lth cosets, where j < l. Assume that the syndromes of these coset
leaders are equal
ej HT = el HT
(ej + el )HT = 0.
This implies that ej + el is a codeword in C, say ci . Thus, ej + el = ci , and
el = ej + ci . This implies that el is in the jth coset, which contradicts the
construction of D. Therefore, no two cosets have the same syndrome. 
22/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

Theorem (6.1)
Every (n, k) linear block code is capable of correcting up to 2n−k error patterns.

Proof. Let cj be the transmitted codeword. Only if the error vector induced by
the channel e is one of the 2n−k coset leaders, the received vector is in the j-th
column of D. Otherwise, cj + e will be associated to some syndrome, and
hence to a different coset leader el plus a non-zero codeword ci (remember all
possible vectors in {0, 1}n appear in the array). Hence,

r = cj + e = cj + el + ci = el + (cj + ci ) = el + cs

and the decoded symbol is ĉ = cs 6= cj . An error occurs. 

23/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding
Example (continued). For the (5, 2) linear block code, with dmin = 3
 
1 0 1 1 0
G= 0 1 1 0 1

the standard array and the syndromes are (codewords in red):

Standard array Syndromes

00000 01101 10110 11011 000
00001 01100 10111 11010 011
00010 01111 10100 11001 101
00100 01001 10010 11111 001
01000 00101 11110 10011 010
10000 11101 00110 01011 100
11000 10101 01110 00011 110
10001 11100 00111 01010 111

Here, coset leaders are all vectors of weight 1 plus two vectors of weight 2
selected among those having different syndrome. Note that if dmin = 1, no
vector of weight 1 could have been used as a coset leader. Why? If a coset
leader lead us to another codeword, errors cannot be detected.
24/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding
Syndrome decoding is optimal if coset leaders are properly chosen.

Theorem (6.2)
If the coset leaders are the minimum weight vectors, syndrome decoding
minimizes the probability of error.

Proof. Let r be the received vector. If located in the i-th column of D and in
the l-th coset, then r is decoded into ci . Because r = el + ci , the distance
between r and ci is

d(r, ci ) = w(r + ci ) = w(el + ci + ci ) = w(el )

Consider now the distance to another codeword cj

d(r, cj ) = w(r + cj ) = w(el + ci + cj ) = w(el + cs ) ≥ w(el ) = d(r, ci )

where the inequality appears if coset leaders are chosen of minimum weight.
Finally, check the optimum receiver in slide 17. 
Note that minimum weight errors are the most likely to occur in a BSC.

25/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding

We are interested in codes with large dmin .

Theorem (6.3)
For a (n, k) linear block code C with minimum distance dmin , all the vectors of
weight t = b(dmin − 1)/2c or less can be used as coset leaders.

Proof. Let x, y be two vectors of weight t = b(dmin − 1)/2c or less. The

weight of the sum is

d(x, y) = w(x + y) ≤ w(x) + w(y) ≤ 2t < dmin

If x and y are in the same coset, then x = ci + ek and y = cj + ek and hence

the sum must belong to C by linearity of the code. But this is impossible
because the weight is less than dmin . Then, no two vectors of weight less than
or equal to t can be in the same coset, and hence all coset leaders can be
vectors of weight t or less. 

26/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Syndrome decoding
A block code of minimum distance dmin can correct all errors of weight equal
or lower than t, and cannot correct all errors of weight t + 1.

Theorem (6.4)
For an (n, k) linear block code C with minimum distance dmin , if all vectors of
weight t or less are used as coset leaders, there is at least one vector of weight
t + 1 that cannot be used as a coset leader.

Proof. Let v ∈ C such that w(v) = dmin . Let x, y be two vectors satisfying
1 x+y =v
2 x, y do not have non-zero components in common places.
By 1) and linearity of the code, x, y must be in the same coset, and by 2)

w(x) + w(y) = w(v) = dmin

Since 2t + 1 ≤ dmin ≤ 2t + 2, by theorem (6.3) if w(y) = t + 1 then w(x) must

be t or t + 1. Since both are in the same coset, if x is chosen as coset leader
then y cannot be chosen. 

27/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Error detection and error correction capabilites

Example (continued). Note that dmin = 3 and t = 1. Let c = 01101 be the

transmitted codeword and the error induced by the channel is e = 10100 (not a
coset leader). The syndrome is (c + e)HT = 101, and the decoded vector is
r + e3 = 11011 6= c. The error is detected but not corrected. 

Looking into the structure of D and the theorems above, a block code can
Not detect 2k − 1 error vectors: those codewords altered into another
codeword.
Detect 2n − 2k error vectors: those generating a non-zero syndrome
vector; for large n, 2k − 1  2n − 2k .
Correct 2n−k error vectors: those error vectors equal to a coset leader;
they are a small fraction of those detected for large n.

28/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Block error probability

The probability of error in a codeword is given by the sum of the probability of

undetected errors Pud (E) plus de probability of detect but not being able to
correct errors Puc (E):

P (E) = Pud (E) + Puc (E)

= Pr(E|no detect) Pr(no detect) + Pr(E|detect) Pr(detect)

Note that this is an upper bound of the bit error probability. Let us evaluate
the four terms:
Pr(E|no detect) = 1, if the error is not detected, it cannot be corrected!
Pr(no detect) is given in next slide as Pu (E). Depends on the code and
on the channel transition probabilities.
Pr(E|detect) is given in the following slides as Pd (E).
Pr(detect) = 1 − Pr(no detect)

29/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Probability of undetected errors

Define the weight enumerator function (WEF) Aj (i) of the code Cj as the
number of codewords having weight i. The probability of undetected errors for
the binary symmetric channel is
n
X
Pu (E|Cj ) = Aj (i)pi (1 − p)n−i
i=1

The WEF is known for many codes.

We can compute an upper bound for the undetected errors probability on the
whole set Γ of possible (n, k) linear block codes. Since all are constructed
using the generator matrix G, there are 2k(n−k) possible parity matrices P.
Let us compute a bound on this error for the ensemble of linear block codes. If
the code is selected randomly, P (Cj ) = 2−k(n−k) and then the average
probability of error is:
|Γ| n |Γ|
X X X
P̄u (E) = P (Cj )Pu (E|Cj ) = 2−k(n−k) pi (1 − p)n−i Aj (i)
j=1 j=1 j=1

30/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Probability of undetected errors

A nonzero vector is contained
in 2(k−1)(n−k) codes or in none of the codes

(why?). Because there are ni vectors of weight i, we have
|Γ|
!
X n (k−1)(n−k)
Aj (i) ≤ 2
j=1
i

and hence
n
!
−(n−k)
X n i
P̄u (E) ≤ 2 p (1 − p)n−i = 2−(n−k) (1 − (1 − p)n ) ≤ 2−(n−k)
i=1
i

That is, there exists (n, k) linear block codes Ck whose Pu (E|Ck ) is decreasing
faster than exponentially with the number of parity check bits (n − k).
Only a few codes have been discovered to exactly reach this bound.

31/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Probability of detected and non corrected errors

A linear block code C can correct all errors of weight t = b(dmin − 1)/2c and
some of weight t + 1. The probability of having detected an error but not being
able to correct it is (for the binary symmetric channel):
n
!
X n i
Pd (E) ≤ p (1 − p)n−i
i=t+1
i

Inequality comes from coset leaders of weight larger than t.

Assume a sphere of radius t around each possible codeword. The number of

vectors of {0, 1}n in the sphere is
! ! ! t
!
n n n X n
1+ + + ··· + =
1 2 t i=0
i

Since we have 2k possible spheres and a total of 2n vectors, the following

inequality must hold !
t
k
X n
2 ≤ 2n
i=0
i
32/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Probability of detected and non corrected errors

In a perfect code this expression holds with equality, and no error vector of
weight t + 1 can be corrected. The spheres around each codeword fully cover
all vectors in the n-dimensional space and do not overlap. This result in
minimum probability of error among all codes of the same (n, k) values.

Part of the Hamming

space perfectly filled by
spheres of radius t
centered on the
codewords of a perfect
code.

33/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Probability of detected and non corrected errors

Exact expressions for the probability of error can also be obtained for
quasi-perfect codes, for which all spheres of radius t are disjoint and the
spheres of radius t + 1 overlap (as a consequence of the weight of some coset
leaders is t + 1). No more than t + 1 errors can be corrected.

A quasi-perfect code can be derived from a perfect code by adding a parity

check digit to the end of each codeword, so if a (n, k) code is a perfect binary
linear code of minimum distance d, then the code thus constructed is a
(n + 1, k) nearly perfect binary linear code of minimum distance d + 1.

34/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Probability of detected and non corrected errors

For quasi perfect codes, the total number of vectors outside the spheres is
t
!
n k
X n
Nt+1 = 2 − 2
i=0
i

that we can equally divide among the 2k codewords so that each sphere is
enlarged by adding !
t
n−k
X n
βt+1 = 2 −
i=0
i
vectors
of distance t + 1 from the transmitted codeword. Consequently, from
n
all t+1 error vectors of distance t + 1 we can correct βt+1 . Thus the error
probability is given by
n
! " ! #
X n i n−i n
Pd (E) = p (1 − p) + − βt+1 pt+1 (1 − p)n−t−1
i=t+2
i t + 1

35/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Perfect codes

The only perfect codes known so far are:

Hamming codes
Golay (23, 12) codes
Trivial (n, 2) codes with n odd and dmin = 2

Why are they so rare? It is hard to pack spheres in high dimensional spaces.

36/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Bounds on dmin

Given that t determines the error rate performance of the code, we need to
compute a value for dmin given a code. Unfortunately there is no simple way of
computing it, and many bounds have been derived. The bounds shown below
provide necessary conditions for Rc given a desired distance dmin .
Singleton bound
Use the bound dmin ≤ n − k + 1. Since dmin − 1 ' 2t, a high power
correction is achieved if n − k is large, at least, n − k ≥ 2t. Codes for
which this bound is met with equality are called Maximum Distance
Separable codes (only repetition codes and Reed-Solomon codes).
dmin 1
δmin = ≤ 1 − Rc +
n n
for large n
δmin ≤ 1 − Rc

37/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Bounds on dmin

Hamming bound
Depart from the inequality
t
!
(n−k)
X n
2 ≥
i=0
i

and take logarithm on both sides

t
!    
1 n 1 t t δmin
' log 2nH ( n ) = H
X
1 − Rc ≥ log 'H
n i=0
i n n 2

where approximations are for large n, using Newton’s binomial and

Stirling’s approximation. The bound is tight for perfect codes.

38/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Bounds on dmin

Asymptotic bounds for Rc versus δmin : high error correction

capabilities imply reduced code rate.
39/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Outstanding block codes

Some well known and widely used block codes are:

Hamming codes
Cyclic codes
Golay (23, 12) codes
Reed-Solomon codes
BCH codes
Low Density Parity Check (LDPC) codes

40/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Hamming codes

In 1950, only two years after Shannon’s

paper, the mathematician Richard
Hamming , office mate of Shannon at
Bell Telephone Labs, asserted that by
proper encoding of information, errors
induced by a noisy channel can be
reduced to any desired level without
sacrificing the rate of information.
Coding theory started with him.
Hamming codes have been used over
the years for their high rate and low
complexity of decoding.

Richard Hamming

41/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Hamming codes

These are codes of dmin = 3. For any m = n − k ≥ 3 there is a code such that
Codeword length: n = 2m − 1
Information symbols: k = 2m − m − 1
Error correcting capability: t = 1
m
Rate: Rc = 1 − 2m −1
Parity check matrix H: its 2m − 1 columns contain all integers of m
binary digits except zero.
Weight enumerator function, is the set of coefficients of the polynomial
1 h i
A(z) = (1 + z)n + n(1 − z)(1 − z 2 )(n−1)/2
n+1

Probabilities of error and of undetected errors can be readily obtained.

42/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Hamming codes

Probability of undetected errors for several (n, k) Hamming codes versus

transition probability of a BSC

43/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Hamming codes

Block probability of detected and not corrected errors, for several (n, k)
Hamming codes versus transition probability of a BSC. Uncoded block
probability in dashed line.
44/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Benefits of channel coding

A → B Given a BSC with a certain p, we can reduce error by increasing
redundancy (reducing bit transmission rate)
A → C Given a probability of error for the BSC, we can tolerate worse
channel p values by increasing redundancy.

45/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Golay (23, 12) code

Derived by the French mathematician Marcel Golay in 1949, it is a cyclic code:

if c is a codeword, any cyclic shift of c is also a codeword. dmin = 7.
Codeword length: 23
Information symbols: 12
Error correcting capability: t = 3
Rate: Rc = 12/23
Weight enumerator function, known.
Usually a parity check bit is added, and then n = 24 and dmin = 8, in this case
it can detect error vectors of weight up to 7 and correct error vectors of weight
3 or less.

46/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Golay (23, 12) code

Probability of detected and not corrected errors for the Golay code versus
transition probability of a BSC
47/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Reed-Solomon codes

Derived by the US mathematicians and engineers Irving Reed and Gustave

Solomon in 1960, probably the most widely used codes. They are cyclic codes,
and have the largest possible dmin among all linear block codes of a given
(n, k) and are defined in GF (2k ).

Alphabet size: q = 2k symbols

Codeword length: N = q − 1 = 2k − 1
Information symbols: K = 1, 2, . . . , N − 1
Minimum distance: Dmin = N − K + 1
Dmin −1
Error correcting capability: t = 2
Rate: Rc = K/N
Weight enumerator function, known.

48/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Reed-Solomon codes

Example. To explore the advantages of nonbinary codes consider these two

codes:
Binary (7,3) code. 27 possible codewords, only 23 are used, i.e. 1
16
of the
vectors.
Ternary (7,3) code (defined in GF (q), q = 3). There are
27q = 221 = 2, 097, 152 input binary words, of which 23q = 29 = 512 are
1
used, i.e. only 4096 of the vectors.
Since a small fraction of vectors are used as codewords, large values of Dmin
can be obtained.

49/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

LDPC codes

Discovered by Robert Gallager in this PhD

thesis of 1960. They have good minimum
distance1 and are able to approach Shannon
capacity bound.
The parity check matrix has a low density of
ones. In regular LDPC codes, H contains wc
n
ones in each column and wr = wc n−k ones in
each row, where wc  min(n, n − k),
wr  min(n, n − k). The code rate is

R = k/n = 1 − wc /wr .

LDPC codes perform near the Shannon limit for

large block lengths. For example, simulations
show a bit error rate of 10−6 for a rate of 90%
Robert Gallager of channel capacity, with block length of 107 .
1
X.Y. Hu, M.P.C. Fossorier, E. Eleftheriou, C. Di, T. Richardson and R. Urbanke, “On the Computation of the
Minimum Distance of Low-Density Parity-Check Codes” IEEE Intl. Conference on Communications (ICC), Paris,
France, 20-24 June 2004.
50/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

LDPC codes

A low-density parity-check matrix with n = 20000 columns of weight wc = 3

and n − k = 10000 rows of weight wr = 6.

Examples taken from D. Mackay, Information Theory, Inference, and Learning Algorithms, Cambridge Press, 2003 51/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

LDPC codes

Systematic and parity bits for the encoded source.

The parity bits depend on nearly half the input bits.

52/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

LDPC codes

Encoding. It is known that only irregular LDPC codes are able to achieve
close to capacity rates at low probability of error1 . These codes have a
variable number of ones in each row and column of H.
Decoding. Optimum decoding is an NP-hard problem. Many suboptimum
iterative algorithms exist that take advantage of the sparse structure of
matrix H.
The bit-flipping family of algorithms operate basically as follows:
1: compute the syndrome s = rHT
2: iter=0
3: if (s == 0 or iter > max num iter) then stop
4: otherwise, take the nonzero components of s and flip those
components of r that appear in the largest number of
unsatisfied parity check equations
5: iter=iter+1
6: go to 1
1
S.Y. Chung, G.D. Forney Jr., T. Richardson, R. Urbanke, “On the Design of Low-Density Parity-Check Codes
within 0.0045 dB of the Shannon Limit,” IEEE Commun. Lett., vol. 5, pp. 58–60, 2001

53/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Iterative decoding of an
LDPC code for a
transmission received
over a BSC channel
transition probability
q = 0.075. The
decoder halts after the
13th iteration when the
best guess violates no
parity checks. The final
decoding is error free.

54/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

LDPC codes

Error probability of the low-density parity-check code for the BSC with
q = 0.075, compared with algebraic codes. Squares: repetition codes and
Hamming (7,4) code. Crosses: BCH codes.

55/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

LDPC codes

Codes in wireless communication systems. In 5G, LDPC have been selected for
user data and polar codes for control data. In 4G, turbo codes were selected.
The advantage of LDPC is that the clock rate of the hardware does not have
to be increased with the increase of the code rate.
56/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Interleaving
Most of codes have been designed for
statistically independent errors (like in
BSC). In many channels however,
errors appear in a bursty way, like in
cellular communication system, or when
reading an optical support with
damaged surface.
A burst of errors of length b is a
sequence of b-bit errors where the first
and last are 1. An (n, k) linear block
code can correct bursts of length
b ≤ b 12 (n − k)c.

A way to deal with bursty errors is to

interleave the data, so that the channel
is turned into a channel with
independent errors, and use a
conventional error correcting code:

57/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Interleaving

The interleaver reorders the encoded data and transmits it over the channel. At
the receiver, the deinterleaver reorders the data and passes it to the decoder.
As a result, errors are spread out in time and appear independent. A block
interleaver of degree m, is formed by m rows and n columns...

58/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Interleaving

At the deinterleaver, symbols are stored in the matrix and read row-wise. As a
result, a burst of l = mb errors is broken into m bursts of length b.
Thus, an (n, k) code that can handle bursts of errors of length b can be
combined with an interleaver to create an interleaved (mn, mk) block code
that can handle bursts of length mb.

Example. In Compact Disk, a Reed-Solomon Rc = 24/28 is concatenated with

an interleaver and another Reed-Solomon code of rate Rc = 28/32 (Cross
Interleave Reed Solomon Code - CIRC). The overall code rate of 3/4, and
dmin = 5. Codes are designed in the Galois field GF (28 ).
The maximum fully correctable burst length of CIRC is up to 4000 bits (2.5mm
of disk surface). After CIRC, the residual error rate is less than 10−9 , which is
enough for CD.

59/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Interleaving

For CD-ROM, this rate is too high so the standard of CD-ROM had added the
EDC (Error Detection Code) and ECC (Error Correction Code). This method
can make the error rate is less than 10−13 which satisfy the requirement of
CD-ROM.

60/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Concatenated codes

Code concatenation, together with interleaving, is used to increase the

codeword length and improve error performance. The minumum distance is the
product of minimum distances.

Although not optimal, decoding is usually done sequentially for each code.

61/62
Some algebra Linear block codes Decoding Probability of error Outstanding codes Interleaving Concatenation

Wrap up

Coding is an intrinsic part of any information communication or storage

system.
The number of codes proposed since 1950 is huge, the use of one or
another depends on the performance sought (in terms of rate and
probability of error), the conditions of the channel (error induced, bursty
or not bursty) and the complexity allowed for decoding.
Design of codes and efficient decoding algorithms are still active areas of
research.

Recommended bibliography:
S. Lin, D. J, Costello, Error Control Coding, Second edition, Prentice Hall, 2004

62/62