Haykin_ch10_pp3.

fm Page 581 Friday, January 4, 2013 5:03 PM

10.3 Discrete Memoryless Channels 581

± E φ1(t) + w(t) T Uniform

Input
∫0 dt multilevel
quantizer
Output

φ1(t) = 2 cos(2πf t)
c
T
(a)

Output
b1
b1
b2
b2 Symbol 1 transmitted
b3 E b3

b4 b4
Input
b5
b5
b6
b6
b7 Symbol 2 transmitted
b8 – E b7

b8
(b) (c)

Figure 10.3 Binary input, Q-ary output discrete memoryless channel. (a) Receiver for binary PSK.
(b) Transfer characteristic of a multilevel quantizer. (c) Channel transition probability diagram. Parts
(b) and (c) are illustrated for eight levels of quantization.

can be transmitted per channel use in a reliable manner. The channel coding theorem
states:
If a discrete memoryless channel has capacity C and a source generates
information at a rate less than C, then there exists a coding technique such that
the output of the source may be transmitted over the channel with an arbitrarily
low probability of symbol error.
For the special case of a binary symmetric channel, the theorem teaches us that if the code
rate r is less than the channel capacity C, then it is possible to find a code that achieves
error-free transmission over the channel. Conversely, it is not possible to find such a code
if the code rate r is greater than the channel capacity C. Thus, the channel coding theorem
specifies the channel capacity C as a fundamental limit on the rate at which the
transmission of reliable (error-free) messages can take place over a discrete memoryless
channel. The issue that matters here is not the SNR, so long as it is large enough, but how
the channel input is encoded.
The most unsatisfactory feature of the channel coding theorem, however, is its
nonconstructive nature. The theorem asserts the existence of good codes but does not tell
us how to find them. By good codes we mean families of channel codes that are capable of
providing reliable transmission of information (i.e., at arbitrarily small probability of
symbol error) over a noisy channel of interest at bit rates up to a maximum value less than
the capacity of that channel. The error-control coding techniques described in this chapter
provide different methods of designing good codes.
Haykin_ch10_pp3.fm Page 582 Friday, January 4, 2013 5:03 PM

582 Chapter 10 Error-Control Coding

Notation
Many of the codes described in this chapter are binary codes, for which the alphabet
consists only of binary symbols 0 and 1. In such a code, the encoding and decoding
functions involve the binary arithmetic operations of modulo-2 addition and multiplication
performed on codewords in the code.
Throughout this chapter, we use the ordinary plus sign (+) to denote modulo-2 addition.
The use of this terminology will not lead to confusion because the whole chapter relies on
binary arithmetic. In so doing, we avoid use of the special symbol as we did in previous
parts of the book. Thus, according to the notation used in this chapter, the rules for
modulo-2 addition are as follows:
0+0 = 0
1+0 = 1
0+1 = 1
1+1 = 0
Because 1 + 1 = 0, it follows that 1 = –1. Hence, in binary arithmetic, subtraction is the
same as addition. The rules for modulo-2 multiplication are as follows:
00 = 0
10 = 0
01 = 0
11 = 1
Division is trivial, in that we have
11 = 1
01 = 0
and division by 0 is not permitted. Modulo-2 addition is the EXCLUSIVE-OR operation
in logic and modulo-2 multiplication is the AND operation.

10.4 Linear Block Codes

By definition:
A code is said to be linear if any two codewords in the code can be added in
modulo-2 arithmetic to produce a third codeword in the code.
Consider, then, an (n,k) linear block code, in which k bits of the n code bits are always
identical to the message sequence to be transmitted. The (n – k) bits in the remaining
portion are computed from the message bits in accordance with a prescribed encoding rule
that determines the mathematical structure of the code. Accordingly, these (n – k) bits are
referred to as parity-check bits. Block codes in which the message bits are transmitted in
unaltered form are called systematic codes. For applications requiring both error detection
and error correction, the use of systematic block codes simplifies implementation of the
decoder.
Let m0, m1, , mk – 1 constitute a block of k arbitrary message bits. Thus, we have 2k
distinct message blocks. Let this sequence of message bits be applied to a linear block
Haykin_ch10_pp3.fm Page 583 Friday, January 4, 2013 5:03 PM

10.4 Linear Block Codes 583

encoder, producing an n-bit codeword whose elements are denoted by c0, c1, , cn – 1. Let
b0, b1, , bn – k – 1 denote the (n – k) parity-check bits in the codeword. For the code to
possess a systematic structure, a codeword is divided into two parts, one of which is
occupied by the message bits and the other by the parity-check bits. Clearly, we have the
option of sending the message bits of a codeword before the parity-check bits, or vice versa.
The former option is illustrated in Figure 10.4, and its use is assumed in the following.
According to the representation of Figure 10.4, the (n – k) leftmost bits of a codeword
are identical to the corresponding parity-check bits and the k rightmost bits of the
codeword are identical to the corresponding message bits. We may therefore write


 b, i = 0 1   n – k – 1
ci =  i (10.1)
 m i+k –n , i = n – k n – k + 1  n – 1

The (n – k) parity-check bits are linear sums of the k message bits, as shown by the
generalized relation
b i = p 0i m 0 + p 1i m 1 +  + p k –1 i m k – 1 (10.2)

where the coefficients are defined as follows:

 1 if b i depends on m j
p ij =  (10.3)
 0 otherwise
The coefficients pij are chosen in such a way that the rows of the generator matrix are
linearly independent and the parity-check equations are unique. The pij used here should
not be confused with the p(j|i) introduced in Section 10.3.
The system of (10.1) and (10.2) defines the mathematical structure of the (n,k) linear
block code. This system of equations may be rewritten in a compact form using matrix
notation. To proceed with this reformulation, we respectively define the 1-by-k message
vector m, the 1-by-(n – k) parity-check vector b, and the 1-by-n code vector c as follows:
m =  m 0 m 1   m k – 1  (10.4)

b =  b 0 b 1   b n – k – 1  (10.5)

c =  c 0 c 1   c n – 1  (10.6)

Note that all three vectors are row vectors. The use of row vectors is adopted in this
chapter for the sake of being consistent with the notation commonly used in the coding
literature. We may thus rewrite the set of simultaneous equations defining the parity
check-bits in the compact matrix form
b = mP (10.7)

b0, b1, . . ., bn – k – 1 m0, m1, . . ., mk – 1

Parity-check bits Message bits

Figure 10.4 Structure of systematic codeword.

Haykin_ch10_pp3.fm Page 584 Friday, January 4, 2013 5:03 PM

584 Chapter 10 Error-Control Coding

The P in (10.7) is the k-by-(n – k) coefficient matrix defined by

p 00 p 01 ⋯ p 0 n – k – 1
p 10 p 11 ⋯ p 1 n – k – 1
P = (10.8)
⋮ ⋮ ⋮
p k –1 0 p k – 1 1 ⋯ p k – 1 n –k – 1
where the element pij is 0 or 1.
From the definitions given in (10.4)–(10.6), we see that c may be expressed as a
partitioned row vector in terms of the vectors m and b as follows:
c= b⋮m (10.9)

Hence, substituting (10.7) into (10.9) and factoring out the common message vector m, we
get
c = m P ⋮ Ik (10.10)

where Ik is the k-by-k identity matrix:

1 0 ⋯ 0
Ik = 0 1 ⋯ 0 (10.11)
⋮ ⋮ ⋱ ⋮
0 0 ⋯ 1
Define the k-by-n generator matrix

G = P ⋮ Ik (10.12)

The generator matrix G of (10.12) is said to be in the canonical form, in that its k rows are
linearly independent; that is, it is not possible to express any row of the matrix G as a
linear combination of the remaining rows. Using the definition of the generator matrix G,
we may simplify (10.10) as
c = mG (10.13)
The full set of codewords, referred to simply as the code, is generated in accordance
with (10.13) by passing the message vector m range through the set of all 2k binary
k-tuples (1-by-k vectors). Moreover, the sum of any two codewords in the code is another
codeword. This basic property of linear block codes is called closure. To prove its validity,
consider a pair of code vectors ci and cj corresponding to a pair of message vectors mi and
mj, respectively. Using (10.13), we may express the sum of ci and cj as
ci + cj = mi G + mj G
=  m i + m j G
The modulo-2 sum of mi and mj represents a new message vector. Correspondingly, the
modulo-2 sum of ci and cj represents a new code vector.
There is another way of expressing the relationship between the message bits and
parity-check bits of a linear block code. Let H denote an (n – k)-by-n matrix, defined as
Haykin_ch10_pp3.fm Page 585 Friday, January 4, 2013 5:03 PM

10.4 Linear Block Codes 585

H = I T
n-k ⋮ P
(10.14)

where PT is an (n – k)-by-k matrix, representing the transpose of the coefficient matrix P,

and In – k is the (n – k)-by-(n – k) identity matrix. Accordingly, we may perform the
following multiplication of partitioned matrices:

T
P
T T
HG = I ⋮ P ⋯
n-k
Ik
T T
= P +P
where we have used the fact that multiplication of a rectangular matrix by an identity
matrix of compatible dimensions leaves the matrix unchanged. In modulo-2 arithmetic,
the matrix sum PT + PT is 0. We therefore have
T
HG = 0 (10.15)
Equivalently, we have GHT =
0, where 0 is a new null matrix. Postmultiplying both sides
of (10.13) by HT, the transpose of H, and then using (10.15), we get the inner product
T T
cH = mGH (10.16)
= 0
The matrix H is called the parity-check matrix of the code and the equations specified by
(10.16) are called parity-check equations.
The generator equation (10.13) and the parity-check detector equation (10.16) are basic
to the description and operation of a linear block code. These two equations are depicted
in the form of block diagrams in Figure 10.5a and b, respectively.

Syndrome: Definition and Properties

The generator matrix G is used in the encoding operation at the transmitter. On the other
hand, the parity-check matrix H is used in the decoding operation at the receiver. In the
context of the latter operation, let r denote the 1-by-n received vector that results from

Generator
Message vector Code vector
matrix
m c
G

(a)

Parity-check
Code vector Null vector
matrix
c 0
H

(b)

Figure 10.5 Block diagram representations of the generator

equation (10.13) and the parity-check equation (10.16).
Haykin_ch10_pp3.fm Page 586 Friday, January 4, 2013 5:03 PM

586 Chapter 10 Error-Control Coding

sending the code vector c over a noisy binary channel. We express the vector r as the sum
of the original code vector c and a new vector e, as shown by
r = c+e (10.17)
The vector e is called the error vector or error pattern. The ith element of e equals 0 if the
corresponding element of r is the same as that of c. On the other hand, the ith element of e
equals 1 if the corresponding element of r is different from that of c, in which case an
error is said to have occurred in the ith location. That is, for i = 1, 2,, n, we have
1 if an error has occurred in the ith location
ei =  (10.18)
0 otherwise
The receiver has the task of decoding the code vector c from the received vector r. The
algorithm commonly used to perform this decoding operation starts with the computation
of a 1-by-(n – k) vector called the error-syndrome vector or simply the syndrome.2 The
importance of the syndrome lies in the fact that it depends only upon the error pattern.
Given a 1-by-n received vector r, the corresponding syndrome is formally defined as
T
s = rH (10.19)
Accordingly, the syndrome has the following important properties.

PROPERTY 1 The syndrome depends only on the error pattern and not on the transmitted codeword.
To prove this property, we first use (10.17) and (10.19), and then (10.16) to write
T
s =  c + e H
T T (10.20)
= cH + eH
T
= eH
Hence, the parity-check matrix H of a code permits us to compute the syndrome s, which
depends only upon the error pattern e.
To expand on Property 1, suppose that the error pattern e contains a pair of errors in
locations i and j caused by the additive channel noise, as shown by
e =  001 i 001 j 00 
Then, substituting this error pattern into (10.20) yields the syndrome
s = hi + hj
where hi and hj are respectively the ith and jth rows of the matrix HT. In words, we may
state the following corollary to Property 1:
For a linear block code, the syndrome s is equal to the sum of those rows of
the transposed parity-check matrix HT where errors have occurred due to
channel noise.

PROPERTY 2 All error patterns that differ by a codeword have the same syndrome.
For k message bits, there are 2k distinct code vectors denoted as ci, where i = 0, 1, , 2k – 1.
Correspondingly, for any error pattern e we define the 2k distinct vectors ei as follows
ei = e + ci for i = 0, 1, , 2k – 1 (10.21)