Channel coding (2)
Prof H Xu
April 3, 2023
Motivation
• Can we improve on repetition codes?
• What if we add redundancy to blocks of data
instead of encoding one bit at a time?
• Linear block codes
Galois Field
Linear block codes
→A block code is a code in which k bits (or, more generally,
symbols) are input and n bits (or, more generally symbols) are
output. We designate the code as an (n,k) code.
→If we input k bits, then there are 2 k distinct messages.
Each message of n symbols associated with each input block
is called a codeword.
→A block code C of length n with 2 k codewords is called
a linear ( n, k ) code if and only if its 2 k codewords form a
k-dimensional subspace of the vector space of all n-tuples.
Linear block codes
→For codewords, it means that the sum of any two codewords
is a codeword.
→Being a linear vector space, there is some basis, say
g0 , g1 , , gk −1 , such that any codeword c C can be
represented as a linear combination of the elements in the
basis:
c = m0 g0 + m1 g1 + + mk −1 gk −1
where mi 0,1 ,1 i k .
→We observe that the all-zero sequence must be a
codeword.
Linear block codes
c = m0 g0 + m1 g1 + + mk −1 gk −1
c = mG
where G : generator matrix
m = ( m0 , m1 , , mk −1 ) : message vector
g0 g 0,0 g 0,1 g 0,n −1
g g1,1 g1, n −1
G= g1
= 1,0
gk −1 g k −1,0 g k −1,1 g k −1,n −1
Linear block codes
Example of encoder:
Example 4.1: A (7,4) code is generated by
1 0 0 0 0 1 1
0 1 0 0 1 0 1
G=
0 0 1 0 1 1 0
0 0 0 1 1 1 1
Suppose m=[1 0 1 0]
1 0 0 0 0 1 1
0 1 0 0 1 0 1
c = mG = [1 0 1 0]
0 0 1 0 1 1 0
0 0 0 1 1 1 1
= [1 0 1 0 1 0 1]
Linear block codes
→Since C is a k-dimensional sudspace of the vector space of
all n-tuple, there is an (n-k)-dimensional dual space generated
by the rows of a matrix H , called the parity-check matrix, such
that GH T = 0 .
→ In particular, for any codeword c C , cH T = 0
→ cH T = 0 is of fundamental importance in decoding of
linear codes.
→Associated with each code C , there is a dual code,
denoted as C ⊥ , that has H as its generator matrix. If
G is the generator matrix for an ( n, k ) code, then H is
the generator matrix for an( n, n − k ) code.
Linear block codes
Example 4.1: A (7,4) code is generated by
1 0 0 0 0 1 1
0 1 0 0 1 0 1
G=
0 0 1 0 1 1 0
0 0 0 1 1 1 1
The parity check matrix is:
0 1 1 1 1 0 0
H = 1 0 1 1 0 1 0
1 1 0 1 0 0 1
Linear block codes
Pictorial representation of encoding:
c = [1 0 1 0 1 0 1]
c5
1
c2 c3
c4 0 1
c7 c1 0
c6
1 1 0
Linear Systematic Block Code
→An ( n, k ) linear systematic code is completely specified by a
k n generator matrix of the following form:
g0
g
G= 1 = I P
k
k −1
g
where I k is the identity matrix, and P is a k (n − k ) matrix.
p0,0 p0,1 p0, n − k −1
p1,0 p1,1 p1, n − k −1
P=
pk −1,0 pk −1,1 pk −1,n − k −1
Linear Systematic Block Code
When P is specified, the code is specified. Selection of the
P matrix defines the code properties such as
→Number of errors that can be corrected.
→Ability to correct random and burst errors.
→For a linear block code, since G = I k P , H = P T I n−k . PT is
the transpose of P .
Prove: given G = Ik P , then H = P T I n−k
Linear Systematic Block Code
Prove: given G = Ik P , then H = P T I n−k
Because c = mG cH T = 0
then mG H T = 0
H = [Q( n−k )k I ( n−k ) ]
m0
GHT = 0
G = Ik P H = [Q( n−k )k I ( n−k ) ]
[ I k Pk( n−k ) ][Q( n−k )k I ( n−k ) ]T = 0
[Q( n−k )k ]T
[ I k Pk( n−k ) ] =0
I ( n−k )
[Q( n−k )k ]T + Pk( n−k ) = 0 [Q( n − k )k ]T = Pk ( n − k )
Linear Systematic Block Code
Example 4.1: A (7,4) code is generated by
1 0 0 0 0 1 1 0 1 1
0 1 0 0 1 0 1 1 0 1
G= Then
0 0 1 0 1 1 0 Pk( n−k ) =
1 1 0
0 0 0 1 1 1 1
1 1 1
The parity-check matrix is given by
H = P T I n − k
0 1 1 1 1 0 0
H = 1 0 1 1 0 1 0
1 1 0 1 0 0 1
Linear Systematic Block Code
Pictorial representation of parity check matrix:
c5
0 1 1 1 1 0 0 c2 c3
H = 1 0 1 1 0 1 0
c4
1 1 0 1 0 0 1
c7 c1 c6
Linear Systematic Block Code
Example 4.2: A ( 6,3) code is generated by
1 0 0 1 1 1 1 1 1
G = 0 1 0 1 0 1 Then Pk( n−k ) = 1 0 1
0 0 1 0 1 1 0 1 1
The parity-check matrix is given by
H = P T I n − k
1 1 0 1 0 0
H = 1 0 1 0 1 0
1 1 1 0 0 1
Linear Systematic Block Code
Pictorial representation of parity check matrix:
c4
1 1 0 1 0 0
H = 1 0 1 0 1 0 c2
1 1 1 0 0 1
c1
c6 c3 c5
Hamming Distance and Error Protection
Properties
Hamming distance between two n-symbol vectors u and v
is defined as the number of places where u and v differ
from
where u = ( u0 , u1 , , un −1 ) v = ( v , v , , vn −1 )
0 1
Example: u = (01101)
v = (11110)
d H (u , v) = 3
Hamming Distance and Error Protection
Properties
→Hamming weight and Hamming distance of codewords
(1) The Hamming weight of a codeword c , denoted wt ( c ) ,
is the number of nonzero components of c .
wt ( c ) = i | ci 0, i = 0,1, , n − 1
(2) For a linear code C , the Hamming distance between any
two codewords is simply described by
d H ( c1 , c2 ) = wt ( c1 − c2 ) = wt ( c3 ) d H ( c1 , c2 ) = wt ( c1 + c2 )
where c3 is the difference between c1 and c 2 .
(3)Triangle inequality
For codewords , c1 , c2 and c3 d H ( c1 , c2 ) + d H ( c2 , c3 ) d H ( c1 , c3 )
Minimum Distance and Minimum Weight
of a Linear Block Code
→Let C be a linear block code. The minimum distance of C ,
denoted as d min , is defined as follows:
d min min d ( u, v ) ; u, v C , u v
i.e the smallest distance between any pair of codewords
in the code.
→The minimum weight of C , denoted wmin , is defined
as follows:
wmin min wt ( c ) : c C , c 0
It can be shown that the minimum distance is also equal to
the minimum weight of any nonzero word in the code.
Error Detection and Correction Capabilities
d min − 1
→A linear block code can correct t = errors or detect
2
. ( d min − 1)
→In most cases, the decoder will associate a received
vector with the code vector closest to it in the sense of
Hamming distance.
→Hybrid mode of error control
One can correct t errors and still detect up to t d
errors provided that 2t + td d min .
Error Detection and Correction Capabilities
Example 4.3: A (7,4) code is generated by
1 0 0 0 0 1 1
0 1 0 0 1 0 1
G= d min = 3 c = [1 0 1 0 1 0 1]
0 0 1 0 1 1 0
0 0 0 1 1 1 1
c5 1
c2 c3 0 1
c4 0
c7 c1 c6 1 1 0
Error Detection and Correction Capabilities
t =1
c5 1
c2 c3 0 0
c4 0
c7 c1 c6 1 1 0
Big picture of linear block codes
For an (n,k) linear block code, there are 2 k possible messages.
That needs 2 k possible codewords. But there are actually 2 n
codewords.
The game is to choose the set of codewords so that the
Hamming distance between these codewords is maximal (set of
2 k codewords chosen among 2 n possible codewords).
Weight Distribution
Let C be an ( n, k ) linear block code and wi denotes the number
of codewords in C with Hamming weight i . The weight
enumerator polynomial is defined as:
n
W ( z ) = wi z i
i =0
Clearly,
w0 = 1
w0 + w1 +
+ wn = 2k
Weight Distribution
Example: Find the weight enumerator polynomial of the code
generated by
codewords
0000000
0001111
0010110
1 0 0 0 1 0 1 0011001
0 1 0 0 1 1 1
0100101
G=
W ( z ) = 1 + 7 z3 + 7 z 4 + z7
0101010
0 0 1 0 1 1 0
0110011
0 0 0 1 0 1 1 0111100
1000011
1001100
1010101
1011010
1100110
1101001
1110000
1111111
Some Commonly-Used Modifications of Linear
Block Codes
(1) Shortened Code →correct more error patterns
A code is shortened by deleting some message symbols
(bits) from the encoding process.
(2) Extended Code →correct more errors
A code is extended by adding some additional redundant
symbols (or bits).
(3) Punctured Code → increase code rate
A code is punctured by deleting some of its parity
symbols (or bits).
Syndrome and Error Detection
→Suppose a codeword c = ( c0 , c1 , , cn −1 ) in a block code C is
transmitted and the received word is given by r = ( r0 , r1 , , rn −1 ) .
→The difference between r and c is called the error
pattern e = r + c = ( r0 + c0 , r1 + c1 , , rn −1 + cn −1 ) = ( e0 , e1 , , en −1 )
→There are a total of 2 possible error patterns. Among
n
these error patterns, only 2n − k of them are correctable and
2 n − 2 k of them are detectable by an linear block code.
→To minimize the probability of a decoding error, it is
desired to design a code, which corrects the 2n − k most
probable error patterns.
Syndrome and Error Detection
→Let C be an ( n, k ) linear code with parity-check matrix H .
Then the ( n − k ) -tuple s = r H T
= ( s0 , s1 , , sn − k −1 )
is called the syndrome of r .
→ r is a codeword in code C if and only if s = 0 .
Hence, if s 0 , r is not a codeword and contains
transmission errors. In this case, we say the presence of
errors is being detected.
→If s 0 , r is a codeword. In this case, r is assumed to
be error-free and accepted by the receiver. A decoding error
is committed if r is a codeword which is different from
the actually transmitted codeword.
Syndrome and Error Detection
→ s = r H = (c + e ) H = c H + e H = e H
T T T T T
T
s0 h0, 0 h0,1 h0,n−1
1
s h1, 0 h1,1 h1,n−1
= (e0 e1 en−1 )
s h hn−k −1,n −1
n−k −1 n−k −1, 0 hn−k −1,1
s0 = e0 h0,0 + e1h0,1 + + en −1h0,n −1
s1 = e0 h1,0 + e1h1,1 + + en −1h1, n −1
sn − k −1 = e0 hn − k −1,0 + + en −1hn − k −1,n −1
An error correction scheme is a method of solving the set of
linear equations.
Syndrome and Error Detection
Example of calculating syndrome
1 0 0 0 1 1
H = 0 1 0 1 0 1 r = (101001)
0 0 1 1 1 0
1 0 0
0 1 0
0 0 1
s = ( s0 , s1 , s2 ) = r H = ( r0 , r1 , r2 , r3 , r4 , r5 )
T
0 1 1
1 0 1
1 1 0
s0 = r0 + r4 + r5 s1 = r1 + r3 + r5 s2 = r2 + r3 + r4
Standard Array
→The standard array is a way of tabulating all of the 2 n n -tuples
of the vector space.
→This 2 n − k 2 k rectangular array is used as a decoding table by
simply locating the minimum-weight n -tuple word in the table.
Standard Array
1 0 0 0 1 1 1
G = 0 1 0 1 0 1 1
0 0 1 1 1 0 1
1. Write down all codewords in the top row ;
2. Each row gives vectors with the same syndrome;
3. Pattern of least weight is put on far left.
Standard Array
0 1 1 1 0 0 0
1 0 0 0 1 1 1 1 0 1 0 1 0 0
G = 0 1 0 1 0 1 1 H =
1 1 0 0 0 1 0
0 0 1 1 1 0 1
1 1 1 0 0 0 1
1. Write down all codewords in the top row ;
2. Each row gives vectors with the same syndrome;
3. Pattern of least weight is put on far left.
Standard Array
0 1 1 1 0 0 0
1 0 0 0 1 1 1 1 0 1 0 1 0 0
G = 0 1 0 1 0 1 1 H =
1 1 0 0 0 1 0
0 0 1 1 1 0 1
1 1 1 0 0 0 1
1. Write down all codewords in the top row ;
2. Each row gives vectors with the same syndrome;
3. Pattern of least weight is put on far left.
How to choose the error patterns
Look at the above example again.
1 0 0 0 1 1 1
G = 0 1 0 1 0 1 1
0 0 1 1 1 0 1
Because t = 4 We can firstly determine the error patterns:
0000001; 0000010; 0000100; 0001000; 0010000; 0100000; 1000000
We also can get the syndromes of these error patterns:
0001; 0010; 0100; 1000; 1101; 1011; 0111;
How to choose the error patterns
Secondly, we choose weight 2 error patterns. If the
syndrome of an error pattern with weight 2 is different from
those of error patterns with weight 1 or 2, the weight 2 error
pattern is valid, otherwise we need to choose others.
For example, we choose an error pattern: 0000011
The syndrome is 0011.
But if we choose an error pattern: 1100000
The syndrome is 1100 which is the syndrome of
0001100. So we need to pick up other error patterns
Standard Array
1 0 0 0 1 1 1
G = 0 1 0 1 0 1 1
0 0 1 1 1 0 1
+
c = m G
One error pattern
( s0 , , s7 )
Randomly set two errors pattern
Calculate s_8 , s8 ( s0 , , s7 )
How to choose the error pattern?
Final syndrome decoding table
Example: Let m=110 → c=1101100 and e=0100000
→r=1001100 0 1 1 1
1 0 1 1
1 1 0 1
s = 1 0 0 1 1 0 01 0 0 0 = 1 0 1 1
0 1 0 0
0 0 1 0
0 0 0 1
From s=[1011]→error pattern=[0100000]
→c=r+e=1001100+0100000=1101100
Performance of linear block codes
The error detection and capability of a linear block code, is
determined by its minimum distance:
→Error detection capability is given by e = d min − 1
d min − 1
→Error correcting-capability t of a code is given by e = 2
For memory less channels, the probability of an erroneous
decoding is
n
n
PCodeword p j (1 − p) j
j =t +1 j
p is the transition probability or bit error probability
over channel.
Performance of linear block codes
The decoded bit error probability is
1 n n
pbit j p j (1 − p)n − j
n j =t +1 j
