Information & Coding Theory (ICT)

Module 2

By
Dr Akriti Nigam
Computer Science & Engineering
Department
BIT, Mesra

1
Digital Communication Systems

Source of User of
Information Information

Source Source
Encoder Decoder

Channel Channel
Encoder Decoder

Modulator De-Modulator

Channel

2
 Error Control Coding

Designing codes for the reliable transmission of digital information over a

noisy channel.
Codes can either correct or merely detect errors

 Codes that can detect errors are called error-detecting codes

 Codes that can correct errors are called error-correcting codes

Error correction is more complex than error detection!

Error control codes are classified into

 Block Codes
 Convolutional Codes
 What are Linear Block Codes?

Linear Block Codes

 Information sequence is segmented into message blocks of
fixed length.
 Each k-bit information message is encoded into an n-bit
codeword (n>k)

2k Binary Block 2k
k-bit Messages Encoder n-bit DISTINCT
codewords

4
 What are Linear Block Codes?

Linear Block Codes

 Modulo-2 sum of any two codewords is ………
also a codeword
 Each codeword v that belongs to a block code C is a
linear combination of k linearly independent codewords
in C, i.e.,

v  u 0g0  u 1g1  ...  u k 1gk-1 ,

v gi  g i 0 g i 1 ... g i ,n 1 
 u0 g 0  u1 g1  ...  u k 1 g k 1

5
 Linear Independence
 A set of vectors g0, g1,…, gk-1 are linearly independent if
there exists no scalars u0, u1,…, uk-1 that satisfy

v  u 0g0  u 1g1  ...  u k 1gk-1  0

Unless u0=u1=…= uk-1=0

 Examples
 [0 1 0 ], [1 0 1], [1 1 1] are ………
 Linearly Dependent
 [0 1 0 ], [1 0 1], [0 0 1] are ………
 Linearly Independent

6
 Linear Block Codes
 Stores k linearly independent codewords
 Encoding process through linear combination of codewords g0, g1,
…, gk-1 based on input message u=[u0, u1,…, uk-1]

 g0   g 00 g 01 ... g 0 ,n 1 

v=u.G g   g
 1
 
 .   .
10 g 11 ... g1,n 1 
. . 

G=   . . .

 .   
 .   . . . 
   
 gk 1   g k 1, 0 g k 1,1 ... g k 1,n 1 
Generator Matrix
7
• Let v = (v0, v1, ..., vn-1 ) be a binary n-tuple, the
Hamming weight (or simply weight) of v, denote by
w(v), is defined as the number of nonzero components of v
• For example, the Hamming weight of v = (1 0 0 0 1 1 0) is 3
• Let v and w be two n-tuple, the Hamming distance
between v and w, denoted d(v,w), is defined as the
number of places where they differ.
• For example, the Hamming distance between v = (1001011) and
w = (0100011) is 3

8
 Example Message Codeword
0000 0000000

 g0  1 1 0 1 0 0 0 0001 1010001 g3
 g  0 1 1 0 1 0 0 0010 1110010 g2
G=  1    
g2  1 1 1 0 0 1 0 0011 0100011
    0100 0110100 g1
 g3  1 0 1 0 0 0 1
0101 1100101
0110 1000110
u= [0 1 1 0] 0111 0010111
1000 1101000 g0
1001 0111001
Linear Block
Encoder (v=u.G) 1010 0011010
1011 1001011
1100 1011100

v= g1+g2 1101 0001101

v= [1 0 0 0 1 1 0] 1110 0101110
1111 1111111

9
 Example
 g0  1 1 0 1 0 0 0
 g  0 0 1 0 1 1 1 Linearly
G=  1     Dependent
g2  1 1 1 1 1 1 1
   
 g3  1 0 1 0 0 0 1

u= [0 1 1 1] u= [1 0 0 1]

Block Encoder Block Encoder

(v=u.G) (v=u.G)

v= g1+g2+g3 NOT DISTINCT v= g0+g3

v= [0 1 1 1 0 0 1] v= [0 1 1 1 0 0 1]

10
 Linear Systematic Block Codes
n-k bits k bits

Redundant Checking Message

Part Part
p-matrix kxk- identity matrix
 p 00 p 01 ... p 0 ,n k 1 1 0 ... 0
 p p11 ... p1,n k 1 0 1 ... 0
 10
 . . . . . .
G=P Ik    
 . . . . . .
 . . . . . .
 
 p k 1, 0 p k 1,1 ... p k 1,n k 1 0 0 ... 1 

11
 The Parity Check Matrix
 For any k x n matrix G with k linearly independent rows,
there exists an (n-k) x n matrix H (Parity Check Matrix),
such that
 G.HT=0

 1 0 ... 0 p 00 p 01 ... p k 1,0 

 0 1 ... 0 p 01 p11 ... p k 1,1 
 
 . . . . . . 
H=Ik P   
T

 . . . . . . 
 . . . . . . 
 
 0 0 ... 1 p 0 ,n k 1 p1,n k 1 ... p k 1,n k 1 

12
Example

1 1 0 1 0 0 0
0 1 1 0 1 0 0
G 
1 1 1 0 0 1 0
 
1 0 1 0 0 0 1

1 0 0 1 0 1 1 
H  0 1 0 1 1 1 0 
 
0 0 1 0 1 1 1 
13
 Encoding Circuit 1
0
1
1
0
1
1
0
0
1
0
0
0
0
G 
1 1 1 0 0 1 0
 
1 0 1 0 0 0 1

Input u Message Register

u0 u1 u2 u3
To channel

Output v

+ + +

v0 v1 v2

Parity Register

[u0 u1 u2 u3] Encoder Circuit [v0 v1 v2 u0 u1 u2 u3]

14
 Syndrome
 Characteristic of parity check matrix (H)

T
v.H  0 vC
T
v.H  0 vC
v Channel r Syndrome
v r=v+e T
+
s=r.H
e Error Pattern

15
 Error Detection

s0 rC
r is NOT a codeword

An Error is Detected: What Options do we have?

 Ask for Retransmission of Block
 Automatic Repeat Request (ARQ)
 Attempt the Correction of Block
 Forward Error Correction

16
 Undetectable Error Patterns

s0 rC
 Can we be sure that r=v ??
 NO! WHY?

eC  rC

17
 Syndrome Circuit

r0 r1 r2 r3 r4 r5 r6

+ + + 1 0 0
0 1 0
 
s0 s1 s2 0 0 1
HT  1 1 0

s 0  r 0  r3  r5  r 6 0 1 1
s 1  r1  r3  r 4  r5  
1 1 1
s 2  r 2  r 4  r5  r 6 1 0 1 

18
 19

Linear Block Codes

Binary Field: The set K = {0, 1} is a binary field. The binary field has two
operations, addition and multiplication

 Addition

 Multiplication
Error Detection and Correction Capabilities:
The minimum distance dmin of a linear code C determines the error
20
detection and correction capabilities of C.
 A linear code C of minimum distance dmin can detect up to t errors iff

 A linear code C of minimum distance dmin can correct up to t errors iff

 Syndrome Decoding 21

Let r denote the received word of length n when codeword c of length n

was sent over a noisy channel.

where e is called the error pattern. Consider first the case of a single error
in the ith position. Then,

Evaluate as

where s is called syndrome of r.

 Using s and noting that is the ith row of HT, we can identify the
error position by comparing s to the rows of HT.
 Note that the zero syndrome indicates that r is a codeword and is
presumably correct.
 Let v = (v0, v1, , ..., vn-1 ) be a code word that was 22

transmitted over a noisy channel

 Let r = (r0, r1, ..., rn-1) be the received vector at the
output of the channel
 e = r + v = (e0, e1,..., en-1) is an n-tuple
 ei = 1 for r i ≠ vi
 ei = 0 for r i = vi
 The n-tuple e is called the error vector (or error
pattern)
 Upon receiving r, the decoder must first determine
whether r contains transmission errors.
23

 If the presence of errors is detected, the decoder will

take actions to locate the errors.
 Correct errors (FEC)
 Request for a retransmission of v (ARQ)
 When r is received, the decoder computes the
following (n – k)-tuple :
s = r • HT
= (s0, s1, ..., sn-k-1)
 s = 0 if and only if r is a code word and receiver accepts
24

r as the transmitted code word.

 s≠0 if and only if r is not a code word and the presence
of errors has been detected.
 When the error pattern e is identical to a nonzero code
word (i.e., r contain errors but s = r • HT = 0), error
patterns of this kind are called undetectable error
patterns
 Since there are 2K-1 nonzero code words, there are 2K-1
undetectable error patterns
 s
1= (0000001). HT =001
E.g.
1 1 1 0 1 0 0 s
H=
0 1 2= (0000010). HT=010
1 1 0 1 0 25

1 1 0 1 0 0 1 s
3= (0000100). HT=100
s
4= (0001000). HT=011
s
5= (0010000). HT=110
s
6= (0100000). HT=111
s
7= (1000000). HT=101

Suppose a (7,4) transmitted codeword was v=1011000 and single bit

error occurred during transmission and received codeword is
r=1001000. Lets check if r=v

Syndrome= r. HT =(1001000). HT
=110
From syndrome table 110=e(0010000)
Hence corrected codeword is computed by flipping the 3rd bit from
left in r i.e r=1011000
 26

Example: Consider a linear block code with the following parity-check

matrix

a. Determine the generator matrix G.

b. Suppose that the received word is r = [1 1 0 1 1 0]. Decode this
received word, i.e., find c and d.
Decoding Sphere: Nearest Neighbor
27

Decoding

0110
1100
0001

0010 1
1001
1
0000 1010
1000
2
0101
0100

0011
28
*Noise capable of flipping 1 bit
E_det= w-1
E_cor=floor(w-1/2)
29

0000 0001 W=1, S={0000, 0001, 1000, 0100}

E_det= 0
E_cor=0

0000 0001 0101 W=2, S={0000, 0101, 1010, 1100}

E_det= 1
E_cor=0

0000 0001 0101 0111 W=3, S={0000, 0111}

E_det= 2
E_cor=1
 Hamming Bound/ Sphere Packing
30

Bound


Hamming Bound/ Sphere Packing
31

Bound

t
 32

Finding Basis of a vector space

 If you have vectors v1, v2, ..., vn, to find a basis for
the span of the given vector, following steps are
to be followed:
 Construct the matrix of vectors
 Transform the matrix into reduced row echelon form
 Identify the vectors in the original matrix
corresponding to the columns, in the reduced
matrix that contain leading 1’s (the first nonzero
element in the row is a 1)
34
35