ELEC1200: A System View of

Communications: from Signals to Packets

Lecture 12

• Channel coding
• (3,2) parity bit code
• Repetition codes

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
 There’s good news and bad news…
The good news: Our
communication scheme
usually allows us to recover
the original signal despite
small amplitude errors
introduced by the
components and channel.

The bad news: larger amplitude errors (hopefully

infrequent) that change the signal irretrievably.
These show up as bit errors in our digital data
stream.
ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
2
 Channel coding
Our plan to deal with bit errors:

bitstream with redundant redundant bitstream

information used for dealing possibly with errors
with errors

Channel Digital Digital Error

Coding Transmitter Receiver Correction

Message bitstream Recovered message bitstream

We’ll add redundant information to the transmitted bit stream (a process

called channel coding) so that we can detect errors at the receiver. Ideally
we’d like to correct commonly occurring errors, e.g., error bursts of
bounded length. Otherwise, we should detect uncorrectable errors and use,
say, retransmission to deal with the problem.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
3
 More good news, bad news…
Shannon’s Noisy Channel Coding Theorem:

A noisy channel has a channel capacity C, which is the maximum rate at

which useful information can be transmitted. This capacity depends
upon the physical properties of the channel.

For any rate R < C, there exists a code such that the probability of
error (after error correction) is arbitrarily small.

• Good news:
– theoretically it is possible to transmit information without
error at any rate below C.
• Bad news:
– the theorem doesn’t tell us how to construct the error-
correcting code!

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
4
 Claude Elwood Shannon

Claude Elwood Shannon

Father of Information Theory
Electrical engineer, mathematician, and native
son of Gaylord. His creation of information
theory, the mathematical theory of
communication, in the 1940s and 1950s inspired
the revolutionary advances in digital
communications and information storage that
have shaped the modern world.

This statue was donated by the

Information Theory Society of the Institute of
Electrical and Electronics Engineers,
whose members follow gratefully in his
footsteps.

Dedicated October 6, 2000

Eugene Daub, Sculptor

More about Claude Shannon can be found at https://thebitplayer.com/

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 5
 ELEC1200: A System View of
Communications: from Signals to Packets
Lecture 12

• Channel coding
• (3,2) parity bit code
• Repetition codes

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 6
 (n,k) Block Codes
• Arrange the message into k-bit blocks.
• Create an n-bit codeword by adding (n-k) bits. These added (n-k)
bits depend in some way on the k message bits.
k n-k
The entire block is called
a “codeword”
Message bits Extra bits

• Note that since there are 2k possible combinations of message bits,

there are 2k valid codewords (codewords the transmitter will send)
• Receiving something other than a valid codeword indicates that at
least one bit error has occurred.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 7
 Code Rate
• The code rate indicates what fraction of the sent bits contain useful
information (i.e., the message). We want this to be close to one.
• For the (n,k) block code, k n-k

k
code rate = Message bits Extra bits
n

n
• Related terms
– The gross bitrate is the rate at which bits (useful or not) are sent. Also called
the data signaling rate.
𝑭𝑭𝒔𝒔
Gross bitrate =
𝑺𝑺𝑺𝑺𝑺𝑺
– The net bitrate is the rate at which useful bits are sent.
𝒌𝒌 𝑭𝑭𝒔𝒔
Net bitrate = �
𝒏𝒏 𝑺𝑺𝑺𝑺𝑺𝑺

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 8
 (3,2) Parity Bit Block Code
• We create a 3-bit codeword from a 2-bit message by adding a
parity bit.
– The parity bit is chosen to ensure that the total number of ones in the
resulting codeword is either even or odd.
– We will use even parity in this course. k =2 n-k = 1

Message bits Extra bit

Message bits Parity bit
n=3
00 0 0 “1”s
k 2
2 “1”s code rate = =
01 1 n 3

10 1 2 “1”s
If 𝑭𝑭𝒔𝒔 = 𝟏𝟏𝟏𝟏𝟏𝟏𝟏𝟏 and 𝑺𝑺𝑺𝑺𝑺𝑺 = 𝟓𝟓
𝑭𝑭𝒔𝒔
11 0 2 “1”s Gross bitrate = 𝑺𝑺𝑺𝑺𝑺𝑺 = 200kbps
𝒌𝒌 𝑭𝑭𝒔𝒔
Net bitrate = � = 133kbps
𝒏𝒏 𝑺𝑺𝑺𝑺𝑺𝑺

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
9
 Channel Coding and Decoding – (3,2) parity code
Message bits Channel coding (Step 2)
00 01 10 11
Transmitter
encoder 000 011 101 110
(Step 1) Encoded bits
(Step 3) Noise

(Step 5) Received bits

Decoded bits Channel coding 100 011 100 110
?? 01 ?? 11
Receiver
decoder (Step 4)
?? = errors Note that bit errors can occur
in both message and parity bits!

Steps:
1. Break message sequence to 2-bit blocks We detect errors in received codewords by
2. Encode 2-bit blocks as 3-bit codeword counting the number of ones.
Even: no errors, Odd: error
3. Transmit over channel and decode bits
We cannot correct errors.
4. Break received bit sequence into 3-bit blocks
(We received 100 twice, but different
5. Decode 3-bit blocks and detect errors. codewords were sent).

ELEC1200 10
Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
 Valid and Invalid Codewords
• Note that there are 22=4 possible message blocks, but 23=8 possible
codeword blocks.
• Only 4 of the 8 possible codewords are valid (those created by
coding message blocks). The other 4 codewords are invalid (contain
errors).
Messages Codewords 101 111
00 000
01 001 100 110
010
00 11 01 011
100
001 011
10 10 101
11 110 000 010
Message Space 111
Codeword Space
Arrows indicate codewords
that differ by only one bit. Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
11
 Message and Codeword Spaces
• Recall the relationship between BER and SNR.
– By increasing the signal power (increasing SNR), we avoid errors (reduce BER).
– Increasing the signal power is done by increasing the distance between the
representations of the symbols “0” and “1” in the signal space.
– This distance is |rmax - rmin|.

• Channel coding extends this idea to the digital domain.

– Instead of “0” and “1” as symbols, our symbols are blocks of data.
– We increase distance by adding extra bits. This increases the energy needed to
transmit each symbol.
– Working in the digital domain extends our abilities. Instead of just avoiding
errors, we can now detect, and sometimes even correct, errors.
– To understand how and why, we need to define a distance measure in the
message and codeword spaces, the Hamming distance.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
12
 Hamming Distance (Richard Hamming, 1950)
• The Hamming distance between two blocks of data (codewords) is the
number of bit positions where corresponding bits differ.
• For example,
– The Hamming distance between (00) and (10) is 1.
– The Hamming distance between (010) and (001) is 2.
– What is the Hamming distance between (10000101) and (11110011)?
• The Hamming distance indicates how many bit errors it takes to
transform one codeword to another.

• We measure the error detection/correction capabilities of a code by

measuring the minimum Hamming distance between all possible pairs
of valid codewords.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
13
 Minimum Hamming Distance (2-bit message space)

Hamming Distances Between Message Blocks

Possible
HD 00 01 10 11
messages

00 00 X 1 1 2

01 01 1 X 2 1

10 10 1 2 X 1

11 11 2 1 1 X

Minimum Hamming Distance = 1

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 14
 Minimum Hamming Distance - (3,2) Parity Code

Hamming Distances Between Valid Codewords

Possible Valid
HD 000 011 101 110
messages codewords

00 000 000 X 2 2 2

01 011 011 2 X 2 2

10 101 101 2 2 X 2

11 110 110 2 2 2 X

Minimum Hamming Distance = 2

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 15
 Error Detection vs. Correction
• Error detection
– We detect that an error has occurred, but don’t try to fix (correct) it
– If an error is detected, some of the things the receiver can do are
• Ignore that codeword, or
• Ask the transmitter to retransmit
• Error correction
– we detect and correct errors

The minimum Hamming distance of a code, d, determines how many bit

errors per codeword we can detect or correct.
– We can detect errors in d-1 or fewer bits of each codeword.
OR
– We can correct errors in (d-1)/2 or fewer bits of each codeword.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 16
 ELEC1200: A System View of
Communications: from Signals to Packets
Lecture 12

• Channel coding
• (3,2) parity bit code
• Repetition codes

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 17
 (n,1) repetition codes
• The message is split to 1-bit blocks.
• Thus, there are two possible symbols:
(“0” or “1”) or (“heads” or “tails”)
• Blocks are expanded to n bits by repeating the bit n times.
• Examples
– (2,1) repetition code
• Input bit 0  Codeword 00
Only valid codewords
• Input bit 1  Codeword 11
– (3,1) repetition code
• Input bit 0  Codeword 000
Only valid codewords
• Input bit 1  Codeword 111

• The minimum Hamming distance between codewords is d = n.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 18
 The (2,1) repetition code
• Each bit is repeated twice.
• The minimum Hamming distance between code words is d = 2.

01 single-bit error

“heads” 00 11 “tails” code rate =

1
2

10

• This code can be used to detect (d-1) = 1-bit errors.

– Each “valid” codeword has an even number of “1” bits.
– This code is equivalent to adding a parity bit to a 1-bit message block.
– There is an error if the number of received “1” bits in the codeword is odd.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 19
 The (3,1) repetition code
• Each bit is repeated 3 times.
• The minimum Hamming distance between
codewords is d = 3. code rate =
1
3
• We can either:
– Detect (d-1) = 2-bit errors
– Correct (d-1)/2 = 1-bit errors “tails”

• Note we must choose one or the other, we 101 111

cannot detect both 2-bit errors and correct
100 110
1-bit errors.

001 011
single-bit error

000 010
“heads”
ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 20
 Detecting 2-bit Errors
• We assume that each codeword contains errors
in at most two bits.
“tails”
101 111
• If we receive a codeword with a mixture of 0’s
and 1’s, we know that an error has occurred. 100 110

• For example, if we receive 100, either

– 000 was sent (a 1-bit error occurred) 001 011
– 111 was sent (a 2-bit error occurred)
000 010

“heads”
• Violations of our assumption, i.e., 3-bit errors,
lead to undetected errors

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 21
 Correcting 1-bit Errors
• We assume that each codeword contains
errors in at most 1 bit.
“tails”
101 111
• We correct errors in each received
codeword by seeing whether 0 or 1 has the 100 110
most “votes”.
• Examples:
– If we receive 100, we assume 000 was sent. 001 011
– If we receive 101, we assume 111 was sent.
000 010

“heads”
• Violations in our assumption lead to
– Incorrect corrections (2-bit errors)
– Undetected errors (3-bit errors)

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 22
 Channel Coding/ Error Correction – (3,1) repetition code
Message bits Channel coding (Step 2)
0101
Transmitter
encoder 000 111 000 111
(Step 1) Encoded bits
(Step 3) Noise

(Step 5) Received bits

Decoded bits Channel coding 100 111 000 110
0101
Receiver
decoder (Step 4)

Steps:
1. Break message sequence to 1-bit blocks.
2. Repeat bits to form 3-bit codewords.
We correct errors in received codewords
3. Transmit over channel and decode bits by majority voting.
4. Break received bit sequence into 3-bit blocks • More 1’s: correct to a “1” bit
5. Decode 3-bit blocks to correct errors. • More 0’s: correct to a “0” bit

ELEC1200 23
Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT
 Summary
• Noise in communication systems cause bit errors.
• Error Correcting Codes can be used to detect or correct errors.
• (n,k) block codes add (n-k) extra bits, e.g., a parity bit, to each k-
bit message block.
• The minimum Hamming distance d between codewords in the code
indicates the error detection/correction capabilities of a code.

• The (n,1) repetition code is the simplest type of ECC.

– It only contains two valid codewords: all-0’s or all-1’s
– Since d = n, we can choose EITHER
• To detect, but not correct, errors in n-1 or fewer bits per codeword.
OR
• To correct errors in (n-1)/2 or fewer bits per codeword.

ELEC1200 Some content taken with permission from material for the course EECS6.02 by C. Sodini, M. Perrot and H. Balakrishnan at MIT 24