Scribd
Upload
14 views

Lecture 3 - Mutual Information. Source Coding and Channel Coding 2

Lectures

Uploaded by

1234 Ма
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
14 views

Lecture 3 - Mutual Information. Source Coding and Channel Coding 2

Lectures

Uploaded by

1234 Ма
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 23

Lecture 3 - Mutual information.

Source coding and channel


coding
Review ● Information Source
● Measure of information
● Entropy
● Information rate
● Joint, Conditional and
Marginal Probability
● Joint and Conditional
entropy
● Chain rule
Plan ● Relative entropy
● Mutual information
● Properties of mutual
information
● Relationship between entropy
and mutual information
● Chain rule for entropy,relative
entropy and mutual
information
● Source coding
● Channel coding
Relative entropy

● Let us start with the definition of the relative entropy, which measures inefficiency of
assuming that a given distribution is q(x) when the true distribution is p(x).
● The relative entropy or Kullback-Leibler distance between two probability
distributions p(x) and q(x) is defined as
Relative entropy

● Important is that the relative entropy is always nonnegative and it is zero if and only
if p(x) = q(x).
● It is not a distance in the mathematical sense since it is not symmetric in its
parameters and it does not satisfy the triangle inequality. Nonetheless, it is often
useful to think of relative entropy as a “distance” between distributions.
Mutual information

● Consider two random variables X and Y with a joint probability mass function p(x, y)
and marginal probability mass functions p(x) and p(y). The mutual information I (X; Y
) is the relative entropy between the joint distribution and the product distribution
p(x)p(y):
Properties of mutual information

● Symmetrical property:

I(X;Y) = I(Y;X)

● Non-negative property:

I(x;y)≥0
Relationship between entropy and mutual information

Theorem: I(X; Y) = H(X) - H(X|Y)

Proof:
Relationship between entropy and mutual information

Theorem: I(X; Y) = H(Y) - H(Y|X)

Proof:

By symmetry property it also follows that,

I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)


Relationship between entropy and mutual information

Theorem: I(X; Y) = H(X) +H(X|Y) - H(X,Y)

Proof:

Since H(X,Y) = H(X) + H(Y|X) by Chain rule,

I(X; Y) = H(X) - H(X|Y) = H(X) +H(X|Y) - H(X,Y)


Relationship between entropy and mutual information

● I(X;X) = H(X) - H(X|X) = H(X)

Thus the mutual information of a random variable with itself is the entropy of the random
variable. This is the reason that entropy is sometimes referred to as self-information.
Relationship between entropy and mutual information

Theorem: (Mutual information and entropy) :


Chain rule for mutual information

● It can be shown from the definitions that the mutual information of (X, Y ) and Z is
the sum of the mutual information of X and Z and the conditional mutual information
of Y and Z given X. That is,

I(X, Y ;Z) = I(X;Z) + I(Y ;Z|X)

● More generally, for n random variables,


Coding theory

● Source coding
○ Loseless data compression
○ Lossy data compression
● Channel coding

While data compression removes as much redundancy as possible, an error-


correcting code adds just the right kind of redundancy (i.e., error correction) needed
to transmit the data efficiently and faithfully across a noisy channel
Source coding

Source coding = Entropy Coding.

A conversion of the output of a discrete memoryless source (DMS) into a sequence of


binary symbols i.e. binary code word, is called Source Coding. The device that performs
this conversion is called the Source Encoder.

Objective of Source Coding: An objective of source coding is to minimize the average


bit rate required for representation of the source by reducing the redundancy of the
information source.
Source coding

● Prefix coding
● Huffman coding
● Shannon-Fano
● Lempel Ziv coding
● DM and adaptive DM (ADPCM)
Few Terms Related to Source Coding Process:

● Codeword Length
● Average Codeword Length
● Code Efficiency
● Code Redundancy
Codeword length

Let X be a DMS with finite entropy H (X) and an alphabet {𝑥1 … … . . 𝑥𝑚} with
corresponding probabilities of occurrence P(xi) (i = 1, …. , m). Let the binary codeword
assigned to symbol xi by the encoder have length ni, measured in bits. The length of the
codeword is the number of binary digits in the codeword.
Average codeword length

The average code word length L, per source symbol is given by

The parameter L represents the average number of bits per source symbol used in
the source coding process.
Code efficiency

● The code efficiency is defined as

Where Lmin is the minimum possible value of L. When approaches unity, the code is said
to be efficient.
Code redundancy

The code redundancy is given by


Channel coding

● Channel coding = Error-Control Coding.


● Encoder adds additional bits to actual information in order to detect and correct
transmission errors
● Error correcting code adds just the right kind of redundancy as possible (i.e., error
correction) needed to transmit the data efficiently across noisy channel
● Channel coding is done at receiver
Channel coding

● Block coding
● Convolutional coding
● Turbo coding

You might also like