Scribd
Upload
18 views23 pages

Lec 2,3

Uploaded by

rameshkumar700409
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
18 views23 pages

Lec 2,3

Uploaded by

rameshkumar700409
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 23

Muzaffarpur

Institute of
Technology

Information
Theory and
Coding
Lec-03 & 04 Entropy, Joint
and Mutual Entropy

Ajay Kumar
Assistant Professor
Dept. of ECE
MIT Muzaffarpur

Entropy of a DMS
{So, S1, S2……..….Sx……………Sk-1 }
§

p0, p1, p2……….px… ……..….pk-1


For DMS
H(S) ≜ 𝑘−1𝑥=0 P𝑥 ∗ log2(1/Pk) bits/symbol
This is the average information which will be result
when the output symbol will observe from the
source §.
MIT Muzaffarpur

Properties Of Entropy
 0≤ H(S) ≤ log2(k)
H(S) = 𝑗=0 P𝑗 ∗
𝑘−1
log2(1/P𝑗)
For :
0 < P𝑗 < 1  P𝑗 log2(1/P𝑗) > 0
Pj = 1  P𝑗 log2(1/P𝑗) = 0
Pj = 0  P𝑗 log2(1/P𝑗) = 0
MIT Muzaffarpur

Proof of lower bound


 Pjlog 2 (1/𝑝𝑗) = 0 given pj =0
− log2 𝑒 ∗ln(𝑝𝑗 )
Proof:- lim log 2 (1/𝑝𝑗 ) = lim ( )
pj→0 pj→0 1
pj
𝑑(ln 𝑝𝑗) 1
(𝑝 )
= - log 2 𝑒 lim = - log 2 𝑒 lim 1
𝑑𝑝𝑗 𝑗

pj→0 1
𝑑(𝑝 )
𝑗
pj→0 (−𝑝 ) 2
𝑗
𝑑𝑝𝑗

= log 2 𝑒 lim pj = 0
pj→0
Pjlog 2 (1/𝑝𝑗) = 0, iff pj =0 or 1
H(S) = 0 iff Pj =1 for some „j‟ and all the rest are
zero.
MIT Muzaffarpur

Proof of Upper bound


 Using inequality
ln(x) ≤ 𝑥 − 1 for x ≥ 0 …………….(I)
Consider
{po, p1, ….. Pk-1} and { q0, q1,…..qk-1}
On S = {S0, S1,……Sk-1}
𝑘−1 𝑘−1
𝑝
𝑘=0 𝑘 log 2 (𝑞𝑘/𝑝𝑘) = log 2 (𝑒) 𝑘=0 𝑝𝑘 𝑙𝑛(𝑞𝑘 /𝑝𝑘)…..(ii)
Using (1), we get
𝑞𝑘
𝑘=0 𝑝𝑘 (𝑝 -1)
𝑘−1 𝐾−1
𝑘=0 𝑝𝑘 log 2 (𝑞𝑘/𝑝𝑘) ≤ log 2 (𝑒)
𝑘

= log 2 (𝑒){ 𝐾−1𝑘=0 𝑞𝑘 − 𝑘=0 𝑝𝑘 }


𝐾−1

= 0.
∴ 𝑘−1𝐾−1
𝑝𝑘 log
𝑘=0𝑘=0 2 (𝑞𝑘/𝑝𝑘)
𝑝𝑘 log ≤0
2 (𝑞𝑘/𝑝𝑘) ≤ 0
MIT Muzaffarpur

Continued…
Equality holds if qk = pk ∀ k
Let qk = 1/k………. ∀ k
H(S) = 𝐾−1
𝑘=0 𝑞𝑘 log 2 (1/𝑞𝑘 ) = log 2 (𝑘)…..(3)
∴ 𝐾−1
𝑘=0 𝑝𝑘 log 2 (1/𝑝𝑘 ) ≤ log2K
H(S) ≤ log2K
MIT Muzaffarpur

Example
consider the situation where you have just two symbols
with
probabilities “p‟ and “(1-p)‟.
Average information per symbol is
Entropy of a binary memoryless source
S = {0,1} p0
p1 = (1 – p0)
H(S) = -polog2p0 – (1-p0)log2(1-p0)
At p =0, H =0 and at p = 1, H = 0 again,
The maximum value of H can be obtained as,
Hmax = ½ log2 2 + ½ log 2 = log22 = 1
Hmax = 1 bit / message
Plot of H can be shown as
MIT Muzaffarpur

Continue…
H(p) =-plog2p – (1-p)log2(1-p) {H(p)→”Entropy Function”}
𝑑𝐻(𝑝) 𝑝 1−𝑝
= -log2p – 𝑝*log2e + log2(1-p) + 1−𝑝 log2e = log2((1-p)/p)
𝑑𝑝

𝑑2𝐻(𝑝) log2e
 =- < 0 for all p : 0<p<1
𝑑𝑝2 𝑝(1−𝑝)
𝑑𝐻(𝑝) 1
 =0⇒𝑝=
𝑑𝑝 2
 H(p = ½) = 1
H(p) is “concave”
MIT Muzaffarpur

Information Rate
 Average rate at which information is transferred
is called information rate.

where, r – Symbol rate


H – Entropy
MIT Muzaffarpur

Question
Q1:- Verify the equation I(xixj) = I(xi) + I(xj), if xi and xj are
independent.

Q2:- DMS X produces four symbols x1, x2, x3, and x4 with
probabilities P(x1) = 0.4, P(x2) = 0.3, P(x3) = 0.2, and P(x4) =
0.1, respectively. Determine H(X). Also obtain the
information contained in the message x1x2x1x3 and
x4x2x3x2 .

Q3:- A binary memoryless system produces two messages


with probabilities p and 1 – p. Show that the entropy is
maximum when both messages are equiprobable.
MIT Muzaffarpur

Question
Q4:-Consider that a digital source sends M independent
messages. Show that the source entropy attains a
maximum value if all the messages are equally likely.

Q5:- An event has four possible outcomes with probabilities of


occurrence p1 = 1/4, p2 = ½, p3 = 1/8 and p4 = 1/8
respectively. Determine the entropy of the system. Also
obtain the rate of information if there are 8 outcomes
per second.

Q6:- An analog signal having bandwidth 1 Hz is sampled at


the Nyquist rate and the samples are quantized into four
levels Q1, Q2, Q3, and Q4 with probabilities of occurrence
p1 = p2 = 3/5 , and p3 = p4 = 2/5 , respectively. The
quantization levels are assumed to be independent. Find
the information rate of the source.
MIT Muzaffarpur

Joint entropy: Entropy of sequences


generated from several sources
• To study the behavior of a communication system, we must
simultaneously study the behavior of transmitter and receiver.
This use the concept of two-dimensional probability scheme.
• finding the entropy of the two-dimensional probability
scheme is called as joint entropy.
• Consider two sources S1 and S2 delivering symbols X=(x1,
x2,…., xm) and Y= (y1, y2,….., yn)
• Let P(xi) and P(yj) denote the probabilities of X=xi and Y=yj,
where xi and yj are symbols generated from two sources.
• Then the joint probability of X=xi and Y=yj is P(xi,yj)
• The joint entropy of X and Y is defined as
MIT Muzaffarpur

Three probabilities and their


entropies
MIT Muzaffarpur

Conditional entropy
MIT Muzaffarpur

Relationship between joint probability and


conditional probability
MIT Muzaffarpur

Relationship between joint probability and


conditional probability
MIT Muzaffarpur

Relationship between joint probability and


conditional probability
MIT Muzaffarpur

Continued..
 If xi and yj are independent events, we
know that P(xi,yj) = P(xi)P(yj) then,

H(X|Y) = H(X), H(Y|X) = H(Y)


&
H(X,Y) = H(X) + H(Y)

o If xi and yj are dependent events, then


H(X,Y) = H(Y) + H(X|Y)
H(X,Y) = H(X) + H(Y|X)
MIT Muzaffarpur

Properties of Mutual Information

1. I(X;Y) =I(Y;X)

2. I(X;Y) ≥ 0

3. I(X;Y) = H(Y) – H(Y/X)

4. I(X;Y) = H(X) + H(Y) – H(X,Y)


MIT Muzaffarpur

Question:-
Consider a BSC with P(x1) = α.
(a) Prove that the mutual information I(X; Y) is given by
I(X; Y) = H(Y) + p log2p + (1 – p) log2(1 – p).
(b) Compute I(X; Y) for α = 0.5 and p = 0.1.
(c) Repeat (b) for α = 0.5 and p = 0.5 and comment on
the result.
MIT Muzaffarpur

Problems on conditional and


joint entropies
Consider a channel with noise
characteristics:-
MIT Muzaffarpur

Five Entropies
• H(X): Average information per character at the transmitter,
or entropy of the transmitter.

• H(Y): Average information per character at the receiver, or


entropy of the receiver.

• H(X,Y): Average information per pair of the transmitted and


received characters, or average uncertainty of the
communication system as a whole.

• H(X|Y): A transmitted character xj may be the result of the


reception of yk with a given probability, or it is a measure of
information about the transmitter, when it is known that Y is
received.

• H(Y|X): A received character yk may be the result of the


transmission of xj with a given probability, or it is a measure of
information about the receiver, where it is known that X is
transmitted.
MIT Muzaffarpur

Analysis of entropies
 H(X) and H(Y) give indications of the
probabilistic nature of the transmitter and
receiver respectively.

 H(X|Y) indicates how well one can recover the


transmitted symbols, from the received symbols;
i.e, it gives the measure of equivocation.

 H(Y|X) indicates how well one can recover the


received symbols from the transmitted symbols;
i.e, it gives the measure of error, or noise.

You might also like