ECE503: Communications Engineering 4

Lecture 1

Dr. Fatma Khallaf

Email: [email protected]
Spring 2025
 Course Code

cecf57n
ECE503: Communications Engineering 4 (2025) | General | Microsoft Teams

2
 General Information

Code/ Course Name ECE503 Communications Engineering 4

Pre-requisite ECE502 Communications Engineering 3
Credit Hours 3 Hours
No. of Weeks 10 Weeks
Day/ Time/ Place Sunday/ 9.00-11.00/ F201
Grades (100%) • 40% Final Exam
• 30% Mid-term Exam
• 30% Others
✓ Quizzes
✓ Assignments
✓ Attendance

3
 Course Materials

Reference:

“Communication System Engineering”

2nd Edition by: John G. Proakis

4
 1 Communications Engineering 1
All Types of AM ( DSB-LC, DSB-SC, SSB, VSB, QAM) – AM modulators, and demodulators,
advantages and disadvantages- Synchronization circuits - AM applications: Telephone channel
multiplexing and superheterodyne receiver -Angle Modulation - Narrow band angle modulated
signals - Spectrum of sinusoidal signal (N.B and W.B) - Generation of wide band FM ( Indirect
and Direct methods)-Demodulation (slope detector, PLL) - De-emphasis and pre-emphasis
filtering -compatible stereo - Intersystem comparison – Sampling process – PAM –
Quantization (uniform and non-uniform) – PCM – Time division multiplexing – Delta, and
adaptive delta modulation – Differential PCM – random process – Stationary and ergodic
processes – Mean, correlation, and covariance functions – Power spectral density – Narrow
band noise.

2 Communications Engineering 2

Baseband Pulse transmission: Matched filters, Intersymbol Interference, Nyquist

Criterion for distortional baseband binary transmission - Signal-Space Analysis: Geometric
representation of signals, likelihood functions, coherent detection of signals in noise: ML and
MAP decoding rules, the correlation receiver. Probability of error calculation – Pass-band
Digital Transmission: Description of ASK, FSK, PSK, DPSK, QAM, MSK modulation
schemes - their implementation PSD c/cs - B.W efficiency (spectral efficiency) - performance
in AWGN channels

5
 3 Communications Engineering 3

DFT and its properties – Fading (fast, slow, and flat) – Frequency selective and non-
selective – Dual Multi-Tone (DMT) – OFDM – Multi- path propagation – Delay
spread values – Guard time and cyclic extension – OFDM parameters – OFDM
versus single carrier modulation - Spread Spectrum – PN sequence generators –
Direct sequence Spread Spectrum – Probability of error – Frequency Hopping
Spread Spectrum – CDMA – DS-CDMA.

4 Communications Engineering 4

Uncertainty, information, and Entropy – Source coding – Properties of source codes:

Uniquely decodable codes, Instantaneous codes, construction of instantaneous
codes, The Kraft inequality - Huffman and Fano codes – Quantization - Discrete
memory-less channels – Mutual information – Channel capacity - linear block codes
– syndrome calculation – Cyclic codes – Convolutional coding– The code tree,
trellis and state diagram - ML decoding of convolutional codes: The Viterbi
algorithm - free distance of the convolutional code.

6
 Contents

• Introduction
• Overview: What is Information Theory?
• Historical Background
• Probability Rules
• Entropy
• Mutual Information

7
 Introduction
➢ Information: is the reduction of uncertainty. Imagine your
friend invites you to dinner for the first time. When you
arrive at the building where he lives you find that you have
misplaced his apartment number. He lives in a building with
4 floors and 8 apartments on each floor. If a neighbor passing
by tells you that your friend lives on the top floor, your
uncertainty about where he lives reduces from 32 choices to 8.
By reducing your uncertainty, the neighbor has conveyed
information to you. How can we quantify the amount of
information?
➢ Information theory is the branch of mathematics that describes
how uncertainty should be quantified, manipulated and
represented.
8
 Overview
What is Information Theory?

➢ Key idea: The movements and transformations of information,

just like those of a fluid, are constrained by mathematical
and physical laws. These laws have deep connections with:
• Probability theory
• Spectral analysis (Fourier (and other) transforms)
• Sampling theory and prediction theory
• Electrical engineering (bandwidth, signal-to-noise ratio)
• Signal processing

9
 Overview

Information theory addresses and answers the two fundamental

questions of communication theory:

1. What is the ultimate data compression?

• Answer: the entropy of the data, H, is its compression limit.
2. What is the ultimate transmission rate of communication?
• Answer: the channel capacity, C (the maximum amount of
traffic that can move over a channel), is its rate limit.

• All communication schemes lie in between these two limits on

the compressibility of data and the capacity of a channel.
Information theory can suggest means to achieve these
theoretical limits.

10
 Historical Background

• Claude SHANNON (Father of the

information age or Information theory) in
1949 wrote his work in information theory:
"Mathematical Theory of Communication".

[Shannon, C. E. (1948). A mathematical theory of

communication. The Bell system technical journal,
27(3), 379-423]. Claude SHANNON
American mathematician
1916 - 2001

• He proved the famous Source Coding Theorem

and the Noisy Channel Coding Theorem, plus
many other related results about channel
capacity.
11
12
 Historical Background

• Source Coding Theorem (noiseless channel): Let a source

have entropy H (bits per symbol) and a channel have a
capacity C (bits per second). Then it is possible to encode
the output of the source in such a way as to transmit at the
average rate C/H–ε symbols per second over a channel
where ε is arbitrary small.
• It is not possible to transmit at an average rate greater than
C/H. (Shannon 1948)

13
 Historical Background

• Noisy Channel Coding Theorem:

Let a discrete channel have the capacity C and a discrete source
have the entropy per second H. If H≤C there exist a coding
system such that the output of the source can be transmitted over
the channel with an arbitrarily small frequency of errors. If H>C
it is possible to encode the source so that the error is less than
H–C+ε where ε is arbitrarily small. There is no method of
encoding which gives an error less than H–C. (Shannon 1948)

14
 Probability Rules
What are random variables? What is probability?
• Random variables are variables that take on values determined by
probability distributions.

• They may be discrete or continuous. For example, a stream of ASCII

encoded text characters in a transmitted message is a discrete random
variable, with a known probability distribution for any given natural
language.

• An analog speech signal represented by a voltage or sound pressure

waveform as a function of time, is a continuous random variable
having a continuous probability density function.

• Most of Information Theory involves probability distributions of

random variables, and conditional probabilities defined over
ensembles of random variables. Indeed, the information content of a symbol
or event is defined by its probability.
15
 Probability Rules

• Probability theory rests upon two rules:

1.

16
 Probability Rules
• Probability theory rests upon two rules:
2.

17
 Entropy

• The information content I of a single event or message is

defined as the base-2 logarithm of its probability p:

I = log2 p (1)

• And its Entropy H is considered the negative of this.

Entropy can be regarded as “measure the amount of
uncertainty of a random quantity”.
• To gain information is to lose uncertainty by the same
amount, so I and H differ only in sign:
H = −I (2)

• Entropy and information have units of bits.

18
 Entropy

• No information is gained (no uncertainty is lost) by the

appearance of an event or the receipt of a message that
was completely certain anyway (p = 1, so I = 0).

• The more improbable an event is, the more informative

it is; and so, the monotonic behavior of Eq. (1) seems
appropriate.

• But why the logarithm?

19
 Entropy
Why the Logarithm?
• The logarithmic measure is justified by the desire for
information to be additive. We want the algebra of our
measures to reflect the Probability Rules.

• When independent packets of information arrive, we would

like to say that the total information received is the sum
of the individual pieces. But the probabilities of
independent events multiply to give their combined
probabilities, and so we must take logarithms in order for
the joint probability of independent events or messages to
contribute additively to the information gained.

20
 Entropy

• In information theory we often wish to compute the base-2

logarithms of quantities (Binary data), but most calculators
only offer (base e=2.718) and decimal (base10) logarithms. So,
the following conversions are useful:

log2 X = 1.443 loge X = 3.322 log10 X

21
 Information Measure

Example of the Information Measure

• Returning to our example, if X is the random variable

which describes which apartment your friend lives in,
initially it can take on 32 values with equal probability
p(x)=1/32.

• Since I= log2 (1/32) = -5, the entropy of X is 5 bits.

After the neighbor tells you that he lives on the top
floor, the probability of X drops 24 of the 32 values and
becomes 1/8 for the other 8 equally probable values. The
entropy of X thus drops to 3 bits. The neighbor has therefore
conveyed 2 bits of information to you.
22
 Entropy of Ensembles
• We now move from considering the information content of a
single event or message, to that of an ensemble (group of items).
• An ensemble is the set of outcomes of one or more random
variables. The outcomes have probabilities attached to them. In
general, these probabilities are non-uniform, with event i
having probability pi, but they must sum to 1 because all
possible outcomes are included; hence they form a probability
distribution:
(3)

• The entropy of an ensemble is simply the average entropy of all

the elements in it. We can compute their average entropy by
weighting each of the (log pi) contributions by its probability pi:
(4)
23
Entropy of Ensembles

24
 Entropy of Ensembles

• Example: A source with bandwidth 4000 Hz is sampled at the

Nyquist rate. Assuming that the resulting sequence A= (-2, -1,
0, 1, 2) and with corresponding probabilities (1/2, 1/4, 1/8, 1/16,
1/16) determine the rate of the source in bits/sec.

• Solution:

• H(x)= -1/2log1/2 -1/4log1/4 -1/8log1/8 -1/16log1/16 -1/16log1/16

= 15/8 bits/sample.
• Since we have 2*4000= 8000 samples/sec the source produces
information at a rate of 8000*(15/8) = 15000 bits/sec.
25
 Joint Entropy
• Joint entropy of XY

(5)

• Note that in comparison with Eq. (4), we have replaced the

‘–’ sign in front by taking the reciprocal of p inside the
logarithm.
• From this definition, it follows that join entropy is additive if
X and Y are independent random variables:

H(X,Y) = H(X) + H(Y) if p(x,y) = p(x)p(y) (6)

 Conditional Entropy

• Conditional entropy of an ensemble X, given an ensemble Y:

(7)

• And we know from the Sum Rule that if we move the

p(y) term from the outer summation over y, to inside the
inner summation over x, the two probability terms combine
and become just p(x,y) summed over all x, y.
• Hence a simpler expression for this conditional entropy is:
(8)

This measures the average uncertainty that remains about X, when Y is known.27
 Chain Rule for Entropy

• The joint entropy, conditional entropy, and marginal

entropy for two ensembles X and Y are related by:

H(X,Y) = H(X) + H(Y|X) = H(Y ) + H(X|Y) (9)

• It should seem natural and intuitive that the joint entropy

of a pair of random variables is the entropy of one plus
the conditional entropy of the other.

28
 Entropy

• Example: Two binary random variables X and Y are

distributed according to the joint distribution
p(X=Y=0) = p(X=0,Y=1) = p(X=Y=1) = 1/3
Compute H(X), H(Y), H(X|Y), H(Y|X), and H(X, Y).

• Solution:

29
 Entropy

H(X,Y) = H(X) + H(Y|X) = H(Y ) + H(X|Y)

30
 Mutual Information

• The mutual information between two random variables measures

the amount of information that one conveys about the other.
Equivalently, it measures the average reduction in uncertainty
about X that results from learning about Y. It is defined:

(10)

• Note that in case X and Y are independent random variables,

then the numerator inside the logarithm equals the denominator.
Then the log term vanishes, and the mutual information equals
zero.

31
 Mutual Information

• Non-negativity: mutual information is always ≥ 0. In the event

that the two random variables are perfectly correlated, then
their mutual information is the entropy of either one alone.

• Another way to say this is: I(X;X)= H(X): the mutual

information of a random variable with itself is just its entropy.
For this reason, the entropy H(X) of a random variable X is
sometimes referred to as its self-information.)

• These properties are reflected in three equivalent definitions for

the mutual information between X and Y

32
 Mutual Information

• In a sense the mutual information I(X;Y) is the intersection

between H(X) and H(Y), since it represents their statistical
dependence.

33
 Venn Diagram

• In the Venn diagram, the portion of H(X) that does not lie
within I(X;Y) is just H(X|Y). The portion of H(Y) that
does not lie within I(X;Y) is just H(Y|X).

34
 Venn Diagram

Fig. Mutual information’s relationship with joint entropy and conditional entropy.
35
 Let’s Conclude

The relationships between entropy, joint entropy, conditional entropy and mutual
information The width of each bar represents its size (in bits).
36
 Let’s Conclude

I can only invite you to go further and learn

more. This is what’s commonly called open your
mind. I’m going to conclude with this, but in
Shannon’s language
Increase the “Entropy” of your thoughts
http://www.science4all.org/article/shannons-
information-theory/
https://www.theguardian.com/science/2014/jun
/22/shannon-information-theory
37
 Let’s Conclude
Without Claude Shannon's information theory there would have
been no internet

38
THANKS
☺

39