ECTE301: Digital Signal Processing

Lecture 5
Random Signals and Correlation Sequences
Part A’s contents
Textbook
Lecture Topic Practice
Chapter
Lab 1
L1 Introduction to DSP 1, 2.1, 2.2
Tutorial 1
Lab 2
L2 Convolution and LTI Systems 2.3, 2.4
Tutorial 2
The z-transform: Tutorial 3
L3 3.1, 3.2, 3.3
Forward Transform and Properties Lab 3
The z-transform:
L4 3.4, 3.5 Tutorial 4
Inverse Transform and Applications
2.4, 2.5,
Random signals and Tutorial 4
L5 2.6, 12.1
Cross-correlation Sequences Report

L6 Revision of Weeks 1 to 6 - -

Digital Signal Processing S. L. Phung & A. Bouzerdoum 2/64

Lecture 5’s sequence

5.1 Random signals

5.2 Correlation sequences

5.3 LTI system in correlation domain

5.4 Applications of correlation

Digital Signal Processing S. L. Phung & A. Bouzerdoum 3/64

Lecture 5's objectives

 Understand random signals, CDF, PDF.

 Define the cross-correlation and auto-correlation sequences.

 Study input-output relationships for LTI systems in ‘correlation domain’.

 Explore real-world signal processing applications of correlation (demo).

Digital Signal Processing S. L. Phung & A. Bouzerdoum 4/64

5.1 Random signals
 Most signals that occur in nature, e.g. speech, music, brain waves,
seismic signals, and noise, are random signals.

 Random signals cannot be predicted ahead of time, and do not have a

deterministic form.

 Consider a random variable X produced by a random process.

 Each observed value of X is called a realization of the random variable.

The observed value is denoted as x.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 5/64

Random signals
 Example 5.1: At a given time n, a coin is tossed. The value of the
random variable x(n) is
● x(n) = 1 if head,
● x(n) = 0 if tail.

Two realizations of a random process

obtained by tossing a coin.
Digital Signal Processing S. L. Phung & A. Bouzerdoum 6/64
Random signals

Two different realizations of the phrase ‘DSP’.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 7/64
5.1.1 CDF and PDF
 A random variable X is defined by its cumulative distribution function
(CDF):

 If X is a continuous variable, then its probability density function (PDF)

is defined as

 Therefore, the CDF can be expressed as

Digital Signal Processing S. L. Phung & A. Bouzerdoum 8/64

CDF and PDF
 Example 5.2: Consider a random variable X with mean 2 and standard
deviation 1. There are 2000 realizations of X.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 9/64

CDF and PDF

Random variable X: mean 2, std 1, and 2000 realizations.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 10/64
CDF and PDF

Random variable X: mean 2, std 1, and 2000 realizations.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 11/64
CDF and PDF

P(x0)

p(x0)

x0
CDF is the area under the curve (blue shaded region) of PDF.
Digital Signal Processing S. L. Phung & A. Bouzerdoum 12/64
CDF and PDF
 Properties of the probability density function:

 Properties of the cumulative probability function:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 13/64

5.1.2 Moments, Mean, and Variance

 The expectation operator is denotes as E[.].

 The expected value of a random variable X is defined as

 The expectation is a linear operator, i.e.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 14/64

Moments

 A random variable is completely characterized by all its moments.

 The r-th moment is defined as

 For example,

Digital Signal Processing S. L. Phung & A. Bouzerdoum 15/64

Moments: Application in Image Processing

-100

-50

0
(a) Original image
50

100

150

-200 -100 0 100

(b) Distorted image (c) A normalized image of (a) and (b)
under affine transform using image moments (video)
Digital Signal Processing S. L. Phung & A. Bouzerdoum 16/64
Mean and Variance
 The first-order moment, denoted by μ, is the mean or expected value:

 The second-order moment is the mean-square value:

 The variance of random variable X is defined as

 σ is the standard deviation of X.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 17/64

5.1.3 Common random signals

We study two common random signals with

 Uniform distribution

 Normal distribution

Digital Signal Processing S. L. Phung & A. Bouzerdoum 18/64

a) Uniform distribution
 A random variable X is uniformly distributed in interval [a, b] if its PDF
has the form

 For the uniform distribution, the mean and the variance are

Digital Signal Processing S. L. Phung & A. Bouzerdoum 19/64

a) Uniform distribution

-1 1 3 -1 1 3

PDF and CDF of a uniform distribution in [-1, 3]

Digital Signal Processing S. L. Phung & A. Bouzerdoum 20/64

b) Normal distribution

 A random variable X is normally distributed if its PDF has the form:

 The normal distribution is also called the Gaussian distribution.

 For the normal distribution, the mean and the variance are

 Normal distribution was illustrated in Example 5.2.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 21/64

b) Normal distribution

Carl Friedrich Gauss (1777-1855)

French Mathematician
https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss

Digital Signal Processing S. L. Phung & A. Bouzerdoum 22/64

Normal distribution

PDF and CDF of a normal distribution

Digital Signal Processing S. L. Phung & A. Bouzerdoum 23/64

5.1.4 Independent random variables
 Two random variables X and Y are statistically independent if their joint
PDF satisfies:

 Informally, the probability of observing {X = x AND Y = y} is equal to:

{probability of observing X = x } ×{probability of observing Y = y}.

 The variance of the sum of two independent random variables is the

sum of their variances:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 24/64

Uncorrelated random variables
 Two random variable X and Y are said to be uncorrelated if

 That is,
{expected value of XY} = {expected value of X} × {expected value of Y}.

 If two variables are independent, then they are uncorrelated.

 If two variables are uncorrelated, they are not necessarily independent.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 25/64

5.1.5 Random sequences
 For a random sequence {X(n)}, each sample X(n) is considered as a
random variable.
 Example 5.3: Consider the following random sequence {X(n)}
X(n) = 2 cos(2πfn) + w(n), where f = 0.05 and w(n) is Gaussian noise.

video

Digital Signal Processing S. L. Phung & A. Bouzerdoum 26/64

Random sequences
 A random sequence is characterized by its PDF and CDF:

 For a random sequence, the mean, mean-square, and variance are time-
varying.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 27/64

Lecture 5’s sequence

5.1 Random signals

5.2 Correlation sequences

5.3 LTI system in correlation domain

5.4 Applications of correlation

Digital Signal Processing S. L. Phung & A. Bouzerdoum 28/64

5.2 Correlation sequences
 The cross-correlation of two energy signals x(n) and y(n) is defined as

 The auto-correlation sequence of x(n) is defined as

 Index l is the time-shift (or lag) parameter.

 The difference between the first and second index is equal to the lag:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 29/64

Correlation sequences
 Following are all valid expressions of the cross-correlation:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 30/64

Correlation sequences
 Example 5.4: Cross-correlation of two sequences.

rxy(5) = 5

ryx(l) = rxy(-l)

Digital Signal Processing S. L. Phung & A. Bouzerdoum 31/64

Correlation sequences
 Example 5.5: Calculate the auto-correlation of x(n):

Answer:

After variable substitution &

overlap processing

Digital Signal Processing S. L. Phung & A. Bouzerdoum 32/64

Correlation sequences

Auto-correlation of signal
Digital Signal Processing S. L. Phung & A. Bouzerdoum 33/64
Properties of correlation sequences
 Important properties of the correlation sequences:

used many
times later

 The energy of a sequence:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 34/64

Normalized correlation sequences

 The correlation sequences are often normalized to the range [-1, 1].

 The normalized correlation sequences are defined as

Digital Signal Processing S. L. Phung & A. Bouzerdoum 35/64

Correlation of random signals
 The cross-correlation of two real random signals X(n) and Y(n) is

 The auto-correlation of X(n) is

 For stationary random signals, the correlation depends only on the lag l ,
where l = (m – n):

Digital Signal Processing S. L. Phung & A. Bouzerdoum 36/64

Covariance and auto-covariance sequences
 The covariance and auto-covariance sequences are given by

 Therefore, the variance of a stationary random process is

 The mean-square value is given by

 The average power of a wide-sense stationary random signal is given by

Digital Signal Processing S. L. Phung & A. Bouzerdoum 37/64

Correlation of random signals
 For WSS signals:

 Other useful properties:

 For a WSS signal with no periodic components:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 38/64

Correlation of random signals
 Example 5.6: Find the mean, auto-correlation, auto-covariance,
variance, and average power of the sinusoidal random signal

A and ω0 are constants. Φ is a uniformly distributed RV in [0, π].

Answer:

 We have:

 Mean:

 Auto-correlation:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 39/64

Correlation of random signals
 X(n) is a WSS signal because rxx(m, n) depends only on m-n.

 Let l = m-n, the auto-correlation sequence is

 The auto-covariance sequence is

 Thus, the variance and average power of the signal are

Digital Signal Processing S. L. Phung & A. Bouzerdoum 40/64

Lecture 5’s sequence

5.1 Random signals

5.2 Correlation sequences

5.3 LTI system in correlation domain

5.4 Applications of correlation

Digital Signal Processing S. L. Phung & A. Bouzerdoum 41/64

5.3 LTI system in correlation domain
 Consider a LTI system with impulse response h(n).

h(n) Notes

 The following properties can be obtained.

h(l)

h(l)

rhh(l)

Digital Signal Processing S. L. Phung & A. Bouzerdoum 42/64

Input-output correlation sequences
 Consider a system described by a difference equation:

 The following properties are obtained:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 43/64

Input-output correlation sequences
 Example 5.7: Find ryx(l) and ryy(l) for the following system

Answer:
 The impulse response is

 Applying the correlation properties:

 These correlation sequences satisfy the different equations:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 44/64

Input-output correlation sequences

Input-output correlations of a LTI system driven by white noise

Digital Signal Processing S. L. Phung & A. Bouzerdoum 45/64

Lecture 5’s sequence

5.1 Random signals

5.2 Correlation sequences

5.3 LTI system in correlation domain

5.4 Applications of correlation

Digital Signal Processing S. L. Phung & A. Bouzerdoum 46/64

5.4 Applications of correlation
Correlation has applications in many areas, including
 pattern matching,
 filtering, prediction, estimation,
 signal modeling and synthesis,
 spectrum estimation,
 detection of hidden periodicities,
 detection of signals buried in noise,
 time-delay (arrival time) estimation, and
 system identification.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 47/64

Applications of correlation

5.4.1 Detection of Hidden Periodicities

5.4.2 Detection of Signal in Noise

5.4.3 System Identification

5.4.4 Image matching

Digital Signal Processing S. L. Phung & A. Bouzerdoum 48/64

5.4.1 Detection of hidden periodicities
 Consider a periodic signal s(n) with a period of N:

 The signal is mixed with noise, represented by zero-mean Gaussian

random signal ω(n).

 We can observe only the noisy signal x(n) instead of s(n):

How to find period N from observed signal x(n)?

Digital Signal Processing S. L. Phung & A. Bouzerdoum 49/64

Detection of hidden periodicities

 We calculate the auto-correlation sequence of x(n):

 Because ω(n) is a zero-mean white noise signal:

 Therefore,

 Except for the sample at l = 0, rxx(l) has a period of N.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 50/64

Detection of hidden periodicities
 Example 5.8:

It is difficult to estimate the period of noisy signal: x(n) = s(n) + noise

Digital Signal Processing S. L. Phung & A. Bouzerdoum 51/64
Detection of hidden periodicities

It is easier to estimate the period from auto-correlation rxx(l): N = 40

Digital Signal Processing S. L. Phung & A. Bouzerdoum 52/64
Detection of hidden periodicities
MATLAB code
f = 1/40; N = floor(1/f); L = 4* N;
n = 0:(L-1);
s = 1.25 * cos(2 * pi * f * n);
w = 1 * randn(1, L);
x = s + w;
[rxx, l] = xcorr(x);
[rss, l] = xcorr(s);

figure(1)
subplot(1,2,1); plot(n, s);
subplot(1,2,2); plot(n, x, 'r’);

figure(2)
subplot(1,2,1); plot(l, rss, ‘b’);
subplot(1,2,2); plot(l, rxx, ‘r’);

Digital Signal Processing S. L. Phung & A. Bouzerdoum 53/64

5.4.2 Detection of signal in noise
 An original signal s(n) is transmitted via a communication channel.

 The received signal x(n) is a delayed and noise-corrupted version of s(n):

 The task is to detect if s(n) is present in x(n).

 By the correlation property, we have:

Digital Signal Processing S. L. Phung & A. Bouzerdoum 54/64

Detection of signal in noise

Correlation-based signal detector

Digital Signal Processing S. L. Phung & A. Bouzerdoum 55/64

Matched filter
 The correlation detector can be implemented using the filter with the
following impulse response:

 The filter's output at time (L-1) :

Matched filter detector

Digital Signal Processing S. L. Phung & A. Bouzerdoum 56/64
Matched filter
 Example 5.9: Detecting a signal using a matched filter.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 57/64

5.4.3 System identification
 How to find the impulse response of an unknown LTI system?

Approach:

 Give the system a white-noise input signal x(n).

 Measure the system's output y(n).

 Calculate the ryx(l). It is equal to h(l).

Digital Signal Processing S. L. Phung & A. Bouzerdoum 58/64

System identification

System identification by cross-correlation

Digital Signal Processing S. L. Phung & A. Bouzerdoum 59/64

System identification
 Example 5.10: Consider an unknown LTI system with

Digital Signal Processing S. L. Phung & A. Bouzerdoum 60/64

5.4.4 Image matching – Face detection

 Example 5.11: Find the location of an object (e.g. a face) in an image.

 A common approach is to scan every rectangular region of the image,
and determine if each region resembles a face.
Digital Signal Processing S. L. Phung & A. Bouzerdoum 61/64
Face detection

First, build a face

template, by averaging
14,000 faces.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 62/64

Face detection

A rectangular region and the face template can be compared

using their correlation coefficient (demo video)
Digital Signal Processing S. L. Phung & A. Bouzerdoum 63/64
Lecture 5’s summary

To revise for this lecture:

 How to compute PDF, CDF, moments, mean, mean-square, variance,

power of a random variable.

 How to compute auto-correlation and cross-correlation sequences.

 How to compute covariance sequences of a random signal.

 How to use correlation sequences in applications.

 Practice: Tutorial 4, Lab 3.

Digital Signal Processing S. L. Phung & A. Bouzerdoum 64/64