Statistical Digital Signal

Processing
Chapter 1: Introduction and Review
of Stochastic Process
Objective of the Course
• To enable students analyze, represent and
manipulate random signals with LTI systems.
– Understand challenges posed by random
signals,
– Understand how to model random signals,
– Understand how to represent random signals,
– Understand how to design LTI system to
estimate random signals,
– Understand efficient algorithms to estimate
random signals.
Bisrat Derebssa, SECE, AAiT, AAU 2
Content of the course
1. Introduction and Review of Stochastic Process
2. Linear Signal Modelling
3. Nonparametric Power Spectrum Estimation
4. Optimum Linear Estimation of Signals
5. Algorithms for Optimum Linear Filter
6. Adaptive Filters

Bisrat Derebssa, SECE, AAiT, AAU 3

References
• Statistical Digital Signal Processing and
Modeling, M. Hayes, Wiley, 1996.
• Statistical and Adaptive signal Processing,
Dimitris G. Manolakis, Vinay K. Ingle,
Stephen M. Kogon, Artech House, 2005
• Optimum Signal Processing, Sophocles J.
Orfanidis, McGraw-Hill, 2007

Bisrat Derebssa, SECE, AAiT, AAU 4

Evaluation
• Assignment and project (50%)
• Final Exam (50%)

Bisrat Derebssa, SECE, AAiT, AAU 5

Random Variables
• Any random situation can be studied by the
axiomatic definitions of probability by
defining 𝑆, ℱ, 𝑃𝑟 .
– 𝑆 = {𝜁1 , 𝜁2 , … } – Universal set of unpredictable
outcomes
– ℱ – collection of subset of 𝑆 whose elements are called
events.
– Pr 𝜁𝑘 , 𝑘 = 1,2, . . . – probability representing the
unpredictability of these events.

Bisrat Derebssa, SECE, AAiT, AAU 6

• Difficult to work with this probability space
for two reasons.
–The basic space contains abstract events and
outcomes that are difficult to manipulate.
• We want random outcomes that can be measured and
manipulated in a meaningful way by using numerical
operations.
–The probability function Pr {·} is a set function
that again is difficult to manipulate by using
calculus.

Bisrat Derebssa, SECE, AAiT, AAU 7

• A random variable x(ζ) is a mapping that assigns a
real number x to every outcome ζ from an abstract
probability space.
• A complex valued random
variable represented as
𝑥(𝜁) = 𝑥𝑅 (𝜁 ) + 𝑗𝑥𝐼 (𝜁

Bisrat Derebssa, SECE, AAiT, AAU 8

•This mapping should satisfy the following two
conditions:
–the interval {X(ζ) ≤ x } is an event in the abstract
probability space for every x ;
–Pr {X(ζ) =∞}=0 and Pr{X(ζ) = −∞} =0.
• A random variable is called discrete-valued if
x takes a discrete set of values {xk};
• Otherwise, it is termed a continuous-valued
random variable.

Bisrat Derebssa, SECE, AAiT, AAU 9

Representation of Random Variables
Cumulative distribution
𝐹𝑋 𝑥 = 𝑃𝑟{𝑋(𝜁) ≤ 𝑥}
function (CDF)
Probability density function 𝑑𝐹𝑋 𝑥
(pdf ) 𝑓𝑋 𝑥 =
𝑑𝑥

෍ 𝑥𝑘 𝑝𝑥
Expectation of a random 𝑥
𝐸 𝑥 𝜁 = 𝜇𝑥 = ∞
variable
න 𝑥𝑓𝑋 𝑥 𝑑𝑥
−∞
∞
Expectation of a function
𝐸 𝑔𝑥 𝜁 = න 𝑔(𝑥)𝑓𝑋 𝑥 𝑑𝑥
of random variable −∞
∞
(𝑚)
Moments 𝑟𝑥 = 𝐸 𝑥𝑚 𝜁 = න 𝑥 𝑚 𝑓𝑋 𝑥 𝑑𝑥
−∞
∞
(𝑚) 𝑚 𝑚𝑓
𝛾𝑥 = 𝐸 𝑥 𝜁 − 𝜇𝑥 =න 𝑥 − 𝜇𝑥 𝑋 𝑥 𝑑𝑥
Central Moments −∞

Bisrat Derebssa, SECE, AAiT, AAU 10

Second central moment or (2)
variance 𝜎𝑥2 = 𝛾𝑥 = 𝐸 𝑥 𝜁 − 𝜇𝑥 2

(3) 1 (3)
Skewness 𝑘𝑥 = 3 𝛾𝑥
𝜎𝑥
~(4) 1 (4)
Kurtosis 𝑘𝑥 = 4 𝛾𝑥 − 3
𝜎𝑥
∞
Characteristic functions 𝚽𝑥 𝜉 = 𝐸 𝑒 𝑗𝜉𝑥 𝜁 = න 𝑓𝑋 𝑥 𝑒 𝑗𝜉𝑥 𝑑𝑥
−∞
∞
ഥ 𝒙 𝑠 = 𝐸 𝑒 𝑠𝑥
𝚽 𝜁
= න 𝑓𝑋 𝑥 𝑒 𝑠𝑥 𝑑𝑥
Moment generating functions −∞

Cumulants generating functions ഥ 𝑥 𝑠 = ln 𝚽

𝚿 ഥ 𝑥 𝑠 = ln 𝐸 𝑒 𝑠𝑥 𝜁

(𝑚)
ഥ𝑥 𝑠 ]
d𝑚 [𝚿
Cumulants 𝑘𝑥 = อ
d𝑠 𝑚
𝑠=0
Bisrat Derebssa, SECE, AAiT, AAU 11
Useful Random Variables
1
Uniformly distributed RV 𝑓𝑥 𝑥 = ቐ𝑏 − 𝑎 𝑎≤𝑥≤𝑏
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
1 1 𝑥−𝜇 2
−2 𝜎
Normal RV 𝑓𝑥 𝑥 = 𝑒 𝑥
2𝜋𝜎𝑥2
𝛽 1
Cauchy RV 𝑓𝑥 𝑥 =
𝜋 𝑥 − 𝜇 2 + 𝛽2

Bisrat Derebssa, SECE, AAiT, AAU 12

 Random Vectors
• A real-valued random vector containing M random
variables is represented as:
𝐗 𝜁 = [𝑥1 𝜁 , 𝑥2 𝜁 , … , 𝑥𝑀 𝜁 ]𝑇
• A random vector is completely characterized by its
joint CDF
𝐹𝐱 𝑥1 , 𝑥2 , … , 𝑥𝑀 = Pr 𝑋1 𝜁 ≤ 𝑥1 , 𝑋2 𝜁 ≤ 𝑥2 , … , 𝑋𝑀 𝜁 ≤ 𝑥𝑀
• Often written as
𝐹𝐗 𝒙 = Pr 𝐗 𝜁 ≤ 𝒙
• Two random variables 𝑋1(𝜁 ) and 𝑋2(𝜁 ) are
independent if the events {𝑋1(𝜁 ) ≤ 𝑥1} and {𝑋(𝜁 ) ≤
𝑥2} are jointly independent. That is,
Pr 𝑋1 𝜁 ≤ 𝑥1 , 𝑋2 𝜁 , ≤ 𝑥2 = Pr 𝑋1 𝜁 ≤ 𝑥1 Pr 𝑋2 𝜁 , ≤ 𝑥2 13
Bisrat Derebssa, SECE, AAiT, AAU
• The probability functions require an
enormous amount of information that is not
easy to obtain or is too complex
mathematically for practical use.
• In practical applications, random vectors are
described by less complete but more
manageable statistical averages.

Bisrat Derebssa, SECE, AAiT, AAU 14

Statistical Description of Random
Vector 𝜇 𝐸 𝑥1 𝜁 1
Mean vector 𝐸 𝐱 𝜁 = 𝝁𝐱 = ⋮ = ⋮
𝐸 𝑥𝑀 𝜁 𝜇𝑀
𝑟11 … 𝑟1𝑀
𝐑 𝐱 = 𝐸 𝐱 𝜁 𝐱𝑯 𝜁 = ⋮ ⋱ ⋮
Autocorrelation matrix 𝑟𝑀1 … 𝑟𝑀𝑀

𝛾11 … 𝛾1𝑀
𝑯
𝚪𝐱 = 𝐸 𝐱 𝜁 − 𝝁𝐱 𝐱 𝜁 − 𝝁𝐱 = ⋮ ⋱ ⋮
Autocovariace matrix 𝛾𝑀1 … 𝛾𝑀𝑀

𝚪𝐱 = 𝐑 𝐱 − 𝝁𝐱 𝝁𝐱 𝐻
𝛾𝑖𝑗
Correlation coefficient 𝜌𝑖𝑗 = = 𝜌𝑗𝑖
𝜎𝑖 𝜎𝑗
Uncorrelatedness 𝛾𝑖𝑗 = 0, for 𝑖 ≠ 𝑗

Orthogonal 𝑟𝑖𝑗 = 0, for 𝑖 ≠ 𝑗

Bisrat Derebssa, SECE, AAiT, AAU 15
Statistical Description of Two
Random Vectors
𝐸 𝑥1 𝜁 𝑦1∗ 𝜁 … 𝐸 𝑥1 𝜁 𝑦𝐿∗ 𝜁
Cross-correlation
𝐑 𝐱𝐲 = 𝐸 𝐱 𝜁 𝐲 𝑯 𝜁 = ⋮ ⋱ ⋮
matrix
𝐸 𝑥𝑀 𝜁 𝑦1∗ 𝜁 … 𝐸 𝑥𝑀 𝜁 𝑦𝐿∗ 𝜁

Cross-covariance 𝑯
𝚪𝐱𝑦 = 𝐸 𝐱 𝜁 − 𝝁𝐱 𝐲 𝜁 − 𝝁𝐲 = 𝐑 𝐱𝐲 − 𝝁𝐱 𝝁𝐲 𝐻
matrix

Uncorrelated 𝚪𝐱𝑦 = 𝟎 → 𝐑 𝐱𝐲 = 𝝁𝐱 𝝁𝐲 𝐻

Orthogonal 𝐑 𝐱𝐲 = 𝟎

Bisrat Derebssa, SECE, AAiT, AAU 16

Linear Transformations of Random
Vector
• Linear transformations are relatively simple
mappings and are given by the matrix operation
𝐲 𝜁 = 𝐀𝐱(𝜁), A is an LxM matrix and x is M dimensional vector
• Assuming L=M, and both are real valued.
𝑓𝑥 (𝐀−1 𝐲)
𝑓𝐲 𝐲 =
det 𝑨
• If L>M, only M random variables 𝐲𝐢 𝜁 can be
independently determined from 𝐱(𝜁),
– The remaining L-M can be obtained from the first 𝐲𝐢 𝜁
• If L<M, we can augment 𝐲 𝜁 into an M-vector by
introducing auxiliary random variables.
Bisrat Derebssa, SECE, AAiT, AAU 17
• For complex valued RV,
𝑓𝑥 (𝐀−1 𝐲)
𝑓𝐲 𝐲 =
det 𝑨 2
• The determination of 𝑓𝐲 𝐲 is tedious and in
practice not necessary.

Bisrat Derebssa, SECE, AAiT, AAU 18

 Statistical Description of Linear
Transformation of Random Vector
Mean vector 𝝁𝐲 = 𝐸 𝐲 𝜁 = 𝐸 𝐀𝐱 𝜁 = 𝐀𝐸 𝐱 𝜁 = 𝐀𝝁𝐱

Autocorrelation matrix 𝐑 𝐲 = 𝐸 𝐲𝒚𝑯 = 𝐸 𝐀𝐱𝐱 𝐻 𝑨𝐻 =𝐀𝐸 𝐱𝐱 𝐻 𝑨𝐻 =𝐀𝐑 𝐱 𝑨𝐻

Autocovariace matrix 𝚪𝐲 = 𝐀𝚪𝐱 𝑨𝐻

𝐑 𝐱𝐲 = 𝐸 𝐱𝒚𝑯 = 𝐸 𝐱𝐱 𝐻 𝑨𝐻 =𝐸 𝐱𝐱 𝐻 𝑨𝐻 =𝐑 𝐱 𝑨𝐻
Cross-correlation
𝐑 𝐲𝐱 = 𝐀𝐑 𝐱

𝚪𝐱𝐲 = 𝚪𝐱 𝑨𝐻
Cross-covariance
𝚪𝐲𝐱 = 𝐀𝚪𝐱

Bisrat Derebssa, SECE, AAiT, AAU 19

Normal Random Vectors
• If the components of the random vector x (ζ ) are
jointly normal, then x (ζ ) is a normal random M -
vector.
• For real valued normal random vector
1 1
− 𝐱−𝛍𝐱 𝑇 𝚪𝐱 −1 𝐱−𝛍𝐱
𝑓𝐱 𝐱 = 𝑀ൗ 1ൗ 𝑒 2
(2𝜋) 2 𝚪𝐱 2
• Its characteristic equation is
1
𝑗𝝃𝑇 𝝁𝐱 − 𝝃𝑇 𝚪𝐱 𝝃
Φ𝐱 (𝝃) = 𝑒 2

Bisrat Derebssa, SECE, AAiT, AAU 20

Properties of normal random vector
• Pdf and all higher order moments
completely specified from mean vector and
covariance matrix.
• If the components of 𝐱 𝜁 are mutually
uncorrelated, they are also independent.
• A linear transformation of a normal random
vector is also normal.

Bisrat Derebssa, SECE, AAiT, AAU 21

Sum of Independent Random
Variables
• If a random variable is a linear combination
of M statistically independent random
variables, the pdf and statistical descriptors
are easy.
𝑦 = 𝑐1 𝑥1 + 𝑐2 𝑥2 + ⋯ + 𝑐𝑀 𝑥𝑀
𝑀
Mean 𝜇𝑦 = ෍ 𝑐𝑘 𝜇𝑥 𝑘
𝑘=1
2
𝑀 𝑀

Variance 𝜎𝑦2 = 𝐸 ෍ 𝑐𝑘 𝑥𝑘 − 𝜇𝑥 𝑘 = ෍ 𝑐𝑘 2 𝜎𝑥2𝑘

𝑘=1 𝑘=1

Probability density
𝑓 𝑦 =𝑓 𝑦 ∗ 𝑓𝑥2 𝑦 ∗ ⋯ ∗ 𝑓𝑥𝑀 𝑦
function Bisrat Derebssa,𝑦SECE, AAiT,
𝑥1AAU 22
• Example: What is the pdf of y if its is the sum of
four identical independent random variables
uniformly distributed over [-0.5, 0.5].
• Solution:
U[-0.5, 0.5]*U[-0.5, 0.5]= fx12 fx12*U[-0.5, 0.5]=fx123 fx123*U[-0.5, 0.5]=fx1234

Bisrat Derebssa, SECE, AAiT, AAU 23

Conditional Density
• Provides a measure of the degree of
dependence of the variables on each other.
• From Bayes' rule, the joint pdf is given as
𝑃 𝑥1 , 𝑥2 = 𝑃 𝑥1 |𝑥2 𝑃 𝑥2 = 𝑃 𝑥2 |𝑥1 𝑃 𝑥1
𝑃 𝑥2 |𝑥1 𝑃 𝑥1
𝑃 𝑥1 |𝑥2 =
𝑃 𝑥2
• If they are independent
𝑃 𝑥1 |𝑥2 = 𝑃 𝑥1

Bisrat Derebssa, SECE, AAiT, AAU 24

Ensemble Averages
• A discrete-time random process is a
sequence of random variables, x(n)
• The mean of the process, mean of each of
these random variables may be calculated as
𝑚𝑥 𝑛 = 𝐸 𝑥(𝑛)
• The variance is
𝜎𝑥2 𝑛 = 𝐸 𝑥 𝑛 − 𝑚𝑥 𝑛 2
• These are ensemble averages.

Bisrat Derebssa, SECE, AAiT, AAU 25

• The autocorrelation of the process is
𝑟𝑥 𝑘, 𝑙 = 𝐸 𝑥 𝑘 𝑥 ∗ (𝑙)
• This provides the statistical relationship
between the random variables x(k) and x(l).
• Wide-sense stationary
– Mean of process is constant,
– Autocorrelation dependent only on (k-l)
– Variance is finite

Bisrat Derebssa, SECE, AAiT, AAU 26

Ergodicity
• The mean and autocorrelation of a random
process are obtained from actual observed
data instead of from probability density
function.
• If a large number of observations is available

• Since the sample mean is average of random

variables, it is itself a random variable.

Bisrat Derebssa, SECE, AAiT, AAU 27

• If the ensemble statistic approaches the
actual statistic, it is called unbiased
estimator.

• If the variance of the estimator is very small

it is called consistent estimator.

• If both are satisfied, it is ergodic to the

mean.
Bisrat Derebssa, SECE, AAiT, AAU 28
• This ergodicity principle may be generalized
to other ensemble averages.

Bisrat Derebssa, SECE, AAiT, AAU 29

Random Processes through Linear
Time-invariant Systems
• Consider a linear time-invariant system with
impulse response h(t) driven by a random
process input X(t)

Bisrat Derebssa, SECE, AAiT, AAU 30

• It is difficult to obtain a complete
specification of the output process in
general,
– The input is known only probabilistically.
• The mean and autocorrelation of the output
can be determined in terms of the mean and
autocorrelation of the input.

Bisrat Derebssa, SECE, AAiT, AAU 31

• The mean of the output is

• If the input is WSS

• Note that mean of output is not function of

time.
Bisrat Derebssa, SECE, AAiT, AAU 32
• The cross correlation between X and Y

• The autocorrelation of the output is

Bisrat Derebssa, SECE, AAiT, AAU 33

Power Spectrum
• The Fourier transform is important in the
representation of random processes.
• Since random signals are only known
probabilistically, it is not possible to
compute the Fourier transform directly.
• For a wide-sense stationary random process,
the autocorrelation is a deterministic
function of time.

Bisrat Derebssa, SECE, AAiT, AAU 34

• The periodogram is an estimation of the
power spectrum

• The autocorrelation sequence can be

obtained from the periodogram

Bisrat Derebssa, SECE, AAiT, AAU 35

• Properties of the periodogram
– It is real valued and symmetric,
– It is non-negative,
– The power in a zero-mean WSS process is
proportional to the area under the curve of the
PSD

Bisrat Derebssa, SECE, AAiT, AAU 36

Spectral Factorization
• The power spectrum evaluated by the z-
transform

• The power spectrum of a WSS process

maybe factorized as

• Q(z) is a minimum phase

– All poles and zeros of Q(z) are inside the unit
circule.

Bisrat Derebssa, SECE, AAiT, AAU 37

• From this representation

– Any regular random process may be realized as

the output of a causal stable filter driven by
white noise.
– The inverse filter 1/Q(z) can be seen as a
whitening filter.
– The inverse filter retains all the information of
x(n).

Bisrat Derebssa, SECE, AAiT, AAU 38

• For a rational P(z), the spectral factorization
is

• Where both A(z) and B(z) are polynomials

with roots inside the unit circle

Bisrat Derebssa, SECE, AAiT, AAU 39

• This is due to the symmetric property of
PSD.

Bisrat Derebssa, SECE, AAiT, AAU 40

Assignment 1
• 1.1 Show that if the mth derivative of the moment
generating function with respect to 𝑠 evaluated at 𝑠 =0
results in the mth moment.
• 1.2 Find the mean, variance, moments and moment
generating functions of Uniform, Normal and Cauchy RV.
• 1.3 Show that a linear transformation of a normal random
vector is also normal.
• 1.4 Find the spectral factorization of the following function.

Bisrat Derebssa, SECE, AAiT, AAU 41

• 1.5 The input to a linear shift-invariant filter with unit sample
response h(n) is a zero-mean wide-sense stationary
processes with autocorrelation rx(k). Find the
autocorrelation of the output processes for all k and its
variance.
1 1
ℎ 𝑛 =𝛿 𝑛 − 𝛿 𝑛−1 + 𝛿 𝑛−2
3 4
|𝑘|
1
𝑟𝑥 𝑘 =
2

Bisrat Derebssa, SECE, AAiT, AAU 42

• Submission format
– Via Email: [email protected]
– Email title: SDSP_[ID]_Assignment1
– File format: SDSP_[ID]_Assignment1.pdf
• pdf only
April 15, 2022
– Submission date: 5PM, September 14 , 2020
– Individual submission

Bisrat Derebssa, SECE, AAiT, AAU 43