ELEC5300 Lecture 3

 Stationary and Wide-Sense Stationary RPs

 Properties of the autocorrelation of WSS RPs

 Ergodicity
 Multiple Random Processes
 Cross correlation function

 Power Spectral Density

 Cross Power Spectral Density
 Important Random Processes
 Continuous Time White Noise
 Bandlimited White Noise
 Gauss-Markov Process
 Random Telegraph Signal
 Wiener Process

ELEC5300 Lecture 3 1
Stationary Random Processes
Definition: A process is stationary if the joint distribution of any set of samples does n
depend on the placement of the time origin, i.e.,
F ( x1 , x2 ,..., xk , t1 , t 2 ,...,t k )  F ( x1 , x2 ,..., xk ; t1  , t 2  ,...,t k   )
for all x1 , x2 ,..., xk , t1 , t 2 ,...,t k and .

 For example, F ( x , x ; t , t )  F ( x , x ; t  , t  )
1 2 1 2 1 2 1 2
 Stationarity is like time invariance.

ELEC5300 Lecture 3 2
Mean and Variance of Stationary RPs are Constant
Proof:
If X(t) is stationary, then f(x;t) = f(x), i.e., the first order density is
the same for all time samples.

 
Thus, m X (t )  x  f ( x; t ) dx  x  f ( x)dx  m


VAR[ X (t )]  ( x  m) 2 f ( x; t )dx   2

ELEC5300 Lecture 3 3
Autocorrelation of a stationary process
The autocorrelation and autocovariance of a stationary process
depend only upon the time difference t2 – t1.
Proof:
RX (t1 , t2 )   x1 x2 f ( x1 , x2 ; t1 , t2 )dx1dx2

  x1 x2 f ( x1 , x2 ; t1  , t2  )dx1dx2  RX (t1  , t2   )

We often abbreviate this by denoting the autocorrelation of a

stationary process by RX(t), where t = t2 – t1.
C X (t1 , t 2 )  RX (t1 , t 2 )  mt1 mt 2   RX (t 2  t1 )  m 2

Definition: A process X(t) is wide sense stationary (WSS) if

and only if its mean is constant and its correlation function
RX(t1,t2) depends only upon t2 – t1.
ELEC5300 Lecture 3 4
Properties of the Autocorrelation of a WSS Process
. Suppose X(t) is WSS with autocorrelation R (t) where t = t 2 – t 1.
X

RX    E[ X (t ) X (t   )]

 RX(0) is the average power of the process, E[X(t)2].

 RX(t) is an even function of t.
 |RX(t)| ≤ RX(0)
Proof:
0  E[ X (t )  X (t   )  ]  E[ X (t ) 2  2 X (t ) X (t   )  X (t   ) 2 ]
2

 E[ X (t ) 2 ]  2 E[ X (t ) X (t   )]  E[ X (t   ) 2 ]
 2 RX  0)  2 RX ( )
 If X(t) is Gaussian, it is also stationary.

ELEC5300 Lecture 3 5
 Properties of the Autocorrelation of a WSS Process
 The slower the autocorrelation delays as t → ∞, the slower the
realizations of the process change.
Proof:
P X (t   )  X (t )     P[| X (t   )  X (t ) |2   2 ]
E[( X (t   )  X (t )) 2 ] 2( R X (0)  R X ( ))
 2

 2

Sample Realizations Autocorrelation Sample Realizations

2 1 2
0 0
-2 0.8 -2
0 10 20 30 40 50 0 10 20 30 40 50
2 0.6 2
0 0
-2 0.4 -2
0 10 20 30 40 50 0 10 20 30 40 50
2 0.2
2
0 0
-2 -2
0
t
t

0 10 20 30 40 50 -20 -10 0 10 20 0 10 20 30 40 50

time time
ELEC5300 Lecture 3 6
Example: Random Bias
 Suppose x is chosen at random from the interval [0,1].
 Define X(n)= x for all n ∈0,1,…
 Since

RX (n1 , n2 )  E[ X (n1 ) X (n2 )]  E[ 2 ]  1 / 3

X(n) is wide sense stationary with constant autocorrelation function.

ELEC5300 Lecture 3 7
Example: Discrete time Gaussian white noise
 Let X(n) be an i.i.d. random process whose marginal distributions are
Gaussian with zero mean and variance s2.
 The correlation function is
2
2  if k  0
RX (n, n  k )  E[ X (n) X (n  k )]    (k )  
0 otherwise
 RX ( k )

 Since the process is zero mean,

C X ( k )  RX ( k )

RX ( k )
2

k
ELEC5300 Lecture 3 8
Ergodicity
 A RP is ergodic if time averaging is equivalent to ensemble
averaging (i.e. taking expectation).
 For example, for an ergodic process:

ELEC5300 Lecture 3 9
Example of a non-ergodic process (random bias)

ELEC5300 Lecture 3 10
Multiple Random Processes
 When discussing
more than one
random process,
e.g. X(t) and Y(t),
remember that both
are determined by
the same outcome
in the original
sample space.

ELEC5300 Lecture 3 11
Definitions of Terms  X (t1 )  Y ( s1 ) 
 X (t )  Y ( s ) 
 Two random processes X(t) and Y(t) are said  2   2 
 
to be independent if the vector random variables    and   
 
 X (t )
k  Y ( s j )
are independent for all k, j, and all choices of times t1,t2,…,tk and s1, s2,…,sj.

 The cross-correlation of X(t) and Y(t) is RXY (t1 , t 2 )  E[ X (t1 )Y (t 2 )]

 X(t) and Y(t) are orthogonal if R XY (t1 , t 2 )  0 for all t1 , t 2

 The cross-covariance of X(t) and Y(t) is

C XY (t1 , t 2 )  E[( X (t1 )  m X (t1 ))(Y (t 2 )  mY (t 2 ))]

 RXY (t1 , t 2 )  m X (t1 )mY (t 2 )
 X(t) and Y(t) are uncorrelated if C XY (t1 , t 2 )  0 for all t1 , t 2

ELEC5300 Lecture 3 12
Jointly Stationary Processes
Let X(t) and Y(t) be random processes.

 X(t) and Y(t) are jointly stationary if their joint distributions of

all orders are invariant under a shift of time origin.

 X(t) and Y(t) are jointly wide sense stationary if they are both
wide sense stationary and their cross-correlation (cross-
covariance) depends only upon t2– t1, i.e.
m X (t )  m X mY (t )  mY
RX (t1 , t 2 )  RX (t 2  t1 ) RY (t1 , t 2 )  RY (t 2  t1 )
RXY (t1 , t 2 )  RXY (t 2  t1 )

ELEC5300 Lecture 3 13
Cross correlation
 The cross correlation between X and Y, is an indicator of how
correlated X is with the value of Y t seconds into the future.
RXY ( )  E[ X (t )Y (t   )]

 Example: Unit delay R X ( )

X (t ) Y (t )  X (t  1)
DELAY

R XY ( )  E[ X (t )Y (t   )]
 E[ X (t ) X (t    1)] RYX ( ) RXY ( )
 R X (  1)

ELEC5300 Lecture 3 14
Example
Suppose that X(t) (the signal) and N(t) (the noise) are zero-mean
jointly WSS RPs
Define Z (t )  X (t )  N (t ) (noise corrupted signal)

Mean: E[ Z (t )]  E[ X (t )  N (t ))]  E[ X (t )]  E[ N (t )]  0

Autocorrelation:
RZ ( )  E[ Z (t ) Z (t   )]  E[ X (t )  N (t )    X (t   )  N (t   ) ]
 E[ X (t ) X (t   )]  E[ X (t ) N (t   )]  E[ N (t ) X (t   )]  E[ N (t ) N (t   )]
 RX  )  RXN ( )  RNX ( )  RN ( )

If we assume X and Y independent, then

RXN ( )  E[ X (t ) N (t   )]  0
RZ ( )  RX  )  RN ( )
RNX ( )  E[ N (t ) X (t   )]  0
ELEC5300 Lecture 3 15
Power Spectral Density (Continuous Time)
 Definition: The power spectral density (PSD) of a continuous-
time WSS random process X(t) is the Fourier/Laplace
Transform of its autocorrelation function,
 

 X  X
 j  s
S X ( j )  R ( )e d or S X ( s )  R ( )e d
 

 The autocorrelation can be recovered using the inverse

transform, e.g.: 
1
 X
j
RX ( )  S ( j  )e d
2 

 The PSD can be thought of as the average power in the signal at

frequency w, where the average is in the statistical sense (i.e.
taken over the ensemble).

ELEC5300 Lecture 3 16
Interpretation of the PSD
 The PSD is the average (in a statistical sense) of the
periodograms of length T of the RP for large T.
time (t) frequency
2 3

0 Fourier Transform 2

-2 0
0 20 40 60 -2 0 2
1
F X T t ,   3
2 2
X T t ,   T 2
0
1

-2 0
0 20 40 60 -2 0 2
2 3
randomness (x)

2
0
1

-2 0
0 20 40 60 -2 0 2
2
PSD 3

2
0
1

-2 0
0 20 40 60 -2 0 2

ELEC5300 Lecture 3
T 17
Expectation of the Periodogram
 For a realization of a RP X(t), define the periodogram of length
T to be the squared magnitude of the Fourier Transform of the
signal from time 0 to time T. 1
F X T t 
2

T
 Taking the expectation
T T
 1   1 
E  F X T t    E   X t e dt  X s e ds 
2  jt j s

T  T 0 0 
T T
1
  E[ X t X s ]e  j ( t  s )
dsdt make change
T 0 0
T T
of variables
1 t=t-s
 X
 j ( t  s )
 R (t  s ) e dsdt
T 0 0
T t T T t
1 1
 t X   X
 j  j
 R ( ) e d dt  R ( ) e ddt
ELEC5300 Lecture 3 T 0
T 0 t T 18
Evaluating the integral
1 2 1
T t switch order
E  F X T t     R X ( )e  j
ddt of integration
T  T 0 t T
0  T T T
1 1
   R X ( )e dtd 
 j
 X
R ( ) e  j
dtd
T T 0 T 0 
0 T
1 1
  (  T ) R X ( )e  j d   (T   ) R X ( )e  j d
T T T 0
T
  
  1   R X ( )e  j d

T 
T  t

T
 X
 j
 R ( ) e d as T  


-T T t

ELEC5300 Lecture 3 19
Properties of the PSD
 For a real-valued random process X(t), SX(jw) is even and real.

Proof: By Euler’s identity e

 j

 cos( )  j sin( ) 
 

 X R
 j
S X ( j )  R ( )e d  X ( )(cos( )  j sin( )) d
 
 
 R

X ( ) cos( )d  j  R X ( ) sin( )d

even in t odd in t
since RX is even


Thus,
S X ( j )  R

X ( ) cos( ) d

ELEC5300 Lecture 3
even in w 20
Properties of the PSD
 The mean squared value of X(t) (i.e. the signal power) is

 
1 1
 X S
2 j
E[ X (t ) ]  RX (0)  S ( j  )e d  X ( j )d
2   0
2 

 By extension, the power in a frequency range 0< w1 ≤ w ≤ w2 is

1 2
 Power in  1 1

      2
 1 2

 2
S X ( j  )d 
2  S X ( j )d
1

S X ( j )

 2  1 1 2 
ELEC5300 Lecture 3 21
Example peak value:  2 Autocorrelation
1
0.8
 Consider a random process X(t) with b=1
0.6
RX ( )   e 2   b=2
0.4

 Fourier transforming, 0.2

0
F{RX ( )} -5 0 time shift (t) 5
 F{ 2 e   u ( )   2 e   (  )u ( )}
 F{ 2 e   u ( )}  F { 2 e   (  )u ( )} 2
PSD
2 2 2 2  1.5
   2
j    j       2 b=1
1
b=2
0.5
2 2
peak value: 0
 -10 -5 0 5 10
frequency (w)
ELEC5300 Lecture 3 22
Power Spectral Density (Discrete-Time Processes)
 Definition: The power spectral density of a discrete time WSS
random process Xn is the discrete time Fourier Transform or Z
transform of its autocorrelation,
 
S X ( e j )   X
R (
k  
k ) e  j k
or S X ( z )   X
R (
k  
k ) z k

 Note that the PSD of discrete time processes is periodic in w

with period 2p.

ELEC5300 Lecture 3 23
Discrete time white noise
 A discrete time i.i.d. (independent and identically distributed)
zero mean RP is often referred to as discrete-time white noise.
 The autocorrelation of a discrete-time white noise process is

where d(x) is a discrete time delta function and s2 is the

variance of the process.

ELEC5300 Lecture 3 24
PSD of Discrete-time white noise
 The PSD of discrete-time white noise is flat.

j
S X (e )    2

k  
( k ) e  j k
  2

 Thus, it contains power at all frequencies equally. It is called

white-noise in analogy to white light. A random process whose
PSD is not flat is sometimes referred to as “colored noise.”

ELEC5300 Lecture 3 25
 Filtered Discrete Time White noise
Let Yn = Xn + Xn–1 where Xn is a discrete time white noise process.
Find SY(jw). RY (k )
2
2
Solution:
RY (k )  E[YnYn  k ]
 E[( X n  X n 1 )( X n  k  X n  k 1 )]
n
 R X (k )  R X (k  1)  R X (k  1)  R X (k )
  2 ( k  1)  2 2 (k )   2 ( k  1) S Y ( j )
5

4
j
SY (e )   2

k  
( (k  1)  2 (k )   (k  1))e  jk
3

  2 ( e  j   2  e j ) 2
2 1
  (2  2 cos( ))
0-4 -2 0 2 4


ELEC5300 Lecture 3 26
Cross Power Spectral Density
 Definition: The cross-power spectral density of two jointly
WSS processes X(t) and Y(t) is


 XY
 j
S XY ( j )  R ( )e d


 Because the cross correlation function is not necessarily

symmetric, the cross power spectral density may be complex
valued.
 Since R ( )  E[ X (t )Y (t   )]  E[Y (t   ) X (t )]
XY

 E[Y ( s ) X ( s   )]  RYX ( )

we have that S ( j )  S * ( j ) (* = complex

XY YX
conjugate)
ELEC5300 Lecture 3 27
Sum of Two Zero-Mean Processes
Suppose Z(t) =X(t)+N(t) where X(t) and N(t) are zero mean. Find SZ(f).
Solution:
RZ ( )  E[ Z (t ) Z (t   )]
 E[( X (t )  N (t ))( X (t   )  N (t   ))]
 E[ X (t ) X (t   )]  E[ X (t ) N (t   )]  E[ N (t ) X (t   )]  E[ N (t ) N (t   )]
 RX ( )  RXN ( )  RNX ( )  RN ( )
By linearity of Fourier transforms,

S Z ( j )  S X ( j )  S XN ( j )  S NX ( j )  S N ( j )

Note: If X(t) and N(t) are also independent, then

RXN ( )  E[ X (t )]E[ N (t   )]  0 and RNX ( )  0
independent zero mean

In this S Z ( j )  S X ( j )  S N ( j )
case,
ELEC5300 Lecture 3 28
Time Delay

WSS RP: X (t ) Y (t )  X (t  d )
DELAY

Since RY ( )  E[Y (t )Y (t   )]
 E[ X (t  d ) X (t  d   )]  RX ( ) Time shifts do not
change the PSD!
Thus, SY ( j )  S X ( j )

Since RXY ( )  E[ X (t )Y (t   )]  E[ X (t ) X (t    d )]  RX (  d )
 
Thus, S XY ( j )  

R XY ( )e  j d  

R X (  d )e  j d
   d
     d
 

R X ( )e  j (  d ) d  e  jd 

R X ( )e  j d

 S X ( j )e  jd
Note that this is complex valued!
ELEC5300 Lecture 3 29
Continuous time white noise
 Any stationary continuous-time process whose autocorrelation
and PSD are given by

where d(x) is a continuous-time delta function is referred to as

white noise.
 Note that the variance of continuous time white noise is infinite
and that two samples, no matter how closely sampled in time
are uncorrelated. Thus, continuous time white noise jumps
around “infinitely far, infinitely fast”
 This process is not physically realizable.

ELEC5300 Lecture 3 30
Bandlimited white noise
 A WSS random process whose PSD is constant for |w| ≤ 2pW
and zero otherwise.

ELEC5300 Lecture 3 31
Properties of Bandlimited White Noise
 Samples at intervals are uncorrelated.

 Using a statistical analog of the sampling theorem, it can be

shown that the process can be completely specified by the
statistics of samples at 1/(2W).

 Bandlimited white noise does not have infinite variance,

although the variance increases as W → ∞ 。

ELEC5300 Lecture 3 32
Bandlimited White Noise with Frequency Offset
 Bandlimited white noise may also be constant in a
neighborhood of a nonzero frequency Wo.

ELEC5300 Lecture 3 33
Gauss Markov Process
 The Gauss-Markov Process is the stationary Gaussian Random
Process whose autocorrelation function is given by

 For the discrete time process,

where the approximation holds for small w.

 We will see later that this process can be viewed as the output of
a lowpass filter with cutoff frequency w = b driven by Gaussian
white noise.

ELEC5300 Lecture 3 34
Random Telegraph Signal
 At any time, the random telegraph signal assumes values ±1. At
the origin both values are equally probable.
 It changes polarity randomly where the probability of k
switches in time T is given by a Poisson distribution, where the
parameter α is the average number of switches per unit time:

 The number and position of the polarity changes in two disjoint

time intervals are independent.

ELEC5300 Lecture 3 35
Mean and Autocorrelation of the Random Telegraph
 Since the value of the random telegraph at any time is equally
likely to be ±1, E[ X (t )]  0
 Autocorrelation:
 We first consider the case where t2 > t1

where K(t) is the number of polarity switches in a period of length t.

 Since (next page)
ELEC5300 Lecture 3 36
Probability that a Poisson RV is Even/Odd
 k 
If K has a Poisson distribution: p K k   e for k  0,1,2,
k!
2 3 4 5 6 
k
Then since 
e  1        
2! 3! 4! 5! 6! k  0 k!

e 
 1  
2 3 4 5 6
     

 
k

2! 3! 4! 5! 6! k 0 k!
e  e  2 3 4 5 6 k
 1         
2 2! 3! 4! 5! 6! k even k!

e  e  2 3 4 5 6 k
 1         
2 2! 3! 4! 5! 6! k odd k!


we have that PK is even   e   k
 e a
 e  1 1  2

k even k!
e 

2
   e
  2 2

 k
 e a
 e  1 1  2
P K is odd   e   e      e
k odd k!  2  2 2
ELEC5300 Lecture 3 37
Plots of Random Telegraph Autocorrelation

4
S X ( j ) 
4 2   2

ELEC5300 Lecture 3 38
Wiener Process Definition
 The Wiener process W(t) is the continuous time Gaussian
random process whose mean and autocorrelation functions are
given by
E[W (t )]  0
RW (t1 , t 2 )   2 min( t1 , t 2 )
200

150

100

50

-50

-100

-150

-200
0 1 2 3 4 5 6
ELEC5300 Lecture 3 39
Properties of the Wiener Process
 It is commonly used to model Brownian motion or a slowly
varying drift.
 The process is non-stationary, in particular, the variance
increases linearly with time:
Var ( X (t ))  RW (t , t )   2t
 The process can be defined as the output of an integrator driven
by white noise.
t 0
t W (t )
white noise
with PSD s2 0
 Alternatively, white noise can be defined as the derivative of the
Wiener process.

ELEC5300 Lecture 3 40