SIGNAL THEORY - version 2012

2. SPECTRAL ANALYSIS APPLIED

TO STOCHASTIC PROCESSES
2.0 THEOREM OF WIENER- KHINTCHINE

An important technique in the study of deterministic signals consists in

using harmonic functions to gain the spectral analysis. The computation of
the spectrum of signals is performed using the Fourier integral. The direct
application of the latter on stochastic signals is, however, in general not
possible, since the condition of integrability is not satisfied; i.e.

will not be finite. Hence, the Fourier integral does not exist.
Remark: a large set of deterministic signals do also not meet the
condition of integrability; e.g. the time harmonic signals: sin(t), cos(t). To
allow spectral analysis, distribution functions have been introduced (Dirac
~); i.e.:

with the condition:

Despite this mathematical incompatibility, it will be shown that the

concept of power spectrum can be applied also to signals not meeting the
condition of integrability for the Fourier integral. To be able to apply
harmonic analysis to stochastic signals the truncated signal T(k)(t) with width
T will be considered. This is equivalent to multiplying the original signal with
a rectangular window with amplitude 1 and width (duration) T, which is
placed symmetrically with respect to the time origin (t = 0):

The power of the truncated signal can be written as:

1
SIGNAL THEORY - version 2012

Since the truncated signal meets the condition of integrability, the

Fourier integral can be evaluated. Indeed, the power of the truncated version
of a practical signal will always be finite, since it is not possible to generate in
practice signals with an infinite power. For a practical signal the condition of
integrability is often checked using the square of the integrand, which then
yields the power.
Let:

Than one can write:

so that:

and, hence:

One defines the Power Spectral Density as:

(in the next paragraph one will conclude that the periodogram is used
here).
Obviously, this equation does not contain information on the phase of
the signal. The Power PT(k) can be obtained then by integrating the power
density function over the entire frequency domain:

2
SIGNAL THEORY - version 2012

When the duration of the truncation interval T increases and in the limit
will tend to infinity, one will assume that the limit exists, i.e. remains finite:

The assumption is justified, because a practical signal always will have

finite power content. To associate the power spectral density with the
stochastic space and not just with one single realization, one will make the
average of all possible realizations.
Hence:

so that:

The latter can be rewritten as:

with:

When the signal is wide sense stationary, BT(t1,t2) = BT(t1-t2) , and

hence:

3
SIGNAL THEORY - version 2012

Substituting t1 = t2 +  yields that dt1 = dt2, so that the limits of the integral
will tend from

Since the signal is assumed to be stationary, one can state that an

arbitrary time shift will not affect the statistical properties. In other words:

and hence:

which can be further reduced to:

Taking the limit for T   one obtains:

In other words: the Power Spectrum Density of the stationary stochastic

signal is the Fourier integral of the autocorrelation function.
Or:

Since the autocorrelation function is even B() = B(-), one can rewrite
the relations as originally published by Wiener and Khintchine as:

4
SIGNAL THEORY - version 2012

These equations are known as the Theorem of Wiener-Khintchine

( ) for stochastic processes.
For ergodic signals B() = R().

2.1 PERIODOGRAM

Consider a realization of the stochastic signal (k)(t) and truncate the

latter to a finite duration T; i.e. prepare a T(k)(t). The PSD of the truncated
signal is defined then as:

Classically, this density is labelled as the periodogram. This periodogram

is a stochastic function, since it will obtain different values for each chosen
realization (k) of the signal (k)(t). To gain knowledge on the spectral
properties of the stochastic space over the time interval T, the ensemble
average is computed (mean value of all possible realizations); i.e.:

It has been demonstrated that when the limit is taken for T  , the PSD
then becomes:

The function q() is a theoretical concept, since in practice the latter

cannot be measured because B() should be known  from 0  . For a
given realization (k)(t) over a time interval T, T(k)(t), however, the spectrum
can be computed using the Fourier integral:

5
SIGNAL THEORY - version 2012

This spectrum depends on the choice of the time window with width T
with respect to the time origin. If the observed signal is stationary and
ergodic, an approximation for the value of q() can be estimated from the
average of different, successive periodograms with width T. Hence, it
becomes possible to gain spectral information using the classical Fourier
integral, which will be implemented for discrete signals with an FFT (Fast
Fourier Transform).
This technique, which forms the basis for the Fourier analyser
measurement instruments, is depicted in Figure 2.1. Using the FFT for each
interval with width T de function XTj(k)() is obtained, and hence also the
densities qTj(k) can be derived. In the example the intervals do not overlap.
However, often 50% overlap is provided. One obtains then:

Since the signal is ergodic (from one single realization, the full set of
statistical properties can be derived), one can state that:

An alternative method to attain this result is given by the Theorem of

Wiener-Khintchine. One has to evaluate the autocorrelation function in the
time domain first, and then apply the Fourier integral to obtain the PSD:

In case that B() will approach its asymptotic value fast (see paragraph
2.2 further), then it becomes possible and interesting to obtain q() via the
Fourier integral, since then only a limited number of lags of B() are required.

6
SIGNAL THEORY - version 2012

Figure 2.1: The computation of the periodogram in Fourier analyzer

instrumentation.

2.2 PROPERTIES OF THE AUTOCORRELATION FUNCTION

Note that for ergodic signals the autocorrelation function of the

realization is equal to the autocorrelation function of the ensemble; i.e.:

1. If the signal has no deterministic component and when the

interval  tends to infinity, then the dependencies of the successive values at
2 different time instances (t1) and (t2) become less and less pronounced and
for  very large, one can assume that these values are statistically
independent. This is illustrated in Figure 2.2.
For such signals one can write that:

where a is the ensemble mean of (t).

Hence, one obtains that:

7
SIGNAL THEORY - version 2012

(t1) (t2) (tn) t

Figure 2.2: The dependency on the attained values of (t) will be high
only if the interval  = t2 –t1 is very small. Then ( t2) – ( t1) will be very small
too. In the case that  = tn – t1 is very large, (tn) and (t1) become statistically
independent.

If  will tend to infinity, R() will asymptotically approach a2. This will be
achieved either in a monotonous or else in an oscillatory way, as is depicted
in the Figure 2.3.

Figure 2.3: An example of the evolution in time of the autocorrelation

function R(). This case shows the oscillatory trend in reaching the
asymptotic value when .

2. Since:

8
SIGNAL THEORY - version 2012

One can derive this result for a wide sense stationary ergodic signal also
directly from the relations of Wiener-Khintchine. Indeed, since:

and, hence:

In other words, the value of the autocorrelation function of the time

origin corresponds to the power of the signal.
One has on the other hand that:

The variance of (t) therefore becomes:

This illustrates that the autocorrelation function can be used to estimate

the variance of a stochastic signal; e.g. a noise source.

3. From equation (‡) one derives that R() = R(-), and hence, that
the autocorrelation is an even function.

4. The value of the autocorrelation function of a wide sense

stationary stochastic signal in the time origin, R(0), cannot be exceeded for
other values of 0. In other words:

Remark: To proof this condition, one will derive the initial statistical
moment of the second order; i.e.:

9
SIGNAL THEORY - version 2012

This is valid, since stationarity and ergodicity imply that:

Hence, R(0)  R().

5. The autocorrelation of a periodical signal is also periodic.

The autocorrelation function can be evaluated, of course, also for any
kind of deterministic or mixed signals. One will compute now the
autocorrelation function for a symmetric periodical square wave signal and
demonstrate that the autocorrelation turns out be periodical as well with the
same period as the one of the concerned signal.
Let:

The attained value remains valid from the given time instance up to T/2
further for both cases. The signal is depicted in Figure 2.4.

Figure 2.4: A periodical symmetrical square wave.

i) observe that the signal is stationary;

ii) that the signal also is ergodic;

10
SIGNAL THEORY - version 2012

iii) the signal has no D.C. component (direct current) ; in other words the
ensemble average equals zero, since the realization average and ensemble
average are equal.

And hence, the time zero reference t0 can be chosen arbitrarily; e.g. t0=0.
This will be assumed to be the case further on.
The autocorrelation, therefore, can be evaluated using:

To compute this function, the original signal will be shifted over a time
delay , the product of the two versions will be made and then averaged. This
is illustrated in Figure 2.5. One can immediately conclude from this that the
autocorrelation function is periodical with the same period T.

Figure 2.5: The original square wave signal (t) and the version shifted
with a time delay .

The autocorrelation in case that becomes:

From Figure 2.5 one rewrites this equation easily as:

11
SIGNAL THEORY - version 2012

and hence:

with:

This autocorrelation function is depicted in Figure 2.6.

Figure 2.6: The autocorrelation function of a symmetrical square wave

function.

As an exercise, one will compute the autocorrelation function for

another deterministic signal.

Let

12
SIGNAL THEORY - version 2012

Proof then that the autocorrelation function yields:

With the condition of periodicity that also holds here:

2.3 THE TELEGRAPH SIGNAL

As a first example of a stochastic process, the classical telegraph signal

will be studied. The goal is to estimate the bandwidth of such a signal, which
can be gained from the knowledge of the PSD. Hereto, one must derive the
autocorrelation function and apply further the theorem of Wiener-
Khintchine. To proceed, one must determine the statistical properties of such
signals first. This will be obtained by careful observation of the signal that is
produced by a telegrapher in a telegraphy communication system. The course
of a telegraph signal is shown in Figure 2.7.

Figure 2.7: The evolution of the classical telegraph signal.

This signal is transmitted asynchronously; i.e. the transmission is not

controlled by a clock. It is representative for classical telegraphy that encodes
the characters from the alphabet, the numbers and some punctuation into
sequences of dots and bars (Morse code). The dots and bars are either
transmitted via light signals (flashing lights) or can be made audible at the
reception. The code that Morse has developed attaches a shorter duration
code to the symbols that are more probable then to those that occur rather

13
SIGNAL THEORY - version 2012

occasionally. In Chapter 4, the followed approach by Morse, will be analysed

and confronted to the theorems of the information theory. One will recall
that the most probable symbols in English, but also in Dutch, is the ‘e’. Morse
encoded the ‘e’ with the shortest possible symbol being a single dot.
Individual dots and dashes and individual characters are separated by a
pause. For a trained Morse telegrapher, the pause has about the same
duration as a dot.
The international distress signal in Morse coding (Save Our Souls) is well
known:
. . . ___ ___ ___ . . .
S O S
Another still nowadays often used announcing tone for the reception of a
SMS in a cell phone (Short Message Service, e.g. in GSM), is encoded as:
. . . ___ ___ . . .
S M S
The line code is voltage driven and based on a symmetric voltage
polarity [-a,+a]. The signal will receive the voltage -a for the pause and the
voltage +a for a dot or a dash. Although classical telegraphy is outdated, it
remains a back-up technique for messaging in many fields.
To be able to answer the question on the occupied bandwidth of the
telegraph signal, a priori knowledge on the latter must be formulated.
How such a stochastic signal can be described? It must satisfy the
following conditions:
1. The average number of polarity changes is given by the
stochastic parameter . In fact  is dependent on the realization
k. For each telegrapher, (k) is the average of polarity changes that
he is producing. Here,  represents the average value of polarity
changes obtained by the keying of all Morse telegraphers in the
world. Hence, the average corresponds to the average value of
the stochastic space; i.e. the ensemble average. In other words:

2. P(=a) = P(=-a); i.e.:

One will observe also that the signal is ergodic, so that also:

Hence, the D.C. component of the signal is zero. The signal is

perfectly bipolar. In case that the transmission of the signal is
performed using electrical lines (transmission lines), this
condition is of great value, since, in this way, it will ensure that
the electrical line will not be charged. To agree with this

14
SIGNAL THEORY - version 2012

statement, a technical equivalent electrical circuit for a section

with length l must be provided. In Figure 2.8 such a classical
equivalent circuit is depicted.

Figure 2.8: Equivalent electrical circuit for a section of an

electrical line with an elementary length l.

3. The probability to have a polarity switch is proportional to the

length of the time interval that one considers.
4. Two successive polarity switches cannot be infinitely close to
each other. Hence, an interval will exist where the probability to
have a single polarity switch is much larger than to have two
changes. In other words: consider an infinitesimal small interval
. The probability to have one single polarity change is much
larger than to have two or more changes. If N is very high, the
probability to have N changes in this interval with width  is close
to 0.
5. The probability to have exactly k polarity changes in an interval
with length  is then given by a Poisson distribution.

Proof: Divide the interval  in n subintervals. If n is sufficiently

large, the probability to have a single polarity change in a
subinterval is, following the definition of , given by p :

Hence, the probability to have no polarity change in the same

subinterval is given by q=1-p. The probability to have there k
switches is then given ten by the binomial distribution P(k)1.
1
Remark: the probability that in the interval [t, t+t] (t) would have exactly one single polarity change,

15
SIGNAL THEORY - version 2012

This can be rewritten as:

What will happen if the number of subintervals n would tend to

infinity? Or:

Since:

One can rewrite the binomial distribution as:

And hence:

Taking the limit for n:

The following approximation is used to obtain this expression:

is given by t. Hence, the probability for no change is simply [1-t]. If (t0) is given, then all the previous
attained values of (t) have no influence on the future value. The future value will via  only depend on the
current value at the time instance t=t0. Hence, the tlegraph signal is a Markov process!

16
SIGNAL THEORY - version 2012

To determine the PSD, which is required to gain insight on the

bandwidth for transmission, one should derive the autocorrelation function.

Remarks:
i) Note that the signal is wide sense stationary;
ii) Hence, conclude that the signal is ergodic, since the used
parameter  is the ensemble average.
Conclusion:

To gain better insight, the classical notation will be proposed, although it

is known that:

The autocorrelation function is the expressed as:

There are only 4 possibilities for x1,2 on t1,2:

With this, the autocorrelation function can be rewritten as:

17
SIGNAL THEORY - version 2012

But, recall that:

That is because it is supposed that the polarity switching is performed in

an infinite small amount of time. Mathematically, this is achieved even in zero
time, so that the pdf is limited to Dirac-distributions. Else, the combined
probability would have to be noted as:

And hence, the autocorrelation function reduces to:

Suppose know that for each t1,t2 the events {(t1)=a, (t2)= a} and
{(t1)=-a, (t2)=-a} have the same probability. This means that for a certain
interval =t1-t2, wherever placed with respect to the signal, the probability to
have no polarity change, or 2 polarity changes, or, in general an even number
of polarity changes, is independent of the attained value (both for +a and for
-a the same probability is assumed).
The same condition is assumed for the events {(t1)=a, (t2)= -a} and
{(t1)=-a, (t2)= a}. This means that for a certain interval =t1-t2, wherever
placed with respect to the signal, the probability to have one single polarity
change, or 3 polarity changes, or, in general an odd number of polarity
changes, is independent of the attained value (both for +a and for -a the same
probability).
One has also noted that P((t1)=+a) = P((t1)=-a) = 0.5, so that the signal is
statistically symmetric with respect to the time axis. Hence P=P and P=P.
The autocorrelation function, therefore, results in:

18
SIGNAL THEORY - version 2012

With the definition of the conditional probability, the latter can be

written as:

Remark: It should not surprise to work with conditional probabilities,

since the telegraph signal is a Markov process.
One recognizes in the equation for the autocorrelation function the
probabilities for odd and for odd polarity changes.

Probability for even number Probability for odd number

Hence:

but  = t2 – t1 and the autocorrelation function B(t1,t2) = B(t2 – t1) = R()

(stationary and ergodic signal). Furthermore, R is symmetrical with respect
to , de length of the time interval, so that the solution also is valid for < 0.
Taking  =  t2 – t1, then  > 0. The autocorrelation function becomes:

The autocorrelation function of the telegraph signal is sketched in Figure

2.9.

19
SIGNAL THEORY - version 2012

Figure 2.9: The autocorrelation function of the telegraph signal.

Using the theorem of Wiener-Khintchine, one can retrieve the PSD.

This can be rewritten as:

The power spectral density of the telegraph signal is depicted in Figure

2.10.
It is easy to retrieve the 3 dB point. One finds that the frequency
corresponding to the latter is given by:

The power of the signal is given by P = R(0) = a2. This result can be
retrieved also from the definition and the pdf:

20
SIGNAL THEORY - version 2012

One also finds the asymptotic value for large time intervals:

The variance can be found further from:

Figure 2.10: The PSD of the telegraph signal, where a=1 and =1.

The pdf of the telegraph signal is illustrated in Figure 2.11.

Figure 2.11: The pdf of the telegraph signal.

21
SIGNAL THEORY - version 2012

2.4 THE BINARY SYNCHRONOUS NRZ- SIGNAL

The binary synchronous Non-Return-to-Zero signal is important in

telecom, since it represents the easiest line coding to transmit binary coded
information in a bit serial manner. It is synchronized by a clock; i.e. a new
symbol is generated each next period of the clock. A possible polarity switch,
therefore, will be invoked also by the clock. The NRZ-signal is depicted in
Figure 2.12.

Figure 2.12: The binary synchronous Non-Return-to-Zero signal.

Binary signals are generated with a bit rate of 1/T bit/s and will take the
voltage values x1 and x2. These voltages are used to encode the characters of a
source with alphabet length equal to 2. This corresponds to a binary source
with symbols {0,1}. Each time the clock period evolves, a polarity change will
occur if the new symbol differs from the previous one, else not.
At the receiving side, a clock with the same period is used to sample the
signal; i.e. to decode the voltages x1,2 into the symbols of the binary source
{0,1}. Both clocks (transmitter-receiver) have the same frequency, but are not
synchronized; i.e. their zero-crossings do not occur at the same time. And
even if both clocks would be synchronized, the incoming time signal will be
shifted from the clock of the receiver due to the delay in the transmission
channel. Hence the moment of sampling will occur somewhat between 0
(perfectly synchronized and no delay) and T. The shift cannot be greater then
T, since this would imply the reception of another symbol from the source. In
Chapter 1, an example has been treated to demonstrate how the phase shift 
should be distributed in order to assure that the signal will be wide sense
stationary. The study showed that  should be uniformly distributed between
0 and T to make the NRZ-signal wide sense stationary. This condition is
necessary to be able to apply the Wiener-Khintchine theorem further to
compute the PSD. Again, with the PSD, the bandwidth of the NRZ-signal can

22
SIGNAL THEORY - version 2012

be determined.
The symbols that the binary source is producing are assumed to be
statistically independent. Also here the polarity switching in the line coding is
assumed to be executed extremely fast, so that mathematically this can be
modelled as happening in an infinite small amount of time, i.e. in 0 s. Then the
pdf can be expressed as:

The signal has only two different possible values: either x1, else x2.
This implies further that:

The ensemble average for the NRZ-signal is expressed as:

The quadratic mean yields:

This follows simply from the property that:

To determine the autocorrelation function one has to execute the

following operation (ergodicity):

In the case that  > T, the values (t) and (t+) are independent of each
other, since the source is supposed to produce symbols which are
independent of each other. Since the signal is stationary, the autocorrelation
in this case can be written as:

In the case that   T one has two mutual exclusive cases: one will
observe a polarity switch, i.e. (t)  (t+), or there is no change in polarity, i.e.
(t)  (t+). The autocorrelation for the two cases, therefore, results in:

23
SIGNAL THEORY - version 2012

This is noted further in a more compact form as:

Both probabilities can be derived from the uniform distribution of  in a

period T.

The event that (t)  (t+) will occur when a polarity change takes place
in the interval with duration ; i.e. when  < . One obtains then:

With this, also the second probability can be found easily, since:

And hence:

This results in:

It is possible to note this even more compact when the function tri() is
introduced:

The autocorrelation function of the NRZ-signal can be written then as:

The evolution of the autocorrelation in function of the duration of the

interval is depicted in Figure 2.13.

24
SIGNAL THEORY - version 2012

Figure 2.13: The autocorrelation of the Non-Return-to-Zero signal.

The PSD is found from the application of the theorem of

Wiener-Khintchine.

This is illustrated in Figure 2.14.

Figure 2.14: The PSD of the Non-Return-to-Zero signal.

One can compute from the PSD that 91% of the power is contained in the

25
SIGNAL THEORY - version 2012

frequency range from D.C. up to the first zero (major bin in Figure 2.14),
which corresponds to 1/T Hz; i.e. in the range:

Hence, as a rule of the thumb one can state that the bandwidth of a NRZ
line encoder yields the same value as the rate of the clock. Hence, 100 Mbit/s
transmission will require about 100 MHz bandwidth.

2.5 WHITE GAUSSIAN NOISE

A stationary signal is Gaussian if the pdf is given by:

Where:

The variance is given by:

Remark: Since the signal is stationary, the density of first order is

independent of the time.
The density of second order is defined as:

where  = t2 - t1; the interval where the second order moment is

evaluated. Again, since the second order density is dependent only on the
length of the interval  and not on the choice of the time instances t2 and t1, the
signal must be wide sense stationary.
 is the correlation coefficient of Pearson:

This can be rewritten as:

26
SIGNAL THEORY - version 2012

Hence, the correlation function corresponds to the normalised

covariance, since -1 <  < +1.
If the process is zero mean, then the correlation coefficient reduces
further to:

If the signal is ergodic, then  represents the r.m.s. value of the signal
(root mean square):

In case that ()=0 for 0 and (0)=1, then the noise source is called
white2.
For a Gaussian white noise source the density of second order can then
be expressed as:

This shows furthermore that then:

This clearly shows that a Gaussian white noise source behaves as a

totally stochastic signal. From this one obtains also that R() = 2() and
S()= Cst. From the PSD one observes that all frequencies seem to be present
with equal importance. In the case of light, when all colours are present with
equal weight, this would be noted as white light, which explains the choice of
the name for white noise.
One can write now the power P and the mean :

2
In order words () = (), and hence B() = 2 (). It can be shown that the condition for a Gaussian
noise source to be ergodic is given by:

I.e., the autocorrelation function satisfies the condition of integrability.

For a white Gaussian noise source this reduces to I < 2, so that B() = R().

27
SIGNAL THEORY - version 2012

Important remark: Every stochastic process, where the PSD yields a

constant for all frequencies, will be called white noise. Since the
autocorrelation function equals a Dirac-distribution, the stochastic variables
(t1) and (t2) are totally uncorrelated. In case the signal, furthermore, is
Gaussian, to be uncorrelated means statistical independency (totally
stochastic stationary process, since the second order density appears as the
product of the first order densities, which are independent of the time).

2.6 KARHUNEN-LOEVE EXPANSION

It is well known that a function f(t) can be expanded in a series:

when a sequence of orthonormal 1(t), 2(t) , … , n(t) , … base functions

are chosen ensuring convergence of the series in the time interval [-T/2,+T/2]:

The base functions can be complex.

One will attempt to extrapolate this technique further to stochastic
processes (or signals). If x(t) is a stochastic signal then the following series
can be proposed.

In case of equality, the coefficients bn can easily be retrieved.

(①)

28
SIGNAL THEORY - version 2012

(②)
Indeed, multiplication of both hand sides of equation (①) with n*(t)
and integration, yields directly equation (②).
The Fourier series is a special example.

Note however that the coefficients an are not orthogonal3.

The question that is asked is whether it is possible to choose the base
functions n(t) in such a way that the coefficients bn would be orthogonal, and
hence here also, uncorrelated?
Uncorrelatedness will mean, as will be shown further, that the
coefficients will satisfy the relation that their covariance is zero:

Often, uncorrelated stochastic variables are statistical independent also.

However, this cannot be generalised. In the cases treated for the
Karhunen-Loève expansions, however, uncorrelated coefficients will be
statistical independent as well.
For a stationary signal the statically independency was written as (see
e.g. the NRZ signal):

Since the base functions can be complex, so can the coefficients be

complex numbers. The cross-correlation relation for the coefficients then
becomes:

Suppose further that the ensemble average of the stochastic space for b
equals zero:

3
The base functions in the Fourier series are orthonormal; i.e.

29
SIGNAL THEORY - version 2012

One will now derive the base functions n(t) that ensure orthogonally of
the coefficients bn.
Only in the case that the ensemble average of the coefficients bn is zero,
orthogonality and uncorrelatedness refer to the same property, because then
the inner product in the orthogonality condition and the covariance operator
refer to the same equation.
Suppose that the autocorrelation function of x(t) is given by B(t1,t2), and
that B(t1,t2) is continuous. Suppose further that the ensemble average of x(t) is
zero, so that the ensemble average of the coefficients bn also becomes zero.
One will be able now to proof easily that the orthogonality of the coefficients
bn results into uncorrelatedness of the latter.
Remarks:
i) bn  bn(t), but depends on the realisation k: x(t) =xk(t).
ii) the condition that the ensemble average of bn equals zero does not
imply that the signal x(t) would be ergodic.

2.6.1 THEOREM OF KARHUNEN-LOEVE

To ensure that the coefficients bn, given by ②, are orthogonal, the base
functions n(t) must satisfy the integral equation:

for a certain value of =n and the variance of bn should be equal to n.

Proof:
Since:

one can write that:

Taking the ensemble average on both sides, yields:

30
SIGNAL THEORY - version 2012

So that:

(③)
This is because m(t) are unique functions independent of the
realization, so that only the coefficients bn depend on the realization (k).
Hence:

On the other hand, taking the complex conjugate of equation (②), and
substituting, yields:

Reminding the general definition of the autocorrelation function, one

can write:

And hence, the theorem is proven.

In case that x(t) is stationary and ergodic, the theorem can be written
also as:

In practice it is very difficult to solve the integral equation. The

31
SIGNAL THEORY - version 2012

advantage of the expansion into this type of series, however, is the optimal
convergence. The expansion is important for compression of coded signals.
The base functions are, indeed, independent of the realization and therefore
can be made available to the receiver on beforehand. Instead of sending the
signal through the channel as such, one has only to transmit the coefficients,
so that at the receiving side, the signal can be reconstructed with the series
expansion after some computation. In case the convergence is optimal, only a
few number of coefficient need to be transmitted to establish a truncated
version of the series, of which the error will be small due to fast convergence
features.

2.6.2 THE PROLATE SPHEROIDAL WAVE FUNCTIONS

Suppose the process x(t) wide sense stationary and ideal baseband low
pass (i.e. an adequate model for white noise filtered by an ideal rectangular
filter). Then the PSD of , S()=S0 for   c and else S()=0 is as depicted in
the Figure 2.15.

Figure 2.15: The PSD and the autocorrelation function of the ideal
baseband low-pass signal.
The autocorrelation function cab easily be computed from the PSD using
the theorem of Wiener-Khintchine:

The Karhunen-Loève series expansion can be performed when base

functions n(t) are found that satisfy the integral equation:

32
SIGNAL THEORY - version 2012

or:

This integral equation has been solved and has solutions n(t,c), where
c=cT. These solutions are known as the prolate spheroidal wave functions.
One can observe that the Eigen values n(c) and the Eigen functions of
n(t,c) of the integral equations are only dependent of c.
Example: c=4: (4) = 0.996; 0.912; 0.519; 0.1 10 for n = 0; 1; 2; 3.
The Eigen functions n(t,c) are depicted in Figure 2.16.

Figure 2.16: Prolate Spheroidal functions: copied from D. Slepian et al.,

Bell System Journal, Vol 4 , p40, 1961.

2.6.3 THE KARHUNEN-LOÈVE EXPANSION APPLIED TO THE

TELEGRAPH SIGNAL

The Karhunen-Loève expansion will be applied to the telegraph signal,

and is purely meant for the information of the reader only, to demonstrate
the complexity of the method.

33
SIGNAL THEORY - version 2012

The autocorrelation function of the telegraph signal has been evaluated

in section 2.3. One found there that:

One will now attempt to derive the orthogonal series expansion for (t)
in the interval [-A,+A]. Hereto the integral equation must be solved:

Introduce following substitutions:

The integral equation then becomes:

Solving this integral equation will lead to find the set of base functions
that will allow further to derive the Karhunen-Loève expansion for a
stochastic signal that has zero ensemble mean. This part is intended purely
for the reader’s information and is no part of the material for examination.
One will try to solve the integral equation by formulating a linear
differential equation to which f(t) must obey and find general solutions of the
latter. Next, one will enter these solutions into the integral equation to find
the value of . The integral equation can be rewritten first as:

Differentiating a first time yields:

or:

34
SIGNAL THEORY - version 2012

Differentiating a second time results in:

and hence:

or:

In other words: in order that the integral equation would hold for f(t),
the latter must satisfy also to the linear second order differential equation.
Therefore, one will solve the differential equation and feed the integral
equation with a general solution, taking into account all possible values of ;
i.e.4:

Assume first the last possibility; i.e.  > 2. Then, one can observe that:

Rewrite the latter as:

The differential equation, then, equals:

4
  0, because of the definition of  = 2 , where  represents the average number of polarity changes
per second, and hence, is positive and since:

35
SIGNAL THEORY - version 2012

The general solutions, therefore, will be of the form:

Substituting this general solution into the integral equation, one will
obtain after integrating and grouping of the terms the following expression:

In order to enable f(t) to satisfy the integral equation, the coefficients in

et and e-t should be zero, as is indicated in equations I a and I b.

and:

Summing the two equations, one obtains:

In case c1 + c2  0, then it is not possible to satisfy this equation for

0<a <1. Hence, the condition c1 = - c2 follows. Substituting the latter into
2

equation l a, one obtains:

It is not possible to satisfy to this equation a, where 0<a2<1. To

conclude, one notes that to the integral equation one cannot obey if >2.
Consider now the case where 0 <  < 2. Then:

Rewriting this using the substitution:

36
SIGNAL THEORY - version 2012

The general solution of the differential equation takes the form:

Substituting jb by a, one will return to the previous case. Also here, one
will note that the equations cannot be satisfied in the case that c1  c2. Hence,
one will retain two possibilities: c1 = c2 and c1 = - c2. In case that c1 = c2, one
finds that:

or, expressed in terms of b:

One will note now the solutions that differ from zero of this equation as
bn (one will find an infinite number of solutions bn). Substituting bn by n
relying on the definition of b, then the right hand side of the equation
becomes f(t) and is hereto satisfied. In case that c1 = - c2, one can
demonstrate in a similar manner that the integral equation can be satisfied, if
b agrees with:

To summarise, it is possible to propose a set of Eigen functions and

Eigen values that satisfy:

Where satisfies to and:

Where satisfies to .

One will demonstrate readily that the two other cases for  do not yield
solutions neither. In case  = 0, then f '(t) = , and if  = 2, then f '(t) = 0, so that:

The integral equation then becomes:

37
SIGNAL THEORY - version 2012

After integration, one obtains:

The only solutions are found when c1 = c2 = 0.

Hence, the telegraph equation (t) can be expanded into the
Karhunen-Loève expansion as:

Where:

And:

Where and are solutions of:

38
SIGNAL THEORY - version 2012

and where:

39