L14: Optimal linear filtering

– Wiener filtering

Lennart Svensson

Department of Signals and Systems

Chalmers University of Technology
 Problem formulation
General solution
Filtering solutions

Reviewing L11-L13
What have we done so far?
Signal models (L11-L12)
Nonparametric models: ACF and PSD.
Parametric models: AR, MA and ARMA.
Signal model estimation (L13)
Nonparametric spectral estimation: the periodogram.
Pros:
fast to compute
asymptotically unbiased.
Cons:
limited resolution for finite N:
the modified periodogram improves this
large variance for all N:
Blackman-Tukey’s method lowers variance.
Parametric spectral estimation: AR-estimation.
1 Estimate rx [k] from data.
2 Reformulate Yule-Walker to get â = Rx−1 rx .
Chalmers University of Technology Lennart Svensson 2/12
 Problem formulation
General solution
Filtering solutions

Learning objectives

After today’s lecture you should be able to

explain what type of problems Wiener-filters can solve.

derive the Wiener-Hopf (WH) equations.

use the WH-equations to derive a causal FIR Wiener filter.

use the WH-equations to derive a non-causal IIR Wiener

filter.

Compute the mean squared error (MSE) of a Wiener-filter.

Chalmers University of Technology Lennart Svensson 3/12

 Problem formulation
General solution Filtering, smoothing and prediction
Filtering solutions

Filtering, smoothing and prediction

Let s[n] and w [n] be zero mean, wide sense stationary
processes and
x[n] = s[n] + w [n].

Objective
Select H(z) to make e[n] as ”small” as possible

d̂ [n]
x[n] H(z) e[n] = d̂ [n] − d [n]
−
d [n]

Small could mean different things. We use mean squared error

E e[n]2 ,


since this is easy to minimize.

Chalmers University of Technology Lennart Svensson 4/12
 Problem formulation
General solution Filtering, smoothing and prediction
Filtering solutions

Filtering, smoothing and prediction

Based on measurements collected up until now, we encounter three

common problems (k > 0):
Filtering – estimating current signal values, d [n] = s[n].
Applications: positioning, control systems, noise or echo
cancellation, etc.

Smoothing – estimating past signal values, d [n] = s[n − k].

Applications: signal analysis, image processing, system
identification (modelling).

Prediction – estimating future signal values, d [n] = s[n + k].

Applications: decision making, planning, weather forecasts,
etc.

Chalmers University of Technology Lennart Svensson 5/12

 Problem formulation
General solution Filtering, smoothing and prediction
Filtering solutions

Filtering, smoothing and prediction

These problems can be illustrated as

time

Prediction:

Filtering:

Smoothing:
Time of interest
Measurements

Chalmers University of Technology Lennart Svensson 6/12

 Problem formulation
General solution Filtering, smoothing and prediction
Filtering solutions

Filtering, smoothing and prediction

We seek a linear estimator (filter)
X
d̂ [n] = h[n] ? x[n] = h[k]x[n − k]
k
of d [n].
As mentioned above, we wish to minimize the mean square
error (MSE),
 !2 
 X 
MSE(h) = E d [n] − h[k]x[n − k]
 
k

where the vector h contains the impulse response coefficients

h[k].
The resulting Wiener filter d̂ [n] is a linear minimum mean
square error (LMMSE) estimator.
Chalmers University of Technology Lennart Svensson 7/12
 Problem formulation
General solution Wiener-Hopf equations
Filtering solutions

Wiener-Hopf (W-H) equations

The W-H equations are very important and can be used to

solve all the problems mentioned above.

Objective: (again) We wish to minimize

 !2 
 X 
MSE(h) = E d [n] − h[k]x[n − k]
 
k

with respect to h.

Derivation 1: the function is quadratic in h

⇒ it is convex in h
⇒ no local optima (except for the global optimum)
⇒ sufficient to differentiate and set to zero!
Chalmers University of Technology Lennart Svensson 8/12
 Problem formulation
General solution Wiener-Hopf equations
Filtering solutions

Wiener-Hopf (W-H) equations

Differentiate the MSE:
 !2 
∂ ∂  X 
MSE(h) = E d [n] − h[k]x[n − k]
∂h[t] ∂h[t]  
k
( ! )
X
= E 2 d [n] − h[k]x[n − k] (−x[n − t])
k
X
= −2rdx [t] + 2 h[k]rx [t − k]
k
Setting this derivative to zero gives the
Wiener-Hopf (WH) equations
X
h[k]rx [t − k] = rdx [t],
k
for all t where h[t] is free to select.
Chalmers University of Technology Lennart Svensson 9/12
 Problem formulation
Causal FIR filters
General solution
MSE of optimal FIR filter
Filtering solutions

FIR filters
Suppose H(z) is a causal FIR filter:
p−1
X
d̂ [n] = h[k]x[n − k].
n=0

The W-H eq’s can be written as

    
rx [0] rx [1] ... rx [p − 1] h[0] rdx [0]
 rx [1] r x [0] ... rx [p − 2]  h[1]   rdx [1] 
   
.. .. .. .. = ..

 ..    
 . . . .  .   . 
rx [p − 1] rx |p − 2] ... rx [0] h[p − 1] rdx [p − 1]
| {z } | {z } | {z }
Rx h rdx

which yields that

hopt = R−1
x rdx .

Chalmers University of Technology Lennart Svensson 10/12

 Problem formulation
Causal FIR filters
General solution
MSE of optimal FIR filter
Filtering solutions

What is the minimum MSE?

The minimum MSE can be calculated by plugging in hopt :

 2 n  o n o
E emin [n] = E emin [n] d [n] − d̂opt [n] = Note: d̂opt ⊥ e[n]
p−1
( ! )
X
=E d [n] − hopt [k]x[n − k] d [n]
k=0
p−1
X
T −1
= rd [0] − hopt [k]rdx [k] = rd [0] − rdx Rx rdx
k=0

Special case: if d [n] and x[n] are uncorrelated, then d̂ [n] = 0

and the MSE is rd [0].
In general, the more correlated (similar) x[n] is to d [n] the
better is the estimate d̂ [n]!
Chalmers University of Technology Lennart Svensson 11/12
 Problem formulation
Causal FIR filters
General solution
MSE of optimal FIR filter
Filtering solutions

Learning objectives

After today’s lecture you should be able to

explain what type of problems Wiener-filters can solve.

derive the Wiener-Hopf (WH) equations.

use the WH-equations to derive a causal FIR Wiener filter.

use the WH-equations to derive a non-causal IIR Wiener

filter.

Compute the mean square error (MSE) of a Wiener-filter.

Chalmers University of Technology Lennart Svensson 12/12