WIENER FILTERS

Week 3

ELE 774 - Adaptive Signal Processing

Linear Optimum Filtering: Statement

Complex-valued stationary (at least w.s.s.) stochastic processes.

Linear discrete-time filter, w0, w1, w2, ... (IIR or FIR (inherently stable))
y(n) is the estimate of the desired response d(n)
e(n) is the estimation error, i.e., difference bw. the filter output and the
desired response

Week 3

ELE 774 - Adaptive Signal

Linear Optimum Filtering: Statement

Problem statement:
Given
Filter input, u(n),
Desired response, d(n),
Find the optimum filter coefficients, w(n)
To make the estimation error as small as possible
How?
An optimization problem.

Week 3

ELE 774 - Adaptive Signal

Linear Optimum Filtering: Statement

Optimization (minimization) criterion:

1. Expectation of the absolute value,
2. Expectation (mean) square value,
3. Expectation of higher powers of the absolute value
of the estimation error.

Minimization of the Mean Square value of the Error (MSE) is mathematically

tractable.
Problem becomes:
Design a linear discrete-time filter whose output y(n) provides an
estimate of a desired response d(n), given a set of input samples u(0),
u(1), u(2) ..., such that the mean-square value of the estimation error
e(n), defined as the difference between the desired response d(n) and
the actual response, is minimized.

Week 3

ELE 774 - Adaptive Signal

Principle of Orthogonality

Filter output is the convolution of the filter IR and the input

Week 3

ELE 774 - Adaptive Signal

Principle of Orthogonality

Error:

MSE (Mean-Square Error) criterion:

Square Quadratic Func. Convex Func.

Minimum is attained when

(Gradient w.r.t. optimization variable

w is zero.)

Week 3

ELE 774 - Adaptive Signal

Derivative in complex variables

Let

then derivation w.r.t. wk is

Hence

or
!!! J: real, why? !!!

Week 3

ELE 774 - Adaptive Signal

Principle of Orthogonality

Partial derivative of J is

Using

Hence

Week 3

and

ELE 774 - Adaptive Signal

Principle of Orthogonality

Since

The necessary and sufficient condition for the cost function J to

attain its minimum value is, for the corresponding value of the
estimation error eo(n) to be orthogonal to each input sample that
enters into the estimation of the desired response at time n.

Error at the minimum is uncorrelated with the filter input!

A good basis for testing whether the linear filter is operating in its
optimum condition.

Week 3

, or

ELE 774 - Adaptive Signal

Principle of Orthogonality

Corollary:

If the filter is operating in optimum conditions (in the MSE sense)

When the filter operates in its optimum condition, the estimate of the
desired response defined by the filter output yo(n) and the
corresponding estimation error eo(n) are orthogonal to each other.

Week 3

ELE 774 - Adaptive Signal

10

Minimum Mean-Square Error

Let the estimate of the desired response that is optimized in the

MSE sense, depending on the inputs which span the space
i.e. (
) be

Then the error in optimal conditions is

or

Also let the minimum MSE be (0)

Week 3

ELE 774 - Adaptive Signal

HW: try to derive this

relation from the corollary.
11

Minimum Mean-Square Error

Normalized MSE: Let

Meaning

If is zero, the optimum filter operates perfectly, in the sense that

there is complete agreement bw. d(n) and
. (Optimum case)

If is unity, there is no agreement whatsoever bw. d(n) and

(Worst case)

Week 3

ELE 774 - Adaptive Signal

12

Wiener-Hopf Equations

We have (principle of orthogonality)

Rearranging

Wiener-Hopf
Equations
(set of
infinite eqn.s)

where

Week 3

ELE 774 - Adaptive Signal

13

Wiener-Hopf Equations

Solution Linear Transversal (FIR) Filter case

M simultaneous equations

Week 3

ELE 774 - Adaptive Signal

14

Wiener-Hopf Equations (Matrix Form)

Let

Then

and

Week 3

ELE 774 - Adaptive Signal

15

Wiener-Hopf Equations (Matrix Form)

Then the Wiener-Hopf equations can be written as

where
is composed of the optimum (FIR) filter coefficients.
The solution is found to be

Note that R is almost always positive-definite.

Week 3

ELE 774 - Adaptive Signal

16

Error-Performance Surface

Substitute

Rewriting

Week 3

ELE 774 - Adaptive Signal

17

Error-Performance Surface

Quadratic function of the filter coefficients convex function, then

or

Wiener-Hopf
Equations

Week 3

ELE 774 - Adaptive Signal

18

Minimum value of Mean-Square Error

We calculated that

The estimate of the desired response is

Hence its variance is

Then
At wo.
(Jmin is independent of w)
Week 3

ELE 774 - Adaptive Signal

19

Canonical Form of the Error-Performance Surface

Rewrite the cost function in matrix form

Next, express J(w) as a perfect square in w

Then, by substituting

In other words,

Week 3

ELE 774 - Adaptive Signal

20

Canonical Form of the Error-Performance Surface

Observations:
J(w) is quadratic in w,
Minimum is attained at w=w o,

Jmin is bounded below, and is always a positive quantity,

Jmin>0

Week 3

ELE 774 - Adaptive Signal

21

Canonical Form of the Error-Performance Surface

Transformations may significantly simplify the analysis,

Use Eigendecomposition for R

Then

Let

Substituting back into J

a vector

Canonical form

The transformed vector v is called as the principal axes of the surface.

Week 3

ELE 774 - Adaptive Signal

22

Canonical Form of the Error-Performance Surface

w2

wo
J(wo)=Jmin

v2
(2)

J(w)=c curve

J(v)=c curve

Jmin
Q
v1
(1)

Transformation
w1

Week 3

ELE 774 - Adaptive Signal

23

Multiple Linear Regressor Model

Wiener Filter tries to match the filter coefficients to the model of the
desired response, d(n).

Desired response can be generated by

1. a linear model, a
2. with noisy observable data, d(n)
3. noise is additive and white.
Model order is m, i.e.
What should the length of the Wiener filter be to achive min. MSE?

Week 3

ELE 774 - Adaptive Signal

24

Multiple Linear Regressor Model

The variance of the desired response is

But we know that

where wo is the filter optimized w.r.t. MSE (Wiener filter) of length M.

1. Underfitted model: M<m

Performance improves quadratically with increasing M.

Worst case: M=0,

2. Critically fitted model: M=m

w o=a, R=Rm,

Week 3

ELE 774 - Adaptive Signal

25

Multiple Linear Regressor Model

3. Overfitted model: M>m

Filter longer than the model does not improve performance.

Week 3

ELE 774 - Adaptive Signal

26

We do not know the values

Example

Let

the model length of the desired response d(n) be 3,

the autocorrelation matrix of the input u(n) be (for conseq. 3 samples)

The cross-correlation of the input and the (observable) desired

response be

The variance of the observable data (desired response) be

The variance of the additive white noise be

Week 3

ELE 774 - Adaptive Signal

27

Example

Question:
a) Find Jmin for a (Wiener) filter length of M=1,2,3,4

b) Draw the error-performance (cost) surface for M=2

c) Compute the canonical form of the error-performance surface.

Solution:
a) we know that

Week 3

and

ELE 774 - Adaptive Signal

then

28

Example

Solution, b)

Week 3

ELE 774 - Adaptive Signal

29

Example

Solution, c) we know that

where for M=2

Then

v2
(2)
Jmin

v1
(1)
Week 3

ELE 774 - Adaptive Signal

30

Application Channel Equalization

x(n)

Channel, h

y(n)

Filter, w

Delay,

z(n)
-

(n)

x(n-)

Transmitted signal passes through the dispersive channel and a

corrupted version (both channel & noise) of x(n) arrives at the receiver.
Problem: Design a receiver filter so that we can obtain a delayed
version of the transmitted signal at its output.
Criterion: 1. Zero Forcing (ZF)
2. Minimum Mean Square Error (MMSE)

Week 3

ELE 774 - Adaptive Signal

31

Application Channel Equalization

MMSE cost function is:

Filter output

Convolution

Filter input

Convolution

Week 3

ELE 774 - Adaptive Signal

32

Application Channel Equalization

Combine last two equations Convolution

Toeplitz matrix performs convolution

Compact form of the filter output

Desired signal is x(n-), or

Week 3

ELE 774 - Adaptive Signal

33

Application Channel Equalization

Rewrite the MMSE cost function

Expanding (data and noise are uncorrelated E{x(n)v(k)}=0 for all n,k)

Re-expressing the expectations

Week 3

ELE 774 - Adaptive Signal

34

Application Channel Equalization

Quadratic function gradient is zero at minimum

The solution is found as

And Jmin is

Jmin depends on the design parameter

Week 3

ELE 774 - Adaptive Signal

35

Application Linearly Constrained

Minimum - Variance Filter

Problem:
1. We want to design an FIR filter which suppresses all frequency
components of the filter input except o, with a gain of g at o.

Week 3

ELE 774 - Adaptive Signal

36

Application Linearly Constrained

Minimum - Variance Filter

Problem:
2. We want to design a beamformer which can resolve an
incident wave coming from angle o (with a scaling factor g),
while at the same time suppress all other waves coming from
other directions.

Week 3

ELE 774 - Adaptive Signal

37

Application Linearly Constrained

Minimum - Variance Filter

Although these problems are physically different, they are

mathematically equivalent.

They can be expressed as follows:

Suppress all components (freq. or dir. ) of a signal while
setting the gain of a certain component constant (o or o)

They can be formulated as a constrained optimization problem:

Cost function: variance of all components (to be minimized)
Constraint (equality): the gain of a single component has to be g.

Observe that there is no desired response!.

Week 3

ELE 774 - Adaptive Signal

38

Application Linearly Constrained

Minimum - Variance Filter

Mathematical model:
Filter output

Beamformer output

Constraints:

Week 3

ELE 774 - Adaptive Signal

39

Application Linearly Constrained

Minimum - Variance Filter

Cost function: output power quadratic convex

Constraint : linear
Method of Lagrange multipliers can be utilized to solve the problem.

output power

constraint

Solution: Set the gradient of J to zero

Optimum beamformer weights are found from the set of equations

similar to Wiener-Hopf equations.

Week 3

ELE 774 - Adaptive Signal

40

Application Linearly Constrained

Minimum - Variance Filter

Rewrite the equations in matrix form:

Hence

How to find ? Use the linear constraint:

to find

Therefore the solution becomes

For o, w o is
the linearly Constrained Minimum-Variance (LCMV) beamformer
For o, w o is
the linearly Constrained Minimum-Variance (LCMV) filter

Week 3

ELE 774 - Adaptive Signal

41

Minimum-Variance Distortionless Response

Beamformer/Filter

Distortionless set g=1, then

We can show that (HW)

Jmin represents an estimate of the variance of the signal impinging on the

antenna array along the direction 0.
Generalize the result to any direction (angular frequency ):

minimum-variance distortionless response (MVDR) spectrum

An estimate of the power of the signal coming from direction
An estimate of the power of the signal coming from frequency

Week 3

ELE 774 - Adaptive Signal

42