H64DSP & H64DS2

Lecture 6
Adaptive Filters

Lecture Outline & Learning Outcomes

To have understanding on the following:
Why need adaptive filtering
Linear Optimal Wiener Filter and Gradient Descent
Method.
Applications of adaptive filtering

LO3 analyse and/or design FIR, IIR and adaptive

digital filters;

Why Adaptive?
Classical approach lowpass, highpass, bandstop and
bandpass filters with fixed coefficients. Filters will only work
for cases where spectra of signals of interest and noise do not
overlap and do not vary.
Optimal approach filters coefficients can be adapted and
are usually selected as the optimum values which minimise a
cost function in terms of mean squared error (or difference
between actual output and desired output).
When no a priori information is available, we need adaptive
filters which can adjust their own parameters based on the
incoming signal.

Structure of an Adaptive Filter

The device is a filtering process to produce an output in response to a sequence of

input data.
The adaptive process control an adjustable set of parameters used in the filtering
process.
FIR filters are popularly used as the structural basis in adaptive filters due to their
inherent stability. We will also only focus on those that use FIR filters.
4

Applications of Adaptive Filtering (1): Identification

The plant is unknown and we use the adaptive filter to obtain a mathematical
model that best fits, in some sense, the unknown plant. The input to the plant and
the filter is the same and the filter is suppose to adapt such that its output y, is
close to the plants output d.
u = input of adaptive filter and plant
y = output of adaptive filter
d = desired response = output of plant
e = d-y = estimation error

Applications of Adaptive Filtering (2): Inverse Modelling

The adaptive filter is to provide an inverse model that best represents (in some
sense) the unknown noisy plant. Ideally, the inverse model has a transfer function
equal to the reciprocal of the transfer function of the plant, such that the
combination of the two gives an ideal transmission medium.
u = output of plant and input to adaptive
filter
y = output of adaptive filter
d = desired response=delayed system
input
e = d-y = estimation error

Applications of Adaptive Filtering (3)

Inverse Modelling Example: Channel Equalisation

Applications of Adaptive Filtering (4): Interference Cancellation

Block Diagram of a Noise Reduction Headset
Near Microphone

d(n) = speech + noise

e(n)

Speech Output

Far Microphone

x(n) = noise'
Adaptive Filter

y(n)

Filter Output
(noise)

e(n)

Example: Headset for pilots. Cockpit has lots of noise eg engine noise which
interferes with pilots voices. Noise in far and near microphones maybe
slightly different but are correlated. The filter will produce output y(n) which
are best estimates of the noise picked up by near microphone. This is
subtracted from d(n) to give the desired speech.

Applications of Adaptive Filtering (5)

Interference Cancellation Example: ECG Signal
Example extracted from Rahman et al,
Signal Processing: An International Journal (SPIJ) Volume (3) : Issue (5)
(see Recommended Reading
In Moodle)

Applications of Adaptive Filtering (6): Prediction

The adaptive filter is used to estimate future values of a signal based on past
values of the signal.
u = input of adaptive filter=delayed version of random signal
y = output of adaptive filter
d = desired response=random signal
e = d-y = estimation error=system output

10

Wiener Filter (1)

Proposed by Norbert Wiener in 1940s.
In many applications, the desired signal (eg speech, image) is not readily
available. It may be corrupted by noise. Need to extract the desired signal
by filtering. Conventional filtering using LP, HP, BS or BP may not be
optimal especially if the environment is time varying.
Suppose in the block diagram below, u(n) = d(n) +v(n) where d(n) is the
desired signal and v(n) is the additive noise, the Wiener filter W(z) is
designed to recover the desired signal d(n), by producing y(n) the best
estimates of d(n).

u(n)

y(n)

11

Wiener Filter (2): FIR based

Other data
12

Wiener Filter (3)

PROBLEM STATEMENT
Suppose for the identification problem, we have the input samples going into the
unknown system un and the output (desired signal) dn where
uon

1n

Input signal : U n

u
( M 1) n

(stored in the delay line at time instant n)

Desired signal : d n d1 d 2
M 1

and the adaptive filter has the output signal

yn wmum n
m 0

Determine the filter weights :

weight vector :W T wo w1 wM 2 wM 1
13

Wiener Filter (4)

such that the mean square error (power of error signal) defined below, is
minimised

J E en

where the error signal is

en d n yn

The solution for the Wiener filter coefficients requires estimates of the
autocorrelation of the input signal and the cross-correlation of the input signal and
the desired signal.

14

Wiener Filter (5)

SOLUTION

In vector form yn U n W
T

Error signal

en d n yn d n U n W

Cost function

J E en E d n U n W

W W EU

E d n 2d nU k W U n W
2

E d 2 P W W
E d n 2 E d nU n
2
2

Un W

RW

15

Wiener Filter (6)

where R is the auto-correlation of the input signal and P is the cross-correlation
of the input signal and the desired signal:

u0 nu0 n

u1nu0 n

RE

u M 1nu0 n

u0 nu1n
u1nu1n

u M 1nu1n

u0 nu M 1n

u1nu M 1n

u M 1nu M 1n

d n u0 n
d u

n 1n

P E

d
u
n M 1n
16

Wiener Filter (7): Performance Criterion

Minimisation of the cost function

E en
:
0
W
2 RW 2 P 0
2

J min

RW P
W R 1 P Wiener Hopf equation

17

Wiener Filter (8): Performance Criterion

18

Example 1 Wiener Filter

Determine the optimal weights for the filter.

19

Points to note:
For pseudo-stationary signals such as speech signals, the filter
coefficients can be periodically recalculated and updated for
every block of say N input samples.
Choice of filter length? Must make a compromise too small
means filter may not be able to function properly; too large
results in computational complexity.
Depending upon the relationship between the input u(n) and
desired signal d(n), the filter can be used in various
applications.

20

Example 2: Channel Equalisation

Shown on next slide

Model of a noisy communication channel where d(n) is the data to be transmitted.

21

 The output x(n) is corrupted by additive white noise v(n)

of zero mean and variance 0.1, giving u(n) where:
u(n)=x(n)+v(n)
and x(n)-0.9458x(n-1)=d(n)
Assume x(n) and v(n) are uncorrelated
Variance of process d(n) is 0.9486
22

Determine the optimum weight for a Wiener filter and hence

the minimum mean square error.
Answer

23

The mean squared error J as a function of the weight w is:

Error Performance Surface

24

 Drawback of Wiener-Hopf solution

Requires calculation of an inverse of a matrix can be
computationally expensive.
Requires a priori information about input signal
(autocorrelation and crosscorrelation)

25

Least Mean Square (LMS) Filter

Introduced by Widrow and Stearns in 1985.
The LMS filter is an adaptive FIR filter which estimates the
filter weights, or coefficients, needed to minimise the
error, e(n), between the output signal y(n) and the desired
signal, d(n).
Each filter weight are updated on a sample by sample basis
(iteratively) based on e(n) such that the solution to the
Wiener-Hopf equation is approached.
Therefore, the LMS adaptive filter can be considered as an
adaptive realization of the Wiener filter and it is used when
the signal statistics are not (completely) known.
The basis of the filter is the use of the instantaneous
estimates of the gradient in the steepest descent method.
26

Gradient Steepest Descent Method (1)

A mathematical algorithm that minimises functions.
Given a function f(x) defined by a set of parameters x1, x2, ., gradient
descent starts with an initial set of parameter values and iteratively moves
toward a set of parameter values that minimises the function.
This iterative minimisation works by taking steps in the negative direction
of the function gradient.

27

Gradient Steepest Descent Method (2)

Update: W n 1 W n J
W n en 2
where is learning rate parameter or convergenc e factor
and J is gradient of instantaneous error
Since J en 2

dn U n W
W

2dn U nTW U kT 2enU n

2

W n 1 W n 2enU n
update value

of tap weight
vector

old value
learning tap

error

of tap weight rate

input
vector
parameter vector signal

28

Gradient Steepest Descent Method (3)

Important considerations are the learning rate parameter and initial
values for weights Wo.
The learning rate parameter determines how big a step downhill is taken
along the performance error surface. It determines the rate of
convergence how fast the algorithm produces the optimal weights
(when gradient should be zero and cost function is minimum).
If is small, it will require many iterations to arrive at minimum cost.
If is large, we may step over the minimum value and this may cause
oscillations about the minimum value.
It is possible to change the step size (adaptive ) at each iteration.
For stability, 0 < < 2/max where max is the maximum eigenvalue of the
autocorrelation matrix of the input data, R.
Wo can be determined by guessing (base on past experience) or by
calculation using known input signal.
29

The LMS Algorithm

The algorithm implements the following loop

Start with initial guess W o and select

Start of l oop
Get the next input sample u and perform the filtering
operation using previous weights yn

M 1

wmum n

m0

Compare the computed output with the desired output

to get error en d n yn
Update the weights W n 1 W n 2enU n
Repeat until minimum is reached
With each new sample, a new set of weights are calculated to adapt to changing signals
30

Example 3: LMS Filter with Gradient Descent Method

Consider the noise cancellation system as shown in the figure below.

The filter has two taps, w(0) and w(1) and the following initial values for the
primary signal and reference signal: d(0)=3, d(1)=-2, d(2)=1, u(0)=3, u(1)=1, u(2)=2. Using a learning rate of 0.1,
(i) write the following equations required for the weight updates:
y(n), e(n), w(0) and w(1)
(ii) hence determine the values for w(0) and w(1) for the first 2 iterations
(Similar question except for part (ii) which requires two interations only,
appears in Example Sheet 4. Please see solution posted in Moodle)
31

Advantages of LMS Filter

low computational complexity only requires local
information
simple to implement FIR based with compact algorithm
allow real-time operation track signals with changing
characteristics
does not need statistics of signals ie correlation based only
on samples of the input signal and the desired signal.

32

Summary
Adaptive filters are time-variant filters which adapts their weights to
minimise a cost function, most commonly power in an error signal, the
difference between the desired signal and the actual signal.
Adaptive filters are applicable for processing dynamically changing signals.
Optimal Wiener filter requires statistical properties of signals which are
usually not available in practice.
The LMS filter algorithm only requires samples of input signal and desired
signal. It seeks minimum cost optimal weights by estimating the gradient
of the instantaneous error and moving in the negative direction of the
gradient towards minimum cost. If learning rate parameter is carefully
chosen, the algorithm will approach the optimal Wiener solution.
FIR-based LMS filters are guaranteed to have unimodal performance error
surfaces only one minimum.
In general, using other adaptive filters, cost functions may have local
minima. If there are, the global minimum is reached by appropriate
selection of learning rate parameter and initial weight values. However,
what we have learnt serves as a basis for more practical variants of the
algorithm.
33

Recommended Reading
See papers on Examples of Adaptive Filter
Applications in Moodle: Recommended Reading.
See also a Simulink program (dspanc) for an example
of adaptive noise cancellation. (Type dspanc at
Matlab command prompt. Experiment with different
values of learning rate parameter). Block diagram is
also given in next slide

34

35