Adaptive Filters – Algorithms (Part 1)

Gerhard Schmidt
Christian-Albrechts-Universität zu Kiel
Faculty of Engineering
Electrical Engineering and Information Technology
Digital Signal Processing and System Theory
Contents of the Lecture

Today:

Exercises:
 Topics for the Talks

Adaptive Algorithms:
 Introductory Remarks
 Recursive Least Squares (RLS) Algorithm
 Least Mean Squares Algorithm (LMS Algorithm) – Part 1
 Least Mean Squares Algorithm (LMS Algorithm) – Part 2
 Affine projection Algorithm (AP Algorithm)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 2
Adaptive Filters – Talks

Possible Topics

Suggestions:
 Hearing aids
 GSM (source) coding
 Localization and tracking
 Active noise control (anti-noise)
 Noise suppression
 Bandwidth extension
 Audio upmix of stereo signals
 Adaptive beamforming
 MPEG audio coding
 Non-linear echo cancellation
 Adaptation of neural networks
 Feedback suppression
…
Your own topic suggestions are
welcome …

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 3
Adaptive Filters – Algorithms

Contents

Exercises:
 Topics for the Talks

Adaptive Algorithms:
 Introductory Remarks
 Recursive Least Squares (RLS) Algorithm
 Least Mean Squares Algorithm (LMS Algorithm) – Part 1
 Least Mean Squares Algorithm (LMS Algorithm) – Part 2
 Affine projection Algorithm (AP Algorithm)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 4
Introductory Remarks

Motivation

Why adaptive filters?

 Signal properties are not known in advance or are time variant.
 System properties are not known in advance or time variant.

Examples:
 Speech signals
 Mobile telephone channels

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 5
Introductory Remarks

Literature

Books:
 E. Hänsler, G. Schmidt: Acoustic Echo and Noise Control, Wiley, 2004
 S. Haykin: Adaptive Filter Theory, Prentice Hall, 2002
 A. Sayed: Fundamentals of Adaptive Filtering, Wiley, 2004
 E. Hänsler: Statistische Signale: Grundlagen und Anwendungen, Springer, 2001
(in German)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 6
Introductory Remarks

Two Hook-Ups of Adaptive Filters

Adaptive filter for channel equalization:

Transmission Adaptive
channel filter

Adaptive filter for system identification:

System

Adaptive
filter

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 7
Introductory Remarks

Application Examples – Part 1

Adaptive filter for cancellation of hybrid echoes:

Adaptive
Hybrid
filter

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 8
Introductory Remarks

Application Examples – Part 2

Adaptive filter for noise reduction with reference signal:

Noisy
Signal signal
source

Transmission
path 1

Noise Adaptive
source filter

Transmission
path 2
Reference
signal
Signal model

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 9
Introductory Remarks

Application Examples – Part 3

Antenna array:

Adaptive Adaptive Adaptive

filter 1 filter 2 filter N

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 10
Introductory Remarks

Application Examples – Part 4

Adaptive equalization without reference signal

Adaptive
filter

Decision
circuit

Assumptions:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 11
Introductory Remarks

Generic Setup

Desired
output
signal

Adaptive filter

Adaptive
algorithm

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 12
Introductory Remarks

Structure of an Adaptive FIR Filter

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 13
Introductory Remarks

Error Measures – Part 1

No local noise

Mean square (signal) error:

System distance:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 14
Introductory Remarks

Mean Square Error and System Distance

Relation of the normalized mean square (signal) error power and the system distance:

Let be white noise:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 15
Introductory Remarks

Adaptation

Local noise

Basic principle:

New = old + correction +

+

Properties:
 „Correction“ depends on the input signal and the error signal .
 Procedures differ by the functions and :

Step size

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 16
Introductory Remarks

Error Measures
Local noise

+
+
Three error measures control the adaptation:
 Coefficient error

 A priori error

 A posteriori error:

Old data New filter

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 17
Adaptive Filters - Algorithms

Contents

Exercises:
 Topics for the Talks

Adaptive Algorithms:
 Introductory Remarks
 Recursive Least Squares (RLS) Algorithm
 Least Mean Squares Algorithm (LMS Algorithm) – Part 1
 Least Mean Squares Algorithm (LMS Algorithm) – Part 2
 Affine projection Algorithm (AP Algorithm)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 18
Recursive Least Squares (RLS) Algorithm

Algorithmic Properties

Attributes of the RLS algorithm:

 No a priori knowledge of signal statistics is required.
 Optimization criterion is the (weighted) sum of squared errors.

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 19
Recursive Least Squares (RLS) Algorithm

Error Criterion

Signal Filter

Filter
filter
Forgetting factor

Alternative:
Adaptation
adaptive
algorithm
algorithm
error:
Filter at time n

Signal at time l

error inserted:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 20
Recursive Least Squares (RLS) Algorithm

Derivation – Part 1

Cost function:

Differentiate with respect to the complex filter coefficients and setting the result to zero:

Definitions:

… Estimate for the auto correlation matrix

… Estimate for the cross correlation vector

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 21
Recursive Least Squares (RLS) Algorithm

Derivation – Part 2

From Simon Haykin, „Adaptive Filter Theory“, Prentice Hall, 2002:

or: The Matrix Cookbook [ http://matrixcookbook.com ]

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 22
Recursive Least Squares (RLS) Algorithm

Derivation – Part 3

Filter
filter

Adaptation
adaptive
algorithm
algorithm

Inserting the results leads to:

„Wiener solution“
… assuming that the auto correlation matrix is invertable

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 23
Recursive Least Squares (RLS) Algorithm

Recursion – Part 1

Recursion of the auto correlation matrix over time:

Recursion of the cross correlation vector over time:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 24
Recursive Least Squares (RLS) Algorithm

Recursion – Part 2

Recursion for the auto correlation matrix:

Matrix Inversion Lemma:

Inserting the Lemma in the recursion:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 25
Recursive Least Squares (RLS) Algorithm

Recursion – Part 3

Recursion for the auto correlation matrix:

Definition of a gain vector:

Inserting this definition leads to:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 26
Recursive Least Squares (RLS) Algorithm

Recursion – Part 4

Definition of a gain factor:

Multiplication by the denominator on the right hand side leads to:

Rewriting leads to:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 27
Recursive Least Squares (RLS) Algorithm

Recursion – Part 5

Recursion of the filter coefficient vector:

Step from n to n+1:

Reducing the right hand side:

Inserting the recursion of the cross correlation vector leads to:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 28
Recursive Least Squares (RLS) Algorithm

Recursion – Part 6

What we have so far:

If we insert the recursive computation of the inverse auto correlation matrix

we obtain:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 29
Recursive Least Squares (RLS) Algorithm

Recursion – Part 7

What we have so far:

Inserting according to

results in

Gain factor Error: old filter with new data

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 30
Recursive Least Squares (RLS) Algorithm

Adaptation Rule – Part 1

Inserting previous results:

Adaptation rule for the filter coefficients according to the RLS algorithm:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 31
Recursive Least Squares (RLS) Algorithm

Summary

Computing a preliminary gain vector (complexity prop. N²):

Update of the inverse auto correlation matrix (complexity prop. N²):

Computing the error signal (complexity prop. N):

Update of the filter vector (complexity prop. N):

Step size (0 … 1), will be treated later …

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 32
Adaptive Filters – Algorithms

Contents

Exercises:
 Topics for the Talks

Adaptive Algorithms:
 Introductory Remarks
 Recursive Least Squares (RLS) Algorithm
 Least Mean Squares Algorithm (LMS Algorithm) – Part 1
 Least Mean Squares Algorithm (LMS Algorithm) – Part 2
 Affine projection Algorithm (AP Algorithm)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 33
Least Mean Square (LMS) Algorithm

Basics – Part 1

Optimization criterion:
 Minimizing the mean square error

Assumptions:
 Real, stationary random processes

Structure:

Unknown system

Adaptive
filter

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 34
Least Mean Square (LMS) Algorithm

Basics – Part 2

Unknown system

Output signal of the adaptive filter:

Adaptive
filter

Error signal:

Mean square error:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 35
Least Mean Square (LMS) Algorithm

Basics – Part 3

Mean square error:

The filter coefficients are adjusted optimally in case of orthogonality:

Abbreviations:
(auto correlation matrix)

(cross correlation vector)

Solution (according to Wiener):

(assuming that the inverse exists)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 36
Least Mean Square (LMS) Algorithm

Basics – Part 4

Mean square error:

Optimal filter vector:

Minimum mean square error:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 37
Least Mean Square (LMS) Algorithm

Basics – Part 5

Mean square error:

Minimum mean square error :

Mean square error:

Quadratic form unique minimum

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 38
Least Mean Square (LMS) Algorithm

Derivation – Part 1

Derivation with respect to the coefficients of the adaptive filter:

… needed later on …

Inserting , results in:

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 39
Least Mean Square (LMS) Algorithm

Derivation – Part 2

What we have so far:

Resolving it to leads to:

With the introduction of a step size , the following adaptation rule can be
formulated:

Method according to Newton

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 40
Least Mean Square (LMS) Algorithm

Derivation – Part 3

Method according to Newton:

Method of steepest descent:

For practical approaches the expectation value is replaced by its instanteneous

value. This leads to the so-called least mean square algorithm:

LMS algorithm

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 41
Adaptive Filters – Algorithms

Summary and Outlook

This week:
 Topics for the Talks
 Introductory Remarks
 Recursive Least Squares (RLS) Algorithm
 Least Mean Squares Algorithm (LMS Algorithm) – Part 1

Next week:

 Least Mean Squares Algorithm (LMS Algorithm) – Part 2

 Affine projection Algorithm (AP Algorithm)

Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 42