2.161 Signal Processing: Continuous and Discrete: Mit Opencourseware
2.161 Signal Processing: Continuous and Discrete: Mit Opencourseware
2.161 Signal Processing: Continuous and Discrete: Mit Opencourseware
http://ocw.mit.edu
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
1 Introduction
In this handout we introduce the concepts of adaptive recursive-least-squares (RLS) FIR filters,
where the coefficients are continually adjusted on a step-by-step basis during the filtering operation.
The filter structure shown in Fig. 1 is similar to the least-mean-squares (LMS) filter described in
the handout Introduction to Least-Squares Adaptive Filters, but differs in the internal algorithmic
structure.
Like the LMS filter, the RLS filter is FIR of length M with coefficients bk , k = 0, 1, 2, . . . , M −1.
The input stream {f (n)} is passed through the filter to produce the sequence {y(n)}. At each time-
step the filter coefficients are updated using an error e(n) = d(n) − y(n) where d(n) is the desired
response (usually based of {f (n)}).
The LMS filter is implicitly designed around ensemble statistics, and uses a gradient descent
method based on expected values of the waveform statistics to seek optimal values for the filter
coefficients. On the other hand, the RLS filter computes the temporal statistics directly at each
time-step to determine the optimal filter coefficients. The RLS filter is adaptive and can adjust
to time varying input statistics. Under most conditions the RLS filter will converge faster than a
LMS filter.
d (n )
R e c u r s iv e - L e a s t- S q u a r e s
A lg o r ith m
n
� � �2
= d (i) − y (i) (2)
i=0
√ √
where d (i) = λn−i d(i), and y (i) = λn−i y(i). The purpose of the factor λ is to weight recent
data points more heavily, and thus allow the filter to track changing statistics in the input data.
The FIR filter output is given by the convolution sum
M
� −1
y(n) = bk f (n − k) (3)
k=0
or
b(n) = R−1 (n)P(n), (7)
where
b(n) = [b0 (n) b1 (n) b2 (n) ... bM −1 (n)]T
is a M × 1 column vector of the filter coefficients,
n
�
R(n) = λn−i f (i)f T (i), (8)
i=0
is a M × M matrix, and
is a M × 1 column vector.
2
Notice the similarity between these definitions of R(n) and P(n) in Eqs. (8) and (9) to the
correlation matrix and cross-correlation vector in least-mean-squares (LMS) filter design. In this
case however, the use of the weighting factor λ destroys the Toeplitz nature of the matrix R(n),
and efficient numerical inversion methods are not available.
The RLS design algorithm does not attempt to solve Eq. (7) at each time-step, which would be
impractical, requiring repeated inversion of R(n). Instead, the method uses an iterative algebraic
procedure to find the updated inverse of R(n) using the result from the previous step. Equation
(8) shows that R(n) may be computed recursively:
However, we need a recursive definition of R−1 (n) based on R−1 (n − 1) to compute the filter
coefficients in Eq, (7).
The matrix inversion lemma (see the Appendix) states that if a square non-singular n × n
matrix A, with known inverse A−1 , is updated with an additive term the new inverse is given by
� �−1 � �−1
A + BCBT = A−1 − A−1 B C−1 + BT A−1 B BT A−1 (11)
which is an algebraic recursion relationship that allows computation of R−1 (n) from the result of
the previous time-step and the current input history vector f (n). Note that no matrix inversion is
required.
In addition, from Eq. (9) P(n) may be defined recursively
so that Eqs.(12) and (13), when combined with Eq. (7), form the basis for the RLS filter design.
If we define an M -dimensional vector of Kalman gains
The filter update equation, Eq. (7), may then be written as a recursion
We also note that y(n) = f T (n)b(n − 1) is the convolution sum generating the filter output y(n)
using the previous set of filter coefficients, and therefore
are the recursive filter update equations in terms of the Kalman gains and the filter output error.
2. Compute the filter output using the previous set of filter coefficients b(n − 1)
y(n) = f T (n)b(n − 1)
and therefore true initialization of R−1 (0) requires knowledge of the the input history for n < 0.
Further, for small values of n, when the statistics are unreliable, there is a risk that R(n) may
become singular. The usual convention is to define
R−1 (0) = δI
where δ is a positive scalar, to ensure that R(n) is well behaved for small n. As n increases the
effect of this initialization error will decrease because of the effect of the weighting factor λ. It
may be necessary to determine a suitable value of δ by experimentation, balancing stability with
convergence rate, but a guideline that is often given is that
δ > 100σf2
where σ
f2 is the variance of the input.
N =
�k∞ k
=
k=0 λ 1 − λ
Common values used are between 0.95 < λ < 0.995, giving 19 < N < 199.
5
% [y, e] = adaptfir(’initial’, .95, 51, 0.01);
% Note: RLSFilt returns y = 0 for initialization
% 2) Filtering:
% [y, b] = RLSFilt(f, d};
% where f is a single input value,
% d is the desired value, and
% y is the computed output value,
% b is the coefficient vector.
%
% Version: 1.0
% Author: D. Rowell 12/9/07
% ------------------------------------------------------------------------
%
function [y, Bout] = RLSFilt(f, d, FIR_M, delta_n )
persistent F B lambda delta M Rinv
% The following is initialization, and is executed once
if (ischar(f) && strcmp(f,’initial’))
lambda = d;
M = FIR_M;
delta = delta_n;
F = zeros(M,1);
Rinv = delta*eye(M);
B = zeros(M,1);
y = 0;
else
% Filtering:
for J = M:-1:2
F(J) = F(J-1);
end;
F(1) = f;
% Perform the convolution
y= F’*B;
error = d - y;
% Kalman gains
K = Rinv*F/(lambda + F’*Rinv*F);
% Update Rinv
Rinvn = (Rinv - K*F’*Rinv)/lambda;
% Update the filter coefficients
B = B + K*error;
Bout=B;
Rinv = Rinvn;
end
4 Examples
4.1 Example 1: Demonstration of Convergence with a Sinusoidal Input
In the handout MATLAB Examples of Least-Squares FIR Filter Design, example we examined a
static least-squares filter design for a case described by Stearns and Hush: a one-step predictor of
a sine wave with with filter lengths M = 2 and M = 3. We also examined this example in the
� �
2πn
f (n) = sin .
12
The one-step linear predictor structure is shown in Fig. 2. The following MATLAB script was
used to examine the ability of the RLS algorithm to act as a one step predictor:
d (n ) = f(n )
f(n -1 ) R L S lin e a r F IR filte r y (n ) +
f(n ) Z
- 1
H (z ) - e rro r
d e la y e (n )
L = 5000;
f=zeros(1,L);
for J=1:L
f(J) = sin(2*pi*J/12);
end
%f = f +0.001*randn(1,length(f));
x = RLSFilt(’initial’,lambda,M, delta);
Delta = 1;
f_delay = zeros(1,Delta+1);
% Filter
for J = 1:length(f)
for K = Delta+1:-1:2
f_delay(K) = f_delay(K-1);
end
f_delay(1) = f(J);
d = f_delay(Delta+1);
[y,b] = RLSFilt(f_delay(Delta+1),f(J));
end;
% Report the final filter coefficients
b
The script was modified and run with M = 2 and 3, and with various values of λ.
The values reported for the filter coefficients with M = 2 were
For M = 3 the RLS algorithm failed and no sensible values were returned. As Stearns and Hush
note, there is no unique solution for the coefficients for M = 3, and the optimal filter coefficients
must satisfy the conditions:
√ √
b(0) − b(2) = 3, b(0) + 3b(1) + 2b(2) = 0
Under these conditions the matrix R(n) is singular. However, when a small random component
was added to the sinusoidal input sequence (see the script), the matrix R(n) remains invertible,
and the values returned were
b(0) = 1.1232, b(1) = 0.0545, b(2) = −0.6088
which satisfy the above conditions. The LMS filter algorithm does not suffer from this problem.
R L S A lg o r ith m
1000 10
800 8
Magnitude
Magnitude
600 6
400 4
200 2
0 0
−3.14 −1.57 0 1.57 3.14 −3.14 −1.57 0 1.57 3.14
Normalized frequency Normalized frequency
9
for K = Delta+1:-1:2
f_delay(K) = f_delay(K-1);
end
f_delay(1) = f(J);
[y(J),b] = RLSFilt(f_delay(Delta+1),f(J));
end;
% Compute the overall filter coefficients
% H(z) = 1 - z^{-Delta}H_{RLS}(z)
b_overall = [1 zeros(1,Delta-1) -b’];
% Find the frequency response
[H,w] = freqz(b_overall,1);
figure(1);
plot(w,20*log10(abs(H)));
figure(2);
zplane(b_overall,1)
The input and output spectra are shown in Fig. 5. The filter frequency response magnitude and
the pole-zero plot of the filter are shown in Fig. 7. The adaptive algorithm has clearly generated a
notch-filter covering the bandwidth of the interference.
7 7
6 6
5 5
Magnitude
Magnitude
4 4
3 3
2 2
1 1
0 0
0 1 2 3 0 1 2 3
Normalized angular frequency Normalized angular frequency
Figure 5: Input and output spectra from a RLS suppression filter with interference in the band
0.3 < Ω < 0.6.
10
0
Frequency response magnitude (dB)
−5
−10
−15
−20
0 0.5 1 1.5 2 2.5 3
Normalized frequency
Figure 6: Frequency response magnitude plot of a RLS suppression filter with interference in the
band 0.3 < Ω < 0.6.
0.8
0.6
0.4
Imaginary Part
0.2
31
0
−0.2
−0.4
−0.6
−0.8
−1
−1 −0.5 0 0.5 1
Real Part
Figure 7: Pole-zero plot of a RLS suppression filter with interference in the band 0.3 < Ω < 0.6.
11
The matrix inversion lemma, also known as the Woodbury matrix identity states that for a square
n × n matrix A with a known inverse A−1 , a new updated inverse may be formed
� �−1 � �−1
A + BCBT = A−1 − A−1 B C−1 + BT A−1 B BT A−1
= I,
12