Lectures on Dynamic Systems and

Control

Mohammed Dahleh Munther A. Dahleh George Verghese

Department of Electrical Engineering and Computer Science
Massachuasetts Institute of Technology1

1�
c
Chapter 2

Least Squares Estimation

2.1 Introduction
If the criterion used to measure the error e � y ; Ax in the case of inconsistent system of
equations is the sum of squared magnitudes of the error components, i.e. e0 e, or equivalently
the square root of this, which is the usual Euclidean norm or 2-norm kek2 , then the problem
is called a least squares problem. Formally it can be written as
x ky ; Axk2 :
min (2.1)
The x that minimizes this criterion is called the least square error estimate, or more simply,
the least squares estimate. The choice of this criterion and the solution of the problem go
back to Legendre (1805) and Gauss (around the same time).

Example 2.1 Suppose we make some measurements yi of an unknown function

f (t) at discrete points ti� i � 1� : : : � N :
yi � f (ti) � i � 1� : : : � N :
We want to �nd the function g(t) in the space � of polynomials of order m ; 1 �
N ; 1 that best approximates f (t) at the measured points ti, where
( mX
;1 )
� � g(t) � �i ti� �i real
i�0

For any g(t) 2 �, we will have yi � g(ti ) + ei for i � 1� : : : � N . Writing this in

 matrix form for the available data, we have
2 y1 3 2 1 t1 t2 � � � tm;1 32 3 2 e1 3
75 � 66 ... 1 1 77 6 �.. 0

64 ... .. 75 + 64 ... 75
4 . 54 .

m;1
} | 1 tN tN{z � � � tN

| y{zN � e
2

y
} | m{zx;1 } | {zeN }
A

X N
The problem is to �nd �0 � : : : � �m;1 such that e0 e � e2i is minimized.
i;1

2.2 Computing the Estimate

The solution, x^, of Equation 2.1 is characterized by:
(y ; Ax^) � R(A):
All elements in a basis of R(A) must be orthogonal to (y ; Ax^). Equivalently this is true for
the set of columns of A, [a1 � : : : � an ]. Thus
(y ; Ax^) � R(A) , a0i (y ; Ax^) � 0 for i � 1� : : : � n
, A0 (y ; Ax^) � 0
, A0 Ax^ � A0y
This system of m equations in the m unknowns of interest is referred to as the normal
equations. We can solve for the unique x^ i� A0 A is invertible. Conditions for this will be
derived shortly. In the sequel, we will present the generalization of the above ideas for in�nite
dimensional vector spaces.

2.3 Preliminary: The Gram Product

Given the array of nA vectors A � [a1 j � � � j anA ] and the array of nB vectors B � [b1 j � � � j bnB ]
from a given inner product space, let � A� B � denote the nA � nB matrix whose (i� j )-th
element is � ai � bj �. We shall refer to this object as the Gram product (but note that this
terminology is not standard!).
If the vector space under consideration is Rm or Cm , then both A and B are matrices
with m rows, but our de�nition of � A� B � can actually handle more general A, B . In
fact, the vector space can be in�nite dimensional, as long as we are only examining �nite
collections of vectors from this space. For instance, we could use the same notation to treat
�nite collections of vectors chosen from the in�nite-dimensional vector space L2 of square
 R1 a2 (t) dt � 1. The inner product in L2
integrable functions,R i.e. functions a(t) for which ;1
is � a(t)� b(t) �� ;1 1 a� (t)b(t) dt. (The space L2 is an example of an in�nite dimensional
Hilbert space, and most of what we know for �nite dimensional spaces | which are also Hilbert
spaces! | has natural generalizations to in�nite dimensional Hilbert spaces. Many of these
generalizations involve introducing notions of topology and measure, so we shall not venture
too far there. It is worth also mentioning here another important in�nite dimensional Hilbert
space that is central to the probabilistic treatment of least squares estimation: the space
of zero-mean random variables, with the expected value E (ab) serving as the inner product
� a� b �.)
For the usual Euclidean inner product in an m-dimensional space, where � ai � bj ��
a0i bj , we simply have � A� B � � A0B . For the inner product de�ned by � ai � bj � � a0i S bj
for a positive de�nite, Hermitian matrix S , we have � A� B � � A0 SB .
� Verify that the symmetry and linearity of the inner product imply the same for the
Gram product, so � AF� BG + CH � � F 0 � A� B � G + F 0 � A� C � H , for any

constant matrices F , G, H (a constant matrix is a matrix of scalars), with A, B , C

denoting arrays whose columns are vectors.

2.4 The Least Squares Estimation Problem

The problem of interest is to �nd the least square error (LSE) estimate of the parameter vector
x that arises in the linear model y � Ax, where A is an array of n vectors, A � [a1 � � � � � an ].
De�ning the error e by
e � y ; Ax
what we want to determine is
xb � arg min
x kek � arg minx ky ; Axk � y� A given
(where \arg minx " should be read as \the value of the argument x that minimizes"). To state
this yet another way, note that as x is varied, Ax ranges over the subspace R(A), so we are
looking for the point
yb � Axb
in R(A) that comes closest to y, as measured by whatever norm we are using.
Rather than restricting the norm in the above expression to be the Euclidean 2-norm
used in Lecture
p 1, we shall now actually permit it to be any norm induced by an inner product,
so kek � � e� e �. This will allow us to solve the so-called weighted least squares problem
in a �nite dimensional space with no additional work, because error criteria of the form
e0 Se for positive de�nite Hermitian S are thereby included. Also, our problem formulation
then applies to in�nite dimensional spaces that have an inner product de�ned on them, with
the restriction that our model Ax be con�ned to a �nite dimensional subspace. This actually
covers the cases of most interest to us� treatment of the more general case involves introducing
further topological notions (closed subspaces, etc.), and we avoid doing this.
 We shall also assume that the vectors ai � i � 1� : : : � n in A are independent. This
assumption is satis�ed by any reasonably parametrized model, for otherwise there would be
an in�nite number of choices of x that attained any achievable value of the error y ; Ax. If
the vectors in A are discovered to be dependent, then a re-parametrization of the model is
needed to yield a well-parametrized model with independent vectors in the new A. (A subtler
problem | and one that we shall say something more about in the context of ill-conditioning
and the singular value decomposition | is that the vectors in A can be nearly dependent,
causing practical di�culties in numerical estimation of the parameters.)
Gram Matrix Lemma
An important route to verifying the independence of the vectors that make up the columns of
A is a lemma that we shall refer to as the Gram Matrix Lemma. This states that the vectors
in A are independent i� the associated Gram matrix (or Gramian) � A� A �� [� ai � aj �]
is invertible� all norms are equivalent, as far as this result is concerned | one can pick any
norm. As noted above, for the case of the usual Euclidean inner product, � A� A � � A0 A. For
an inner product of the form � ai � aj � � a0i Saj , where S is Hermitian and positive de�nite,
we have � A� A �� A0 SA. The lemma applies to the in�nite dimensional setting as well (e.g.
L2), provided we are only considering the independence of a �nite subset of vectors.
If the vectors in A arePdependent, there is some nonzero vector � such that A� �
Proof: P
j aj �j � 0. But then j � ai � aj � �j � 0, by the linearity of the inner product� in
matrix form, we can write � A� A � � � 0 | so � A� A � is not invertible.
Conversely, if � A� A � is not invertible, then � A� A � � � 0 forPsome nonzero
P a � ��.�But0,
then �0 � A� A � � � 0, so by th e linearity of inner products � � a �
i.e. the norm of the vector P aj �j � A� is zero, so the vectors in A are dependent.
i i j j

2.5 The Projection Theorem and the Least Squares Estimate

The solution to our least squares problem is now given by the Projection Theorem, also referred
to as the Orthogonality Principle, which states that
eb � (y ; Axb) � R(A)
from which - | as we shall see | xb can be determined. In words, the theorem/\principle"
states that the point yb � Axb in the subspace R(A) that comes closest to y is characterized
by the fact that the associated error eb � y ; yb is orthogonal to R(A), i.e., orthogonal to the
space spanned by the vectors in A. This principle was presented and proved in the previous
chapter. We repeat the proof here in the context of the above problem.
Proof: We �rst show that y has a unique decomposition of the form y � y1 +y2 , where y1 2 R(A)
and y2 2 R�(A). We can write any y1 2 R(A) in the form y1 � A� for some vector �.
If we want (y ; y1 ) 2 R�(A), we must see if there is an � that satis�es
� ai � (y ; A�) � � 0 � i � 1� : : : � n
or, using our Gram product notation,
� A� (y ; A�) � � 0
Rearranging this equation and using the linearity of the Gram product, we get
� A� A � � � � A� y �
which is in the form of the normal equations that we encountered in Lecture 1. Under
our assumption that the vectors making up the columns of A are independent, the Gram
matrix lemma shows that � A� A � is invertible, so the unique solution of the preceding
equation is
� � � A� A �;1 � A� y �
We now have the decomposition that we sought.
To show that the preceding decomposition is unique, let y � y1a + y2a be another such
decomposition, with y1a 2 R(A) and y2a 2 R�(A). Then
y1 ; y1a � y2 ; y2a
and the left side is in R(A) while the right side is in its orthogonal complement. It is
easy to show that the only vector common to a subspace and its orthogonal complement
is the zero vector, so y1 ; y1a � 0 and y2 ; y2a � 0, i.e., the decomposition of y is unique.
To proceed, decompose the error e � y ; Ax similarly (and uniquely) into the sum of
e1 2 R(A) and e2 2 R�(A). Note that
kek2 � ke1 k2 + ke2 k2
Now we can rewrite e � y ; Ax as
e1 + e2 � y1 + y2 ; Ax
or
e2 ; y2 � y1 ; e1 ; Ax
Since the right side of the above equation lies in R(A) and the left side lies in R�(A),
each side separately must equal 0 | again because this is the only vector common to
a subspace and its orthogonal complement. We thus have e2 � y2 , and the choice of x
can do nothing to a�ect e2 . On the other hand, e1 � y1 ; Ax � A(� ; x), and the best
we can do as far as minimizing kek2 is to make e1 � 0 by choosing x � �, so xb � �, i.e.,
 xb � � A� A �;1 � A� y �

This solves the least squares estimation problem that we have posed.
The above result, though rather abstractly developed, is immediately applicable to many
concrete cases of interest.
� Specializing to the case of Rm or Cm, and choosing x to minimize the usual Euclidean
norm,

X
m
kek2 � e0 e � jeij2
i�1
we have
xb � (A0 A);1 A0y

Note that if the columns of A form a mutually orthogonal set (i.e. an orthogonal basis
for R(A)), then A0 A is diagonal, and its inversion is trivial.
� If instead we choose to minimize e0Se for some positive de�nite Hermitian S (�6 I ), we
have a weighted least squares problem, with solution given by

xb � (A0 SA);1 A0Sy

For instance, with a diagonal S , the criterion that we are trying to minimize becomes
X
m
siijei j2
i�1
where the sii are all positive. We can thereby preferentially weight those equations in
our linear system for which we want a smaller error in the �nal solution� a larger value
of sii will encourage a smaller ei .
Such weighting is important in any practical situation, where di�erent measurements yi
may have been subjected to di�erent levels of noise or uncertainty. One might expect
that sii should be inversely proportional to the noise intensity on the ith equation. In
fact, a probabilistic derivation, assuming zero-mean noise on each equation in the system
but noise that is uncorrelated across equations, shows that sii should vary inversely with
the variance of ei .
A full matrix S rather than a diagonal one would make sense if the errors were correlated
across measurements. A probabilistic treatment shows that the proper weighting matrix
is S � (E [ee0 ]);1 , the inverse of the covariance matrix of e. In the deterministic setting,
one has far less guidance on picking a good S .
 � The boxed result also allows us to immediately write down the choice of coe�cients xi
that minimizes the integral
Z
[ y(t) ; a1 (t)x1 ; a2 (t)x2 ; � � � ; an (t)xn ]2 dt

for speci�ed functions y(t) and ai (t). If, for instance, y(t) is of �nite extent (or �nite
\support") T , and the ai (t) are sinusoids whose frequencies are integral multiples of
2��T , then the formulas that we obtain for the xi are just the familiar Fourier series
expressions. A simpli�cation in this example is that the vectors in A are orthogonal, so
� A� A � is diagonal.
2.6 Recursive Least Squares (optional)
What if the data is coming in sequentially� Do we have to recompute everything each time
a new data point comes in, or can we write our new, updated estimate in terms of our old
estimate�
Consider the model
yi � Aix + ei � i � 0� 1� : : : � (2.2)
where yi 2 Cm�1 , Ai 2 Cm�n , x 2 Cn�1 , and ei 2 Cm�1 . The vector ek represents the
mismatch between the measurement yk and the model for it, Ak x, where Ak is known and x
is the vector of parameters to be estimated. At each time k, we wish to �nd
�X
k
! �X
k !
xbk � arg min
x (yi ; Ai x)0i Si (yi ; Ai x) � arg mxin e0 Siei
i � (2.3)
i�1 i�1
where Si 2 Cm�m is a positive de�nite Hermitian matrix of weights, so that we can vary the
importance of the ei 's and components of the ei 's in determining xbk .
To compute xbk+1 , let:
2 y0
3 2 A0
3 2 e0
3
66 y1 77 66 A1 77 66 e1 77
yk+1 � 666 : 77 �
75 Ak+1 � 666 : 77 �
75 ek+1 � 666 : 77 �
75
4 : 4 : 4 :

yk+1 Ak+1 ek+1

and
S k+1 � diag (S0 � S1 � : : : � Sk+1 )
where Si is the weighting matrix for ei .
Our problem is then equivalent to
 min(e0k+1 S k+1 ek+1 )
subject to: yk+1 � Ak+1 xk+1 + ek+1
The solution can thus be written as
(A0k+1S k+1 Ak+1 )xbk+1 � A0k+1 S k+1yk+1
or in summation form as
�kX
+1 ! kX
+1
A0 Si Ai
i xbk+1 � A0iSiyi
i�0 i�0
De�ning
kX
+1
Qk+1 � A0iSiAi :
i�0
we can write a recursion for Qk+1 as follows:
Qk+1 � Qk + A0k+1Sk+1Ak+1:
Rearranging the summation form equation for xbk+1 , we get
h� �
1 Pk A0 S A xb + A0 S y
i
xbk+1 � Q;k+1 i�0 i i i k k+1 k +1 k +1
h i
� Q;k+1
1 Q xb + A0 S y
k k k+1 k+1 k+1
This clearly displays the new estimate as a weighted combination of the old estimate and the
new data, so we have the desired recursion. Another useful form of this result is obtained by
substituting from the recursion for Qk+1 above to get
1 ;A0 S A xb ; A0 S y � �
xbk+1 � xbk ; Q;k+1 k+1 k+1 k+1 k k+1 k+1 k+1
which �nally reduces to
xbk+1 � xbk + Q;k+1 k+1 k+1 (|yk+1 ;{zAk+1 xbk})
1 A0 S
| {z } innovations
Kalman Filter Gain
The quantity Q;k+1 k+1 k+1 is called the Kalman gain, and yk+1 ; Ak+1 xbk is called the
1 A0 S
innovations, since it compares the di�erence between a data update and the prediction given
the last estimate.
Unfortunately, as one acquires more and more data, i.e. as k grows large, the Kalman gain
goes to zero. One data point cannot make much headway against the mass of previous data
which has `hardened' the estimate. If we leave this estimator as is|without modi�cation|the
estimator `goes to sleep' after a while, and thus doesn't adapt well to parameter changes. The
homework investigates the concept of a `fading memory' so that the estimator doesn't go to
sleep.
An Implementation Issue
Another concept which is important in the implementation of the RLS algorithm is the com-
putation of Q;k+1
1 . If the dimension of Q is very large, computation of its inverse can b e
k
computationally expensive, so one would like to have a recursion for Q;k+1
1 .
This recursion is easy to obtain. Applying the handy matrix identity

� �;1

(A + BCD);1 � A;1 ; A;1 B DA;1 B + C ;1 DA;1

to the recursion for Qk+1 yields
� �;1
Q;k+1
1 � Q;1 ; Q;1 A0
k
;1 0 ;1
k k+1 Ak+1 Qk Ak+1 + Sk+1 Ak+1Q;k 1 :
Upon de�ning
Pk+1 � Q;k+1
1 �

this becomes � �;1

Pk+1 � Pk ; Pk A0k+1 Sk;+11 + A P A0
k+1 k k+1 Ak+1 Pk :
which is called the (discrete-time) Riccati equation.
Interpretation
We have xbk and yk+1 available for computing our updated estimate. Interpreting xbk as a
measurement, we see our model becomes
" # " # " #
xbk I ek
yk+1 � Ak+1 x + ek+1 :
The criterion, then, by which we choose xbk+1 is thus
; �
xbk+1 � argmin e0k Qk ek + e0k+1Sk+1ek+1 :
In this context, one interprets Qk as the weighting factor for the previous estimate.
 Exercises
Exercise 2.1 Least Squares Fit of an Ellipse
Suppose a particular object is modeled as moving in an elliptical orbit centered at the origin.
Its nominal trajectory is described in rectangular coordinates (r� s) by the constraint equation x1 r2 +
x2 s2 + x3 rs � 1, where x1 , x2 , and x3 are unknown parameters that specify the orbit. We have
available the following noisy measurements of the object's coordinates (r� s) at ten di�erent points on
its orbit:
(0.6728, 0.0589) (0.3380, 0.4093) (0.2510, 0.3559) (-0.0684, 0.5449)
(-0.4329, 0.3657) (-0.6921, 0.0252) (-0.3681, -0.2020) (0.0019, -0.3769)
(0.0825, -0.3508) (0.5294, -0.2918)
The ten measurements are believed to be equally reliable. For your convenience, these ten pairs of
measured (r� s) values have been stored in column vectors named r and s that you can access through
the 6.241 locker on Athena. After add 6.241, and once in the directory in which you are running
Matlab, you can copy the data using cp /mit/6.241/Public/fall95/hw1rs.mat hw1rs.mat. Then,
in Matlab, type load hw1rs to load the desired data� type who to con�rm that the vectors r and s are
indeed available.
Using the assumed constraint equation, we can arrange the given information in the form of the
linear system of (approximate) equations Ax � b, where A is a known 10 � 3 matrix, b is a known 10 � 1
vector, and x � ( x1 � x2 � x3 )T . This system of 10 equations in 3 unknowns is inconsistent. We wish to
�nd the solution x that minimizes the Euclidean norm (or length) of the error Ax ; b. Compare the
solutions obtained by using the following four Matlab invocations, each of which in principle gives the
desired least-square-error solution:
(a) x � Anb
(b) x � pinv(A) � b
(c) x � inv(A0 � A) � A0 � b
(d) [q� r] � qr(A), followed by implementation of the approach described in Exercise 3.1.
For more information on these commands, try help slash, help qr, help pinv, help inv, etc.
[Incidentally, the prime, 0 , in Matlab takes the transpose of the complex conjugate of a matrix� if you
want the ordinary transpose of a complex matrix C , you have to write C:0 or transp(C ).]
You should include in your solutions a plot the ellipse that corresponds to your estimate of x.
If you create the following function �le in your Matlab directory, with the name ellipse.m, you can
obtain the polar coordinates theta, rho of n points on the ellipse speci�ed by the parameter vector x.
To do this, enter [theta,rho]�ellipse(x,n)� at the Matlab prompt. You can then plot the ellipse
by using the polar(theta,rho) command.
function [theta,rho]�ellipse(x,n)
% [theta,rho]�ellipse(x,n)
%
% The vector x � [x(1),x(2),x(3)]', de�nes an ellipse centered at the origin
% via the equation x(1)*r^ 2 + x(2)*s^ 2 +x(3)*r*s � 1.
% This routine generates the polar coordinates of points on the ellipse,
% to send to a plot command. It does this by solving for the radial
% distance in n equally spaced angular directions.
% Use polar(theta,rho) to actually plot the ellipse.
 theta � 0:(2*pi/n):(2*pi)�
a � x(1)*cos(theta).^ 2 + x(2)*sin(theta).^ 2 + x(3)*(cos(theta).*sin(theta))�

rho � ones(size(a))./sqrt(a)�

Exercise 2.2 Approximation by a Polynomial

Let f (t) � 0:5e , t 2 [0� 2].
0:8t

(a) Suppose 16 exact measurements of f (t) are available to you, taken at the times ti listed in the
array T below:

T � [2 � 10;3� 0:136� 0:268� 0:402� 0:536� 0:668� 0:802� 0:936�

1:068� 1:202� 1:336� 1:468� 1:602� 1:736� 1:868� 2:000]
Use Matlab to generate these measurements:
yi � f (ti ) i � 1� : : : � 16 ti 2 T
Now determine the coe�cients of the least square error polynomial approximation of the mea-
surements, for
1. a polynomial of degree 15, p15 (t)�
2. a polynomial of degree 2, p2 (t).

Compare the quality of the two approximations by plotting y(ti ), p15 (ti ) and p2 (ti ) for all ti
in T . To see how well we are approximating the function on the whole interval, also plot f (t),
p15 (t) and p2 (t) on the interval [0� 2]. (Pick a very �ne grid for the interval, e.g. t�[0:1000]'/500.)
Report your observations and comments.

(b) Now suppose that your measurements are a�ected by some noise. Generate the measurements
using
yi � f (ti ) + e(ti ) i � 1� : : : � 16 ti 2 T

where the vector of noise values can be generated in the following way:
randn(0 seed0� 0)�
e � randn(size(T ))�
Again determine the coe�cients of the least square error polynomial approximation of the mea-
surements for
1. a polynomial of degree 15, p15 (t)�
2. a polynomial of degree 2, p2 (t).
Compare the two approximations as in part (a). Report your observations and comments.
Explain any surprising results.
(c) So far we have obtained polynomial approximations of f (t) � t 2 [0� 2] � by approximating the
measurements at ti 2 T . We are now interested in minimizing the square error of the polynomial

approximation over the whole interval [0� 2]:

Z2
min kf (t) ; pn (t)k22 � min jf (t) ; pn (t)j2 dt
0
where pn (t) is some polynomial of degree n. Find the polynomial p2 (t) of degree 2 that solves
the above problem. Are the optimal p2 (t) in this case and the optimal p2 (t) of parts (a) and (b)
very di�erent from each other� Elaborate.

Exercise 2.3 Combining Estimates

Suppose y1 � C1 x + e1 and y2 � C2 x + e2 , where x is an n-vector, and C1 , C2 have full column
rank. Let x^1 denote the value of x that minimizes eT1 S1 e1 , and x^2 denote the value that minimizes
eT2 S2 e2 , where S1 and S2 are positive de�nite matrices. Show that the value x^ of x that minimizes

eT1 S1 e1 + e2T S2 e2 can be written entirely in terms of x^1 , x^2 , and the n � n matrices Q1 � C1T S1 C1 and
Q2 � C2T S2 C2 . What is the signi�cance of this result�

Exercise 2.4 Exponentially Windowed Estimates

Suppose we observe the scalar measurements
yi � ci x + ei � i � 1� 2� : : :
where ci and x are possibly vectors (row- and column-vectors respectively).

(a) Show (by reducing this to a problem that we already know how to solve | don't start from
scratch!) that the value x^k of x that minimizes the criterion

X
k
f k;i e2i � some �xed f� 0 � f � 1

i�1
is given by
�X
k �;1�X
k �
x^k � f k;i cTi ci f k;i ciT yi
i�1 i�1
The so-called fade or forgetting factor f allows us to preferentially weight the more recent mea-
surements by picking 0 � f � 1, so that old data is discounted at an exponential rate. We
then say that the data has been subjected to exponential fading or forgetting or weighting or
windowing or tapering or ... . This is usually desirable, in order to keep the �lter adaptive to
changes that may occur in x. Otherwise the �lter becomes progressively less attentive to new
data and falls asleep, with its gain approaching 0.
(b) Now show that
x^k � x^k;1 + Q;k 1 cTk (yk ; ck x^k;1 )
where
Qk � fQk;1 + cTk ck � Q0 � 0
The vector gk � Q;k 1 cTk is termed the gain of the estimator.
(c) If x and ci are scalars, and ci is a constant c, determine gk as a function of k. What is the
steady-state gain g1� Does g1 increase or decrease as f increases | and why do you expect

this�

Exercise 2.5 Suppose our model for some waveform y(t) is y(t) � � sin (!t), where � is a scalar,
and suppose we have measurements y(t1 )� : : : � y(tp ). Because of modeling errors and the presence of
measurement noise, we will generally not �nd any choice of model parameters that allows us to pre-
cisely account for all p measurements.

(a) If ! is known, �nd the value of � that minimizes

X
p
[y(ti ) ; � sin(!ti )]2
i�1
(b) Determine this value of � if ! � 2 and if the measured values of y(t) are:
y(1) � +2:31 y(2) � ;2:01 y(3) � ;1:33 y(4) � +3:23
y(5) � ;1:28 y(6) � ;1:66 y(7) � +3:28 y(8) � ;0:88
(I generated this data using the equation y(t) � 3 sin(2t) + e(t) evaluated at the integer values
t � 1� : : : � 8, and with e(t) for each t being a random number uniformly distributed in the interval
- 0.5 to +0.5.)
(c) Suppose that � and ! are unknown, and that we wish to determine the values of these two variables
that minimize the above criterion. Assume you are given initial estimates �0 and !0 for the

minimizing values of these variables. Using the Gauss-Newton algorithm for this nonlinear least
squares problem, i.e. applying LLSE to the problem obtained by linearizing about the initial
estimates, determine explicitly the estimates �1 and !1 obtained after one iteration of this
algorithm. Use the following notation to help you write out the solution in a condensed form:
X X X
a� sin2 (!0 ti ) � b � t2i cos2 (!0 ti ) � c � ti [sin(w0 ti )][cos(w0 ti )]

(d) What values do you get for �1 and !1 with the data given in (b) above if the initial guesses
are �0 � 3:2 and !0 � 1:8 � Continue the iterative estimation a few more steps. Repeat the

procedure when the initial guesses are �0 � 3:5 and !0 � 2:5, verifying that the algorithm does
not converge.
(e) Since only ! enters the model nonlinearly, we might think of a decomposed algorithm, in which �
is estimated using linear least squares and ! is estimated via nonlinear least squares. Suppose,

for example, that our initial estimate of ! is !0 � 1:8. Now obtain an estimate �1 of � using the
linear least squares method that you used in (b). Then obtain an (improved�) estimate !1 of !,
using one iteration of a Gauss-Newton algorithm (similar to what is needed in (c), except that
now you are only trying to estimate !). Next obtain the estimate �2 via linear least squares,
and so on. Compare your results with what you obtain via this decomposed procedure when
your initial estimate is !0 � 2:5 instead of 1.8.

Exercise 2.6 Comparing Di�erent Estimators

This problem asks you to compare the behavior of di�erent parameter estimation algorithms by
�tting a model of the type y(t) � a sin(2�t) + b cos(4�t) to noisy data taken at values of t that are .02
apart in the interval (0,2].
First synthesize the data on which you will test the algorithms. Even though your estimation
algorithms will assume that a and b are constant, we are interested in seeing how they track parameter
changes as well. Accordingly, let a � 2, b � 2 for the �rst 50 points, and a � 1, b � 3 for the next 50
points. To get (approximately) normally distributed random variables, we use the function randn to
produce variables with mean 0 and variance 1.
An elegant way to generate the data in Matlab, exploiting Matlab's facility with vectors, is to
de�ne the vectors t1 � 0:02 : 0:02 : 1:0 and t2 � 1:02 : 0:02 : 2:0, then set
y1 � 2 � sin(2 � pi � t1) + 2 � cos(4 � pi � t1) + s � randn(size(t1))
and
y2 � sin(2 � pi � t2) + 3 � cos(4 � pi � t2) + s � randn(size(t2))
where s determines the standard deviation of the noise. Pick s � 1 for this problem. Finally, set
y � [y1� y2]. No loops, no counters, no fuss!!
Now estimate a and b from y using the following algorithms. Assume prior estimates a^0 � 3
and ^b0 � 1, weighted equally with the measurements (so all weights can be taken as 1 without loss of
generality). Plot your results to aid comparison.
(i) Recursive least squares.
(ii) Recursive least squares with exponentially fading memory, as in Problem 3. Use f � :96.
(iii) The algorithm in (ii), but with Qk of Problem 3 replaced by qk � (1�n)�trace(Qk ), where
n is the number of parameters, so n � 2 in this case. (Recall that the trace of a matrix is the sum of
its diagonal elements. Note that qk itself satis�es a recursion, which you should write down.)
(iv) An algorithm of the form
x^ � x^ + :04 cT (y ; c x^ )
k k;1 ck cTk k k k k;1
where ck � [sin(2�t)� cos(4�t)] evaluated at the kth sampling instant, so t � :02k.

Exercise 2.7 Recursive Estimation of a State Vector

This course will soon begin to consider state-space models of the form
x` � Ax`;1
(2.4)
where x` is an n-vector denoting the state at time ` of our model of some system, and A is a known
n � n matrix. For example, suppose the system of interest is a rotating machine, with angular position
d` and angular velocity !` at time t � `T , where T is some �xed sampling interval. If we believed the
machine to be rotating at constant speed, we would be led to the model
�d � �1 T ��d �
` `;1
!` � 0 1 !`;1
Assume A to be nonsingular throughout this problem.
For the rotating machine example above, it is often of interest to obtain least-square-error esti-
mates of the position and (constant) velocity, using noisy measurements of the angular position dj at
the sampling instants. More generally, it is of interest to obtain a least-square-error estimate of the
state vector xi in the model (2.4) from noisy p-component measurements yj that are related to xj by
a linear equation of the form
yj � Cxj + ej � j � 1� : : : � i
where C is a p � n matrix. We shall also assume that a prior estimate x^0 of x0 is available:
x^0 � x0 + e0
Let x^iji denote the value of xi that minimizes
Xi
kej k2
j �0
This is the estimate of xi given the prior estimate and measurements up to time i, or the \�ltered
estimate" of xi . Similarly, let x^iji;1 denote the value of xi that minimizes
X
i;1
kej k2
j �0
This is the least-square-error estimate of xi given the prior estimate and measurements up to time
i ; 1, and is termed the \one-step prediction" of xi .

a) Set up the linear system of equations whose least square error solution would be x^iji . Similarly,

set up the linear system of equations whose least square error solution would be x^iji;1 .

b) Show that x^iji;1 � Ax^i;1ji;1 .

c) Determine a recursion that expresses x^iji in terms of x^i;1ji;1 and yi . This is the prototype of what

is known as the Kalman �lter. A more elaborate version of the Kalman �lter would include additive

noise driving the state-space model, and other embellishments, all in a stochastic context (rather than
the deterministic one given here).

Exercise 2.8 Let x^ denote the value of x that minimizes ky ; Axk2 , where A has full column rank.
Let x denote the value of x that minimizes this same criterion, but now subject to the constraint that
z � Dx, where D has full row rank. Show that
� �;1
x � x^ + (AT A);1 DT D(AT A);1 DT (z ; Dx^)
(Hint: One approach to solving this is to use our recursive least squares formulation, but modi�ed for
the limiting case where one of the measurement sets | namely z � Dx in this case | is known to
have no error. You may have to use some of the matrix identities from the previous chapter).