Scribd
Upload
20 views

Sec 4-7

The document describes the least squares approximation method. It seeks to find a polynomial approximation p(x) that minimizes the average error over an interval by minimizing the integral of the squared difference between the function f(x) and the approximation p(x). This results in a linear system of equations whose solution gives the coefficients of p(x) that provide the best least squares approximation of f(x). The optimal approximation is expanded in terms of Legendre polynomials.

Uploaded by

Florin
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
20 views

Sec 4-7

The document describes the least squares approximation method. It seeks to find a polynomial approximation p(x) that minimizes the average error over an interval by minimizing the integral of the squared difference between the function f(x) and the approximation p(x). This results in a linear system of equations whose solution gives the coefficients of p(x) that provide the best least squares approximation of f(x). The optimal approximation is expanded in terms of Legendre polynomials.

Uploaded by

Florin
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 15

LEAST SQUARES APPROXIMATION

Another approach to approximating a function f (x)


on an interval a ≤ x ≤ b is to seek an approximation
p(x) with a small ‘average error’ over the interval of
approximation. A convenient definition of the average
error of the approximation is given by
" Z b #1
1 2
E(p; f ) ≡ [f (x) − p(x)]2 dx (1)
b−a a
This is also called the root-mean-square-error (de-
noted subsequently by RMSE ) in the approximation
of f (x) by p(x). Note first that choosing p(x) to
minimize E(p; f ) is equivalent to minimizing
Z b
[f (x) − p(x)]2 dx
a
thus dispensing with the square root and multiplying
fraction (although the minimums are generally differ-
ent). The minimizing of (1) is called the least squares
approximation problem.
Example. Let f (x) = ex, let p(x) = α0 + α1x, α0,
α1 unknown. Approximate f (x) over [−1, 1]. Choose
α0, α1 to minimize
Z 1
g(α0, α1) ≡ [ex − α0 − α1x]2 dx (2)
−1
Z 1 ( 2x 2 2 2 x
)
e + α0 + α1x − 2α0e
g(α0, α1) = x dx
−1 −2α1xe + 2α0α1x
Integrating,

g(α0, α1) = c1α20 + c2α21 + c3α0α1 + c4α0 + c5α1 + c6


with constants {c1, . . . , c6}, e.g.
³ ´
c1 = 2, 1
c6 = e − e−1 /2.
g is a quadratic polynomial in the two variables α0,
α1. To find its minimum, solve the system
∂g ∂g
= 0, =0
∂α0 ∂α1
It is simpler to return to (2) to differentiate, obtaining
Z 1
2 [ex − α0 − α1x] (−1) dx = 0
−1
Z 1
2 [ex − α0 − α1x] (−x) dx = 0
−1
This simplifies to
Z 1
2α0 = exdx = e − e−1
−1
Z 1
2
α1 = xexdx = 2e−1
3 −1

e − e−1 .
α0 = = 1.1752
2
.
α1 = 3e−1 = 1.1036
Using these values for α0 and α1, we denote the re-
sulting linear approximation by

1(x) = α0 + α1x
It is called the best linear approximation to ex in the
sense of least squares. For the error,
.
max |ex − 1(x)| = 0.439
−1≤x≤1
Errors in linear approximations of ex:

Approximation Max Error RMSE


Taylor t1(x) 0.718 0.246
Least squares 1(x) 0.439 0.162
Chebyshev c1(x) 0.372 0.184
Minimax m1(x) 0.279 0.190

x
y=e

y=l (x) 1
1

x
-1 1

The linear least squares approximation to ex


THE GENERAL CASE

Approximate f (x) on [a, b], and let n ≥ 0. Seek p(x)


to minimize the RMSE. Write

p(x) = α0 + α1x + · · · + αnxn

Z1 " #2
f (x) − α0 − α1x
g(α0, α1, . . . , αn) ≡ dx
− · · · − αnxn
−1
Find coefficients α0, α1, . . . , αn to minimize this in-
tegral. The integral g(α0, α1, . . . , αn) is a quadratic
polynomial in the n + 1 variables α0, α1, . . . , αn.

To minimize g(α0, α1, . . . , αn), invoke the conditions


∂g
= 0, i = 0, 1, . . . , n
∂αi
This yields a set of n+1 equations that must be satis-
fied by a minimizing set α0, α1, . . . , αn for g. Manip-
ulating this set of conditions leads to a simultaneous
linear system.
To better understand the form of the linear system,
consider the special case of [a, b] = [0, 1]. Differenti-
ating g with respect to each αi, we obtain
Z 1
2 [ex − α0 − · · · − αnxn] (−1) dx = 0
−1
Z 1
2 [ex − α0 − · · · − αnxn] (−x) dx = 0
−1
..
Z 1
2 [ex − α0 − · · · − αnxn] (−xn) dx = 0
−1
Then the linear system is
n
X Z 1
αj
= xif (x) dx, i = 0, 1, . . . , n
j=0 i + j + 1 0

We will study the solution of simultaneous linear sys-


tems in Chapter 6. There we will see that this linear
system is ‘ill-conditioned’ and is difficult to solve ac-
curately, even for moderately sized values of n such as
n = 5. As a consequence, this is not a good approach
to solving for a minimizer of g(α0, α1, . . . , αn).
LEGENDRE POLYNOMIALS

Define the Legendre polynomials as follows.

P0(x) = 1

1 dn h³ 2 ´ni
Pn(x) = n
· n x −1 , n = 1, 2, . . .
n!2 dx
For example,

P1(x) = x
1³ 2 ´
P2(x) = 3x − 1
2
1³ 3 ´
P3(x) = 5x − 3x
2
1³ 4 2
´
P4(x) = 35x − 30x + 3
8
The Legendre polynomials have many special proper-
ties, and they are widely used in numerical analysis
and applied mathematics.
y

x
-1 1

P (x)
1
P (x)
2
P (x)
3
P (x)
4

-1

Legendre polynomials of degrees 1, 2, 3, 4


PROPERTIES

Introduce the special notation


Z b
(f, g) = f (x)g(x) dx
a
for general functions f (x) and g(x).

• Degree and normalization:

deg Pn = n, Pn(1) = 1, n≥0

• Triple recursion relation: For n ≥ 1,


2n + 1 n
Pn+1(x) = xPn(x) − Pn−1(x)
n+1 n+1

• Orthogonality and size:



³ ´ 
0,
 i 6= j
Pi, Pj = 2

 , i=j
2j + 1
• Zeroes:
All zeroes of Pn(x) are located in [−1, 1] ;
all zeroes are simple roots of Pn(x)

• Basis: Every polynomial p(x) of degree ≤ n can


be written in the form
n
X
p(x) = β j Pj (x)
j=0
with the choice of β 0, β 1, . . . , β n uniquely deter-
mined from p(x):
³ ´
p, Pj
βj = ³ ´, j = 0, 1, . . . , n
Pj , Pj
FINDING THE LEAST SQUARES
APPROXIMATION

We solve the least squares approximation problem on


only the interval [−1, 1]. Approximation problems on
other intervals [a, b] can be accomplished using a lin-
ear change of variable.

We seek to find a polynomial p(x) of degree n that


minimizes
Z b
[f (x) − p(x)]2 dx
a
This is equivalent to minimizing

(f − p, f − p) (3)
We begin by writing p(x) in the form
n
X
p(x) = β j Pj (x)
j=0
n
X
p(x) = β j Pj (x)
j=0
Substitute into (3), obtaining
ge (β 0
, β 1, . . . , β n) ≡ (f − p, f − p)

n
X n
X
= f − β j Pj , f − β iPi
j=0 i=0
Expand this into the following:
n (f, P )2
X j
ge = (f, f ) −
j=0 (Pj , Pj )
n ³ " #2
X ´ (f, Pj )
+ Pj , Pj β j −
j=0 (Pj , Pj )
Looking at this carefully, we see that it is smallest
when
(f, Pj )
βj = , j = 0, 1, . . . , n
(Pj , Pj )
Looking at this carefully, we see that it is smallest
when
(f, Pj )
βj = , j = 0, 1, . . . , n
(Pj , Pj )
The minimum for this choice of coefficients is
n (f, P )2
X j
ge = (f, f ) −
j=0 (Pj , Pj )
We call
n (f, P )
X j
n(x) = Pj (x) (4)
j=0 (Pj , Pj )
the least squares approximation of degree n to f (x)
on [−1, 1].

If β n = 0, then its actual degree is less than n.


Example. Approximate f (x) = ex on [−1, 1]. We use
(4) with n = 3:
3
X (f, Pj )
3(x) = β j Pj (x), βj = (5)
j=0 (Pj , Pj )
The coefficients {β 0, β 1, β 2, β 3} are as follows.

j 0 1 2 3
βj 2.35040 0.73576 0.14313 0.02013

Using (5) and simplifying,


2 3
3(x) = .996294 + .997955x + .536722x + .176139x

The error in various cubic approximations:

Approximation Max Error RMSE


Taylor t3(x) .0516 .0145
Least squares 3(x) .0112 .00334
Chebyshev c3(x) .00666 .00384
Minimax m3(x) .00553 .00388
y

0.0112

x
-1 1

-0.00460

Error in the cubic least squares approximation to ex

You might also like