Chebyshev Polynomials

Least Squares, redux

Numerical Analysis and Computing

Lecture Notes #12
— Approximation Theory —
Chebyshev Polynomials & Least Squares, redux

Joe Mahaffy,
[email protected]
Department of Mathematics
Dynamical Systems Group
Computational Sciences Research Center
San Diego State University
San Diego, CA 92182-7720
http://www-rohan.sdsu.edu/∼jmahaffy

Spring 2010
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (1/45)
 Chebyshev Polynomials
Least Squares, redux

Outline

1 Chebyshev Polynomials
Orthogonal Polynomials
Chebyshev Polynomials, Intro & Definitions
Properties

2 Least Squares, redux

Examples
More than one variable? — No problem!

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (2/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonal Polynomials: A Quick Summary

So far we have seen the use of orthogonal polynomials can help us

solve the normal equations which arise in discrete and continuous
least squares problems, without the need for expensive and
numerically difficult matrix inversions.

The ideas and techniques we developed — i.e. Gram-Schmidt

orthogonalization with respect to a weight function over any
interval have applications far beyond least squares problems.

The Legendre Polynomials are orthogonal on the interval [−1, 1]

with respect to the weight function w (x) = 1. — One curious
property of the Legendre polynomials is that their roots (all real)
yield the optimal node placement for Gaussian quadrature.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (3/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Legendre Polynomials Background

The Legendre polynomials are solutions to the Legendre

Differential Equation (which arises in numerous problems
exhibiting spherical symmetry)

d 2y dy
(1 − x 2 ) 2
− 2x + ℓ(ℓ + 1)y = 0, ℓ∈N
dx dx
or equivalently
· ¸
d dy
(1 − x 2 ) + ℓ(ℓ + 1)y = 0, ℓ∈N
dx dx

Applications: Celestial Mechanics (Legendre’s original applica-

tion), Electrodynamics, etc...

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (4/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Other Orthogonal Polynomials Background

“Orthogonal polynomials have very useful properties in the solution

of mathematical and physical problems. [... They] provide a
natural way to solve, expand, and interpret solutions to many types
of important differential equations. Orthogonal polynomials are
especially easy to generate using Gram-Schmidt
orthonormalization.”
“The roots of orthogonal polynomials possess many rather
surprising and useful properties.”

(http://mathworld.wolfram.com/OrthogonalPolynomials.html)

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (5/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Laguerre Polynomials Background

The Laguerre polynomials are solutions to the Laguerre

differential equation

d2 dy
x 2
+ (1 − x) + λy = 0
dx dx
They are associated with the radial solution to the Schrödinger
equation for the Hydrogen atom’s electron (Spherical Harmonics).
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (6/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

More Orthogonal Polynomials Background

Polynomials Interval w(x)√

Chebyshev (1st) [−1, 1] 1/ 2
√ 1−x
Chebyshev (2nd) [−1, 1] 1 − x2
Gegenbauer [−1, 1] (1 − x 2 )α−1/2
2
Hermite∗ (−∞, ∞) e −x
Jacobi (−1, 1) (1 − x)α (1 + x)β
Legendre [−1, 1] 1
Laguerre [0, ∞) e −x
Laguerre (assoc) [0, ∞) x k e −x

Today we’ll take a closer look at Chebyshev polynomials of the first

kind.
∗ These are the Hermite orthogonal polynomials, not to be confused
with the Hermite interpolating polynomials...

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (7/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials: The Sales Pitch.

1 − t 2 t −(n+1)
I ¡ ¢
1
Tn (z) = dt
2πi (1 − 2tz + t 2 )

Chebyshev Polynomials are used to minimize approximation

error. We will use them to solve the following problems:

[1] Find an optimal placement of the interpolating points

{x0 , x1 , . . . , xn } to minimize the error in Lagrange interpola-
tion.
[2] Find a means of reducing the degree of an approximating poly-
nomial with minimal loss of accuracy.
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (8/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials: Definitions.

The Chebyshev polynomials {Tn (x)} are orthogonal on√the interval

(−1, 1) with respect to the weight function w (x) = 1/ 1 − x 2 , i.e.
Z 1
hTi (x), Tj (x)iw (x) ≡ Ti (x)Tj (x)∗ w (x)dx = αi δi,j .
−1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (9/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials: Definitions.

The Chebyshev polynomials {Tn (x)} are orthogonal on√the interval

(−1, 1) with respect to the weight function w (x) = 1/ 1 − x 2 , i.e.
Z 1
hTi (x), Tj (x)iw (x) ≡ Ti (x)Tj (x)∗ w (x)dx = αi δi,j .
−1

We could use the Gram-Schmidt orthogonalization process to find

them, but it is easier to give the definition and then check the
properties...

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (9/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials: Definitions.

The Chebyshev polynomials {Tn (x)} are orthogonal on√the interval

(−1, 1) with respect to the weight function w (x) = 1/ 1 − x 2 , i.e.
Z 1
hTi (x), Tj (x)iw (x) ≡ Ti (x)Tj (x)∗ w (x)dx = αi δi,j .
−1

We could use the Gram-Schmidt orthogonalization process to find

them, but it is easier to give the definition and then check the
properties...

Definition (Chebyshev Polynomials)

For x ∈ [−1, 1], define

Tn (x) = cos(n arccos x), ∀n ≥ 0.

Note: T0 (x) = cos(0) = 1, T1 (x) = x.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (9/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials, Tn (x), n ≥ 2.

We introduce the notation θ = arccos x, and get

Tn (θ(x)) ≡ Tn (θ) = cos(nθ), where θ ∈ [0, π].

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (10/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials, Tn (x), n ≥ 2.

We introduce the notation θ = arccos x, and get

Tn (θ(x)) ≡ Tn (θ) = cos(nθ), where θ ∈ [0, π].

We can find a recurrence relation, using these observations:

Tn+1 (θ) = cos((n + 1)θ) = cos(nθ) cos(θ) − sin(nθ) sin(θ)

Tn−1 (θ) = cos((n − 1)θ) = cos(nθ) cos(θ) + sin(nθ) sin(θ)
Tn+1 (θ) + Tn−1 (θ) = 2 cos(nθ) cos(θ).

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (10/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Chebyshev Polynomials, Tn (x), n ≥ 2.

We introduce the notation θ = arccos x, and get

Tn (θ(x)) ≡ Tn (θ) = cos(nθ), where θ ∈ [0, π].

We can find a recurrence relation, using these observations:

Tn+1 (θ) = cos((n + 1)θ) = cos(nθ) cos(θ) − sin(nθ) sin(θ)

Tn−1 (θ) = cos((n − 1)θ) = cos(nθ) cos(θ) + sin(nθ) sin(θ)
Tn+1 (θ) + Tn−1 (θ) = 2 cos(nθ) cos(θ).

Returning to the original variable x, we have

Tn+1 (x) = 2x cos(n arccos x) − Tn−1 (x),

or
Tn+1 (x) = 2xTn (x) − Tn−1 (x).
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (10/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Chebyshev Polynomials

1
T1(x)

0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (11/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Chebyshev Polynomials

1
T1(x)
T2(x)

0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (11/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Chebyshev Polynomials

1
T1(x)
T2(x)
T3(x)

0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (11/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Chebyshev Polynomials

1
T1(x)
T2(x)
T3(x)
T4(x)
0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (11/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Chebyshev Polynomials

1
T1(x)
T2(x)
T3(x)
T4(x)
0.5 T5(x)

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (11/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonality of the Chebyshev Polynomials, I

1 1
Tn (x)Tm (x) dx
Z Z
√ dx = cos(n arccos x) cos(m arccos x) √ .
−1 1 − x2 −1 1 − x2

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (12/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonality of the Chebyshev Polynomials, I

1 1
Tn (x)Tm (x) dx
Z Z
√ dx = cos(n arccos x) cos(m arccos x) √ .
−1 1 − x2 −1 1 − x2
Reintroducing θ = arccos x gives,
dx
dθ = − √ ,
1 − x2
and the integral becomes
Z 0 Z π
− cos(nθ) cos(mθ) dθ = cos(nθ) cos(mθ) dθ.
π 0

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (12/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonality of the Chebyshev Polynomials, I

1 1
Tn (x)Tm (x) dx
Z Z
√ dx = cos(n arccos x) cos(m arccos x) √ .
−1 1 − x2 −1 1 − x2
Reintroducing θ = arccos x gives,
dx
dθ = − √ ,
1 − x2
and the integral becomes
Z 0 Z π
− cos(nθ) cos(mθ) dθ = cos(nθ) cos(mθ) dθ.
π 0
Now, we use the fact that
cos(n + m)θ + cos(n − m)θ
cos(nθ) cos(mθ) = ...
2
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (12/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonality of the Chebyshev Polynomials, II

We have: π
cos(n + m)θ + cos(n − m)θ
Z
dθ.
0 2

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (13/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonality of the Chebyshev Polynomials, II

We have: π
cos(n + m)θ + cos(n − m)θ
Z
dθ.
0 2
If m 6= n, we get
· ¸π
1 1
sin((n + m)θ) + sin((n − m)θ) = 0,
2(n + m) 2(n − m) 0

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (13/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Orthogonality of the Chebyshev Polynomials, II

We have: π
cos(n + m)θ + cos(n − m)θ
Z
dθ.
0 2
If m 6= n, we get
· ¸π
1 1
sin((n + m)θ) + sin((n − m)θ) = 0,
2(n + m) 2(n − m) 0

if m = n, we have
· ¸π
1 x π
sin((n + m)θ) + = .
2(n + m) 2 0 2

Hence, the Chebyshev polynomials are orthogonal.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (13/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Zeros and Extrema of Chebyshev Polynomials.

Theorem
The Chebyshev polynomial of degree n ≥ 1 has n simple zeros in
[−1, 1] at µ ¶
2k − 1
xk = cos π , k = 1, . . . , n.
2n
Moreover, Tn (x) assumes its absolute extrema at
µ ¶
′ kπ
xk = cos , with Tn (xk′ ) = (−1)k , k = 1, . . . , n − 1.
n

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (14/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Zeros and Extrema of Chebyshev Polynomials.

Theorem
The Chebyshev polynomial of degree n ≥ 1 has n simple zeros in
[−1, 1] at µ ¶
2k − 1
xk = cos π , k = 1, . . . , n.
2n
Moreover, Tn (x) assumes its absolute extrema at
µ ¶
′ kπ
xk = cos , with Tn (xk′ ) = (−1)k , k = 1, . . . , n − 1.
n

Payoff: No matter what the degree of the polynomial, the oscilla-

tions are kept under control!!!

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (14/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Zeros and Extrema of Chebyshev Polynomials — Proof.

Proof.
Let: „ « „ «
2k − 1 kπ
xk = cos π , xk′ = cos .
2n n
Then:
“ “ “ ”””
Tn (xk ) = cos(n arccos(xk )) = cos n arccos cos 2k−1
2n
π
“ ” √
= cos 2k−1
2
π = 0,

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (15/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Zeros and Extrema of Chebyshev Polynomials — Proof.

Proof.
Let: „ « „ «
2k − 1 kπ
xk = cos π , xk′ = cos .
2n n
Then:
“ “ “ ”””
Tn (xk ) = cos(n arccos(xk )) = cos n arccos cos 2k−1
2n
π
“ ” √
= cos 2k−1
2
π = 0,

d n sin(n arccos(x))
Tn′ (x) = [cos(n arccos(x))] = √ ,
dx 1 − x2

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (15/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Zeros and Extrema of Chebyshev Polynomials — Proof.

Proof.
Let: „ « „ «
2k − 1 kπ
xk = cos π , xk′ = cos .
2n n
Then:
“ “ “ ”””
Tn (xk ) = cos(n arccos(xk )) = cos n arccos cos 2k−1
2n
π
“ ” √
= cos 2k−1
2
π = 0,

d n sin(n arccos(x))
Tn′ (x) = [cos(n arccos(x))] = √ ,
dx 1 − x2
n sin(n arccos(cos( kπ
n ))) n sin(kπ) √
Tn′ (xk′ ) = = = 0,
sin( kπ
q
1−cos2 ( kπ
n ) n )

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (15/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Zeros and Extrema of Chebyshev Polynomials — Proof.

Proof.
Let: „ « „ «
2k − 1 kπ
xk = cos π , xk′ = cos .
2n n
Then:
“ “ “ ”””
Tn (xk ) = cos(n arccos(xk )) = cos n arccos cos 2k−1
2n
π
“ ” √
= cos 2k−1
2
π = 0,

d n sin(n arccos(x))
Tn′ (x) = [cos(n arccos(x))] = √ ,
dx 1 − x2
n sin(n arccos(cos( kπ
n ))) n sin(kπ) √
Tn′ (xk′ ) = = = 0,
sin( kπ
q
1−cos2 ( kπ
n ) n )

“ “ “ ””” √
Tn (xk′ ) = cos n arccos cos kπ
n
= cos(kπ) = (−1)k .

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (15/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Monic Chebyshev Polynomials, I

Definition (Monic Polynomial)

A monic polynomial is a polynomial with leading coefficient 1.

We get the monic Chebyshev polynomials T̃n (x) by dividing Tn (x)

by 2n−1 , n ≥ 1. Hence,
1
T̃0 (x) = 1, T̃n (x) = Tn (x), n ≥ 1.
2n−1
They satisfy the following recurrence relations

T̃2 (x) = x T̃1 (x) − 12 T̃0 (x)

T̃n+1 (x) = x T̃n (x) − 14 T̃n−1 (x).

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (16/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Monic Chebyshev Polynomials, II

The location of the zeros and extrema of T̃n (x) coincides with
those of Tn (x), however the extreme values are
(−1)k
T̃n (xk′ ) = , k = 1, . . . , n − 1.
2n−1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (17/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Monic Chebyshev Polynomials, II

The location of the zeros and extrema of T̃n (x) coincides with
those of Tn (x), however the extreme values are
(−1)k
T̃n (xk′ ) = , k = 1, . . . , n − 1.
2n−1

Definition
Let P̃n denote the set of all monic polynomials of degree n.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (17/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Monic Chebyshev Polynomials, II

The location of the zeros and extrema of T̃n (x) coincides with
those of Tn (x), however the extreme values are
(−1)k
T̃n (xk′ ) = , k = 1, . . . , n − 1.
2n−1

Definition
Let P̃n denote the set of all monic polynomials of degree n.

Theorem (Min-Max)
The monic Chebyshev polynomials T̃n (x), have the property that
1
= max |T̃n (x)| ≤ max |Pn (x)|, ∀Pn (x) ∈ P̃n .
2n−1 x∈[−1,1] x∈[−1,1]

Moreover, equality can only occur if Pn (x) ≡ T̃n (x).

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (17/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Monic Chebyshev Polynomials

1
T1(x)

0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (18/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Monic Chebyshev Polynomials

1
T1(x)
T2(x)

0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (18/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Monic Chebyshev Polynomials

1
T1(x)
T2(x)
T3(x)

0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (18/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Monic Chebyshev Polynomials

1
T1(x)
T2(x)
T3(x)
T4(x)
0.5

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (18/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

The Monic Chebyshev Polynomials

1
T1(x)
T2(x)
T3(x)
T4(x)
0.5 T5(x)

-0.5

-1
-1 -0.5 0 0.5 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (18/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Optimal Node Placement in Lagrange Interpolation, I

If x0 , x1 , . . . , xn are distinct points in the interval [−1, 1] and

f ∈ C n+1 [−1, 1], and P(x) the nth degree interpolating Lagrange
polynomial, then ∀x ∈ [−1, 1] ∃ξ(x) ∈ (−1, 1) so that
n
f (n+1) (ξ(x)) Y
f (x) − P(x) = (x − xk ).
(n + 1)!
k=0

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (19/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Optimal Node Placement in Lagrange Interpolation, I

If x0 , x1 , . . . , xn are distinct points in the interval [−1, 1] and

f ∈ C n+1 [−1, 1], and P(x) the nth degree interpolating Lagrange
polynomial, then ∀x ∈ [−1, 1] ∃ξ(x) ∈ (−1, 1) so that
n
f (n+1) (ξ(x)) Y
f (x) − P(x) = (x − xk ).
(n + 1)!
k=0

We have no control over f (n+1) (ξ(x)), but we can

Qnplace the nodes
in a clever way as to minimize the maximum of k=0 (x − xk ).

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (19/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Optimal Node Placement in Lagrange Interpolation, I

If x0 , x1 , . . . , xn are distinct points in the interval [−1, 1] and

f ∈ C n+1 [−1, 1], and P(x) the nth degree interpolating Lagrange
polynomial, then ∀x ∈ [−1, 1] ∃ξ(x) ∈ (−1, 1) so that
n
f (n+1) (ξ(x)) Y
f (x) − P(x) = (x − xk ).
(n + 1)!
k=0

We have no control over f (n+1) (ξ(x)), but we can

Qnplace the nodes
in a clever
Q way as to minimize the maximum of k=0 (x − xk ).
Since nk=0 (x − xk ) is a monic polynomial of degree (n + 1), we
know the min-max is obtained when the nodes are chosen so that
n µ ¶
Y 2k + 1
(x − xk ) = T̃n+1 (x), i.e. xk = cos π .
2(n + 1)
k=0

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (19/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Optimal Node Placement in Lagrange Interpolation, II

Theorem
If P(x) is the interpolating polynomial of degree at most n with
nodes at the roots of Tn+1 (x), then

1
max |f (x) − P(x)| ≤ max |f (n+1) (x)|,
x∈[−1,1] 2n (n + 1)! x∈[−1,1]

∀f ∈ C n+1 [−1, 1].

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (20/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Optimal Node Placement in Lagrange Interpolation, II

Theorem
If P(x) is the interpolating polynomial of degree at most n with
nodes at the roots of Tn+1 (x), then

1
max |f (x) − P(x)| ≤ max |f (n+1) (x)|,
x∈[−1,1] 2n (n + 1)! x∈[−1,1]

∀f ∈ C n+1 [−1, 1].

Extending to any interval: The transformation

1
x̃ = [(b − a)x + (a + b)]
2
transforms the nodes xk in [−1, 1] into the corresponding nodes x̃k
in [a, b].
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (20/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Example: Interpolating f (x) = x 2 e x .

3

f(x)
2.5

1.5

0.5

0
0 0.2 0.4 0.6 0.8 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (21/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Example: Interpolating f (x) = x 2 e x .

3

f(x)
2.5 Taylor2(x)

1.5

0.5

0
0 0.2 0.4 0.6 0.8 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (21/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Example: Interpolating f (x) = x 2 e x .

3

f(x)
2.5 Taylor2(x)
Lagrange2EquiSpaced(x)

1.5

0.5

0
0 0.2 0.4 0.6 0.8 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (21/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Example: Interpolating f (x) = x 2 e x .

3

f(x)
2.5 Taylor2(x)
Lagrange2EquiSpaced(x)
Lagrange2Optimal(x)
2

1.5

0.5

0
0 0.2 0.4 0.6 0.8 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (21/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Example: Interpolating f (x) = x 2 e x — The Error.

Deviation: EquiSpaced
Deviation: Optimal
0.1

-0.1

0 0.2 0.4 0.6 0.8 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (22/45)
 Orthogonal Polynomials
Chebyshev Polynomials
Chebyshev Polynomials, Intro & Definitions
Least Squares, redux
Properties

Example: Interpolating f (x) = x 2 e x — The Error.

Degree 2 Degree 3
0.15 0.02
Error: EquiSpaced Error: EquiSpaced
Error: Optimal Error: Optimal
0.1 0.015

0.05 0.01

0 0.005

−0.05 0

−0.1 −0.005

−0.15 −0.01

−0.2 −0.015
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
−5 Degree 5
−3 Degree 4 x 10
x 10 10
1.5 Error: EquiSpaced
Error: EquiSpaced
Error: Optimal
Error: Optimal
1

0.5 5

−0.5 0

−1

−1.5 −5
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (23/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Least Squares: Revisited Introduction

Before we move on to new and exciting orthogonal polynomials

with exotic names... Let’s take a moment (or two) and look at the
usage of Least Squares Approximation.

This section is a “how-to” with quite a few applied example of

least squares approximation...

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (24/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example #1 — Warm-up 1 of 3

First we consider the problem of fitting 1st, 2nd, and 3rd degree
polynomials to the following data:
x = [1.0 1.1 1.3 1.5 1.9 2.1]’
y = [1.84 1.90 2.31 2.65 2.74 3.18]’
matlab¿¿ [First we define the matrices]
A1 = [ones(size(x)) x];
A2 = [A1 x.*x];
A3 = [A2 x.*x.*x ];
[Then we solve the Normal Equations]
pcoef1 = A1\y;
pcoef2 = A2\y;
pcoef3 = A3\y;

Note: The matrices A1, A2, and A3 are “tall and skinny.” Normally we
would compute (An′ · An)−1 (An′ · y ), however when matlab encounters
An\y it automatically gives us a solution in the least squares sense.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (25/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example #1 — Warm-up 2 of 3

We now have the coefficients for the polynomials, let’s plot:

matlab>> xv = 1.0:0.01:2.1;
p1 = polyval(flipud(pcoef1),xv);
p2 = polyval(flipud(pcoef2),xv);
p3 = polyval(flipud(pcoef3),xv);
plot(xv,p3,’k-’,’linewidth’,3); hold on;
plot(x,y,’ko’,’linewidth’,3); hold off

Optimal Linear Approximation Optimal Quadratic Approximation Optimal Cubic Approximation

3.2 3.2 3.2

3 3 3

2.8 2.8 2.8

2.6 2.6 2.6

2.4 2.4 2.4

2.2 2.2 2.2

2 LSQ−error: 0.087656 2 LSQ−error: 0.069900 2 LSQ−error: 0.044729

1.8 1.8 1.8

1 1.2 1.4 1.6 1.8 2 1 1.2 1.4 1.6 1.8 2 1 1.2 1.4 1.6 1.8 2

Figure: The least squares polynomials p1 (x), p2 (x), and p3 (x).

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (26/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example #1 — Warm-up 3 of 3

Finally, we compute the error

matlab>> p1err = polyval(flipud(pcoef1),x) - y;
p2err = polyval(flipud(pcoef2),x) - y;
p3err = polyval(flipud(pcoef3),x) - y;
disp([sum(p1err.*p1err) sum(p2err.*p2err)
sum(p3err.*p3err)])

Which gives us the fitting errors

P1Err = 0.0877, P2Err = 0.0699, P3Err = 0.0447

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (27/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example #2 — Something More Exotic 1 of 2

Consider the same data:

x = [1.0 1.1 1.3 1.5 1.9 2.1]’
y = [1.84 1.90 2.31 2.65 2.74 3.18]’

√
But let’s find the best fit of the form a + b x to this data! Notice
that this expression is linear in its parameters a, b, so we can solve
the corresponding least squares problem!
matlab>> A = [ones(size(x)) sqrt(x)];
pcoef = A\y;
xv = 1.0:0.01:2.1;
fv = pcoef(1) + pcoef(2)*sqrt(xv);
plot(xv,fv,’k-’,’linewidth’,3); hold on;
plot(x,y,’ko’,’linewidth’,3); hold off;

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (28/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example #2 — Something More Exotic 2 of 2

0.5
Optimal a+b x Approximation
3.2

2.8

2.6

2.4

2.2

2 LSQ−error: 0.074920

1.8

1 1.2 1.4 1.6 1.8 2

√
Figure: The best fit of the form a + b x.
We compute the fitting error:
matlab>> ferr = pcoef(1) + pcoef(2)*sqrt(x) - y;
disp(sum(ferr.*ferr))

Err √
Which gives us P{a+b x} = 0.0749

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (29/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Getting Even More Exotic? 1 of 2

As long as the model is linear in its parameters, we can solve the

least squares problem.
Non-linear dependence will have to wait until Math 693a.
We can fit this model:
√
M1 (a, b, c, d) = a + bx 3/2 + c/ x + de sin(x)

Just define the matrix

matlab>> A = [ones(size(x)) x.∧ (3/2) 1./sqrt(x) exp(sin(x))];

and compute the coefficients

matlab>> coef = A\y;

etc...

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (30/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Getting Even More Exotic? 2 of 2

Optimal "Exotic" Model
3.2

2.8

2.6

2.4

2.2

2 LSQ−error: 0.062771

1.8

1 1.2 1.4 1.6 1.8 2

The optimal approximation of this form is
11.0059
16.4133 − 0.9970x 3/2 − √ − 1.1332e sin(x)
x
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (31/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Getting Multi-Dimensional

It seems quite unlikely the model

√
M1 (a, b, c, d) = a + bx 3/2 + c/ x + de sin(x)

will ever be useful.

However, we have forgotten about one important aspect of the
problem — so far our models have depended on only one variable,
x.
How do we go about fitting multi-dimensional data?

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (32/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 1 of 10

matlab>> x=1:0.25:5;
y=1:0.25:5;
[X,Y]=meshgrid(x,y);
Fxy=1+sqrt(X)+Y.∧ 3+0.05*randn(size(X));
surf(x,y,Fxy)

150

100

50

0
6
4 5
4
2 3
2
0 1

Figure: 2D-data set, the vertexes on the surface are our data points.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (33/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 2 of 10

Lets try to fit a simple 3-parameter model to this data

M(a, b, c) = a + bx + cy

matlab>> sz = size(X);
Bm = reshape(X,prod(sz),1);
Cm = reshape(Y,prod(sz),1);
Am = ones(size(Bm));
RHS = reshape(Fxy,prod(sz),1);
A = [Am Bm Cm];
coef = A \ RHS;
fit = coef(1) + coef(2)*X + coef(3)*Y;
fitError = Fxy - fit;
surf(x,y,fitError)

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (34/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 3 of 10

Fit (Model 1) Error (Model 1)

150 30

20
100
10

0
50
−10

0 −20
6 6
4 5 4 5
4 4
2 3 2 3
2 2
0 1 0 1

Figure: The optimal model fit, and the fitting error for the
least squares best-fit in the model space M(a, b, c) = a +
bx + cy . Here, the total LSQ-error is 42,282.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (35/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 4 of 10

Lets try to fit a simple 4-parameter (bi-linear) model to this data

M(a, b, c) = a + bx + cy + dxy

matlab>> sz = size(X);
Bm = reshape(X,prod(sz),1);
Cm = reshape(Y,prod(sz),1);
Dm = reshape(X.*Y,prod(sz),1);
Am = ones(size(Bm));
RHS = reshape(Fxy,prod(sz),1);
A = [Am Bm Cm Dm];
coef = A \ RHS;
fit = coef(1) + coef(2)*X + coef(3)*Y + coef(4)*X.*Y;
fitError = Fxy - fit;
surf(x,y,fitError)

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (36/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 5 of 10

Fit (Model 2) Error (Model 2)

150 30

20
100
10

0
50
−10

0 −20
6 6
4 5 4 5
4 4
2 3 2 3
2 2
0 1 0 1

Figure: The fitting error for the least squares best-fit in the
model space M(a, b, c) = a + bx + cy + dxy . — Still a pretty
bad fit. Here, the total LSQ-error is still 42,282.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (37/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 6 of 10

Since the main problem is in the y -direction, we fit try a

4-parameter model with a quadratic term in y

M(a, b, c) = a + bx + cy + dy 2

matlab>> sz = size(X);
Bm = reshape(X,prod(sz),1);
Cm = reshape(Y,prod(sz),1);
Dm = reshape(Y.*Y,prod(sz),1);
Am = ones(size(Bm));
RHS = reshape(Fxy,prod(sz),1);
A = [Am Bm Cm Dm];
coef = A \ RHS;
fit = coef(1) + coef(2)*X + coef(3)*Y + coef(4)*Y.*Y;
fitError = Fxy - fit;
surf(x,y,fitError)

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (38/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 7 of 10

Fit (Model 3) Error (Model 3)

150 4

2
100
0

50
−2

0 −4
6 6
5 4 5
4
4 4
2 3 2 3
2 2
0 1 0 1

Figure: The fitting error for the least squares best-fit in the
model space M(a, b, c) = a + bx + cy + dy 2 . — We see a
significant drop in the error (one order of magnitude); and
the total LSQ-error has dropped to 578.8.
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (39/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 7 12 of 10

We notice something interesting: the addition of the xy -term to

the model did not produce a drop in the LSQ-error. However, the
y 2 allowed us to capture a lot more of the action.
The change in the LSQ-error as a function of an added term is one
way to decide what is a useful addition to the model.
Why not add both the xy and y 2 always?

xy y2 Both
κ(A) 86.2 107.3 170.5
κ(AT A) 7,422 11,515 29,066
Table: Condition numbers for the A-matrices (and
associated Normal Equations) for the different models.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (40/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 8 of 10

We fit a 5-parameter model with a quadratic term in y

M(a, b, c) = a + bx + cy + dy 2 + ey 3
matlab>> sz = size(X);
Bm = reshape(X,prod(sz),1);
Cm = reshape(Y,prod(sz),1);
Dm = reshape(Y.*Y,prod(sz),1);
Em = reshape(Y.*Y.*Y,prod(sz),1);
Am = ones(size(Bm));
RHS = reshape(Fxy,prod(sz),1);
A = [Am Bm Cm Dm Em];
coef = A \ RHS;
fit = coef(1) + coef(2)*X + coef(3)*Y + coef(4)*Y.*Y +
coef(5)*Y.∧ 3;
fitError = Fxy - fit;
surf(x,y,fitError)

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (41/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 9 of 10

Fit (Model 4) Error (Model 4)

150 0.2

0.1
100

0
50
−0.1

0 −0.2
6 6
4 5 5
4
4 4
2 3 2 3
2 2
0 1 0 1

Figure: The fitting error for the least squares best-fit in the
model space M(a, b, c) = a + bx + cy + dy 2 + ey 3 . — We
now have a pretty good fit. The LSQ-error is now down to
0.9864.
Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (42/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Example: Going 2D 10 of 10

Model LSQ-error κ(AT A)

a + bx + cy 42,282 278
a + bx + cy + dxy 42,282 7,422
a + bx + cy + dy 2 578.8 11,515
a + bx + cy + ey 3 2.695 107,204
a + bx + cy + dy 2 + ey 3 0.9864 1,873,124

Table: Summary of LSQ-error and conditioning of the Normal

Equations for the various models. We notice that additional
columns in the A-matrix (additional model parameters) have a
severe effect on the conditioning of the Normal Equations.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (43/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Moving to Even Higher Dimensions

At this point we can state the Linear Least Squares fitting problem
in any number of dimensions, and we can use exotic models if we
want to.
In 3D we need 10 parameters to fit a model with all linear, and
second order terms
M(a, b, c, d, e, f , g , h, i, j) =
a + bx + cy + dz + ex 2 + fy 2 + gz 2 + hxy + ixz + jyz

With nx , ny , and nz data points in the x-, y -, and z-directions

(respectively) we end up with a matrix A of dimension
(nx · ny · nz ) × 10.

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (44/45)
 Chebyshev Polynomials Examples
Least Squares, redux More than one variable? — No problem!

Ill-conditioning of the Normal Equations

Needless(?) to say, the normal equations can be quite

ill-conditioned in this case. The ill-conditioning can be eased by
searching for a set of orthogonal functions with respect to the
inner product
Z xb Z yb Z zb
hf (x), g (x)i = f (x, y , z)g (x, y , z)∗ dx dy dz
xa ya za

That’s *sometimes* possible, but we’ll leave the details as an

exercise for a dark and stormy night...

Joe Mahaffy, [email protected] Chebyshev Polynomials & Least Squares, redux — (45/45)