CHAPTER V APPROXIMATION OF FUNCTIONS

◊ Objective:
f(x) is given in a very complicated form and is difficult to use,
want to approximate f(x) by a simple polynomial in an interval.
Can we do it? How do we do it accurately?
5.1 Weierstrass Theorem—yes we can!
5.1.1 Weierstrass approximation
◊ Let f(x) be continuous for a ≤ x ≤ b and  > 0.
There exists a polynomial P(x) such that
| f(x) - P(x) | ≤  for a ≤ x ≤ b.
 we can always find a p(x) that is as close to f(x) as we want.
◊ The polynomial can be constructed using Bernstein polynomial,
n k n
Pn(x) =  f(n ) (k) xk(1-x)n-k for 0 ≤ x ≤ 1
k=0

n
where (k) are the binomial coefficients.

◊ It can be shown that lim Pn(x) → f(x) for 0 ≤ x ≤ 1.

n→∞

◊ Example: for f(x) = ex, for -1 ≤ x ≤ 1,

find: Weierstrass’s approximation of f(x) using n=3.
Solution: let 0≤ =(x+1)/2≤ 1. For n=3, =k/n=0, 1/3, 2/3 & 1.
 f(=0)=f(x=-1)= e-1, f(=1/3) =f(x=-1/3)= e-1/3,
f(=2/3) =f(x=1/3)= e1/3, & f(=1) =f(x=1)= e1
w3(x) = e-1(1-)3 +e-1/33(1-)2 + e1/332 (1-) + e13

Notes organized by Prof. Renwei Mei, University of Florida 1

5.1.2 Comparison with Taylor series expansion
◊ Comment: Bernstein polynomial converges very slowly!
* Thus it is impractical to use them. (See next graph)
* Must find something else that is practical and accurate.
◊ Question: Is Taylor series expansion a good alternative?
◊ Taylor series expansion is another way to approximate f(x).
Consider f(x) = ex for -1 ≤ x ≤ 1.
1 2 1 3
Then P3(x) = 1+ x + x + x for f(x) = ex near x=0.
2 6
Error: E3(x) = ex - (1 + x + x2/2+ x3/6)
= x4e/4!,  [0, x] or [x, 0]
3 Error
f(x) Third order 0.04
2.5 polynomial 0
2 approx. -0.04 Taylor Series
-0.08 Weierstrass
1.5
-0.12
1 exact Error in
-0.16
Weiers. third order
0.5 -0.2
TS3 polynomial approx.
0 -0.24
-1 -0.5 0 0.5 1 x -1 -0.5 0 0.5 1 x

Comparison: 3rd order TS expansion & Weierstrass construction with f(x).

Comparison of error distributions of two 3rd order polynomial approximations.

Problem: E3(x)~ x4 is not evenly distributed in the interval [-1, 1].

Want: a better way to obtain Pn(x) so that Error(x) is more
evenly distributed but max{|E(x)|} is smaller.
Notes organized by Prof. Renwei Mei, University of Florida 2
5.2 Minimax Approximation
5.2.1 Illustration of method
◊ Consider f(x) = ex, x  [-1, 1]
* Approximate f(x) by:
*
q1(x) = a0 + a1 x for x  [-1, 1]

so that max | E(x) | = max | f(x) - q*(x) |

x [-1‚1] 1
x [-1‚1]

= smallest among all possible values of (a0, a1).

* Sketch:

3
exp(x)
2.5 q1*(x)

1.5

0.5
x3
0
-1.5 -1 -0.5 0 0.5 1 x

*
* Apparently, there are three locations: -1, x3, 1, where |q1(x)-f(x)|

can be large. Intuitively, we want to spread out the error

uniformly over the interval of interest. Thus we should force
E(x= -1) = E(x=1) = 1
and E(x = x3 = z) = -1

Notes organized by Prof. Renwei Mei, University of Florida 3

  e-1 - (a0 - a1) = 1 (i)
e - (a0 + a1) = 1 (ii)
ez - (a0 + a1z) = -1 (iii)
Since the error has a local maximum at x3, we have
dE
|x=z=x3 = 0  ez - a1 = 0. (iv)
dx
(i) - (ii)  a1 = sinh(1) = 1.175201;
(iv)  x3 = ln(a1) = 0.161439
 a0 = cosh(1)/2 + a1(1- x3) = 1.264279, 1 = 0.278802.
Shown below is E1(x) = ex - (1.264279+1.175201x).

0.4
Error
0.2 Minimax

-0.2

-0.4
-1 -0.5 0 0.5 1 x

* Indeed the negative error and positive error are evened out.

Notes organized by Prof. Renwei Mei, University of Florida 4

 *
◊ If we try q3(x) = a0 + a1x+ a2x2 + a3x3 for x  [-1, 1]

& minimize max |E(x)| = max |f(x) - q*(x)|

x [-1‚1] 3
x [-1‚1]
*
 q3(x) = 0.994579 + 0.995668x + 0.542973 x2 + 0.179533x3

(after a lot of tedious algebra!)

*
* The error E3(x) = f(x) - q3(x) is shown below together with the

error from the 3rd order Taylor series expansion.

0.06
Error Error in third order
polynomial approx.
0.04
Minimax
0.02 Taylor Series

-0.02
-1 -0.5 0 0.5 1 x

* The maximum error |E3(x)| is 0.00553 « 0.0516 from Taylor

series expansion.
* Clearly, there is an improvement over the Taylor series
expansion method for polynomials of same (3rd) order.

Notes organized by Prof. Renwei Mei, University of Florida 5

5.2.2 Comments on the minimax approximation method
◊ Clearly, Taylor series approach, while simple, does not lead to
an evenly distributed error. Although the error near x=x0 is very
small, its maximum error in the interval is much higher than the
one obtained via minimax approach.
◊ HOWEVER, in general, it is very tedious to find (a0, a1, a2, ...)
in minimizing max|E(x)| = max |f(x) - (a0 +a1x +a2x2 +...)|
◊ Analytically, it depends on how complicated f(x) is.
◊ Use of absolute function is not suited for analytical treatment.
◊ We must find an alternative robust approach that performs
similarly to the minimax approximation.
◊ Accuracy of the minimax approximation:
Given f(x), x  [a, b], the error bound in approximating f(x)
by using minimax method is
[(b − a) / 2]n +1
n ( f )  max | f ( n +1) ( x) |
n
(n + 1)!2 a  x b

This error bound does not always become smaller with

increasing n, but it will give a fairly accurate bound for
many common functions f (x).

Notes organized by Prof. Renwei Mei, University of Florida 6

5.3 General Idea of Least Square Approximation
5.3.1 Inner product and L2 –norm of functions
◊ Let w(x) = weight function >0 on x  [a, b]
b n
a |x| w(x) dx = finite for all n≥0
• Inner product of two functions f & g:

(f, g) = ab w(x) f(x) g(x) dx

• L2 -norm of f(x):

|| f ||2 = [ ab w(x) f2(x) dx]1/2 = (f, f)

• Cauchy-Schwartz inequality: |(f, g)| ≤ || f ||2 || g ||2

• Triangle inequality: || f + g ||2 ≤ || f ||2 + || g ||2
• Examples of common w(x):
w(x) = 1; w(x) = 1/ 1-x2, x[-1, 1]
w(x) = e-x,0 ≤ x < ∞; w(x) = e-x2, -∞ < x < ∞
• Orthogonal sequence of functions {i(x)}: if (i, j) = 0 for i≠j.

Notes organized by Prof. Renwei Mei, University of Florida 7

5.3.2 Illustration of least square approximation for functions
◊ Consider L2-norm of the error (w(x)=1):
*
For f(x) = ex, r1(x) = a0 + a1x
2
the overall error is: E = −11 [f(x) - r(x)] dx

To minimize E, we require
∂E ∂E
=0 and =0 
∂a0 ∂a1
∂E 1 ∂ 2
=  ∂a [f(x) - a0 -a1x] dx = -2 −11 [ex - a0 -a1x]dx = 0
∂a0 0
−1
∂E 1 ∂ 2 1
=  ∂a [f(x) - a0 - a1x] dx = 2  [ex - a0 - a1x](-x)dx = 0.
∂a1 1
−1 −1
1 1
 a0 −1 dx = −1 ex dx = e1 – e-1 = 2.350402

& a1 −11 x2dx = −11 e x xdx = 2e-1 = 0.735789

 a0 = 1.175201 and a1 = 1.103638.

*
r1(x) = 1.175201 + 1.103638x

Although max |E(x)| = 0.43944 > 0.2788 (from minimax),

axb

the procedure is much easier.

Note: Taylor series expansion gives P1(x) = 1+x which has
maximum error of 0.7183> max{LSQ error}.

Notes organized by Prof. Renwei Mei, University of Florida 8

* Comparison of least square and minimax approximations
with the given function:

3
f(x)
2.5

2 Linear approximations

1.5

1
exact
0.5 LSQ
Minimax
0
x
-1 -0.5 0 0.5 1

* Comparison of errors in the linear approximations:

0.8
Error
0.6 LSQ
Minimax
0.4
Taylor series
0.2

-0.2

-0.4
-1 -0.5 0 0.5 1 x

Notes organized by Prof. Renwei Mei, University of Florida 9

5.3.3 Generalization to least square approximation
◊ Given a continuous function f(x) on x[a, b],
*
FIND a polynomial rn(x) of degree ≤n that minimizes
b * 2
a w(x) [f(x) - rn(x)] dx
among all polynomials of rn(x) of degree ≤n for the given w(x).
n
* Example: For w(x)=1, approximate f(x) using f(x)~ r(x) =  ajxj
j =0
n
by minimizing E = ab [f(x) -  aj xj]2 dx
j =0
∂E
Solution: Minimization of Error E  =0
∂aj
n
 -2 ab [f(x) -  ai xi] xj dx, j=0, 1, 2,...n
i =0
n b b
  ai a xi+j dx = a f(x) xj dx
i =0
n 1 b
i.e.  ai i+j+1 [bi+j+1 - a i+j+1] = a f(x) xj dx  cj
i =0
n
  Aij ai = cj
i =0

1
Discussion: For a=0, b=1, the coefficient matrix is: Aij = i+j+1

Aij = Hilbert matrix which is ill-conditioned.

 solution for ai can be in great error!
*
 need to develop alternative method to find rn(x).

Notes organized by Prof. Renwei Mei, University of Florida 10

5.4 Orthogonal Polynomial
5.4.1 Why are monomials bad for approximations?
◊ Why are monomials xn (n≥0) in the above example bad choices?

1.2

1.0

0.8
n
x
0.6
n=2
0.4
3
0.2 5

0.0
0.0 0.2 0.4 0.6 0.8 1.0 1.2
x

!!! xn look too much alike for large n.

 equations are nearly dependent on each other.
Instability of the system for Aij ai = cj (ill-conditioned)
n
◊ We thus want to represent f(x) using rn(x) =  i i(x),
i=0

where i(x) is independent of j(x) (i≠j),

that is, i(x) is orthogonal to each other.
* Can we find such a set of orthogonal i(x)?
* If we can, how do we solve for i reliably?

Notes organized by Prof. Renwei Mei, University of Florida 11

5.4.2 Gram-Schmidt orthonormalization
 Gram-Schmidt orthogonalization, also called the Gram-
Schmidt process, is a procedure which takes a
nonorthogonal set of linearly independent functions and
constructs an orthogonal basis over an arbitrary interval
with respect to an arbitrary weighting function w(x).
 There exists a sequence of polynomials {n(x)|n≥0} with degree

(n) = n for all n and

(n, m) = 0 for n≠m, n>0 and m>0.
• In fact, we can construct such a sequence with
i. (n, n) = 1;
ii. coefficient of xn (highest power) in n(x) is positive.
Then, this {n(x)} is unique for a given w(x).
• Procedure:
* n=0: 0(x) = c.

Normalization requires: (0, 0) = c2 ab w(x) dx = 1

-1/2
 c = [ ab w(x) dx ] .

* n=1: Let 1(x) = x + a1,0 0(x);

enforce orthogonality: (1, 0) =0  (x, 0) + a1,0 (0, 0) = 0

 a1,0 = -(x, 0)/(0, 0) = -(x, 0) = -c ab w(x) xdx

-1/2
 a1,0 = - ab w(x)xdx / [ ab w(x) dx] .

Notes organized by Prof. Renwei Mei, University of Florida 12

 Now define 1(x)  1(x) / ||1(x) ||2
 || 1(x) ||2 = 1 & (0, 1) =0.
* Suppose 2(x), 3(x),…, n-1(x) have been obtained.
* After n-1(x) is found,
a) Let n(x) = xn + an,n-1 n-1(x) + an,n-2 n-2(x) + ... + an,0 0(x)
with (i, j) = 0 for i≠j.
then enforce (n, j) = 0  (xn, j) + an,j (j, j) = 0

 an,j = - (xn, j) / (j, j) = -(xn, j)

b) Now define n(x) = n(x) / ||n(x) ||2,
we have || n(x) ||2 =1 & (n, j) = 0, j≠n.
• Example: Given w(x) =1, [a, b] = [-1, 1}
Then, c = [ −11 w(x) dx ]-1/2 = 1/ 2 = 0(x).

a1,0 = -(x, 0) = -c −11 x dx = 0.

1(x) = x  (
|| 1(x) ||2 = −11 x 2 dx )1 / 2 = (2 / 3)1 / 2
1(x) = 3 / 2 x
1
Similarly 2(x) = 5 / 2 (3x2 - 1)... Legendre polynomials.
2
• Note: (j, j) =1 is not necessary as long as we know the value
of || j(x) ||2 which is usually the case for well-known
orthogonal polynomials.

Notes organized by Prof. Renwei Mei, University of Florida 13

5.4.3 Various common orthogonal polynomials
i) Legendre polynomials
* Weight function: w(x) = 1 on [-1, 1]
* Legendre functions are solutions to Legendre's differential eqn.:

This ordinary differential equation is frequently encountered in

physics and other technical fields. In particular, it occurs when
solving Laplace's equation (and related partial differential
equations) in spherical coordinates.
(-1)n dn 2 n
* General expression: Pn(x) = n n[(1-x ) ], n≥1.
2 n! dx
* The first 6 Legendre polynomials are
P0(x) 1
P1(x) x
P2(x) (3x2-1) /2
P3(x) (5x3-3x) /2
P4(x) (35x4-30x2 +3) /8
P5(x) (63x5-70x3 +15x) /8
P6(x) (231x6-315x4 +105x2-5) /16

Notes organized by Prof. Renwei Mei, University of Florida 14

 * Orthogonality of Pn(x):
we can show that (Pn(x), Pm(x)) = 0 for n≠m
i.e. Pn(x) is orthogonal on [-1, 1] w.r.t. w(x) = 1.
2
When n = m, (Pn(x), Pn(x)) = 2n+1 (≠1; OK!)

ii) Chebyshev polynomial:

* Weight: w(x) = (1-x2)-1/2, on [-1, 1]

* General expression: Tn(x) =cos[n cos-1(x)], n≥0.
Let =cos-1(x)  T n(x) = cos(n).
 The first few Chebyshev polynomials of the first kind
T0(x) 1
T1(x) x =cos()
T2(x) 2x2-1 =cos(2)=2cos2-1
T3(x) 4x3-3 x =cos(3)=4cos3- 3cos
T4(x) 8x4-8 x2 +1
T5(x) 16x5-20x3 +5x
T6(x) 32x6-48x4 +18x2-1

Notes organized by Prof. Renwei Mei, University of Florida 15

  0‚ n≠m
‚ m=n=0
* Orthogonality: (Tn, Tm) =  
 2‚ n=m>0

* Recursion: Tn+1(x) = 2xTn (x) - Tn-1(x), n≥1

• Chebyshev polynomials are important in approximation
theory because the roots of the Chebyshev polynomials of
the first kind, which are also called Chebyshev nodes, are
used as nodes in polynomial interpolation.
• The resulting interpolation polynomial minimizes the
problem of Runge's phenomenon and provides an
approximation that is close to the polynomial of best
approximation to a continuous function under the
maximum norm.

Notes organized by Prof. Renwei Mei, University of Florida 16

5.5 Least Square Approximation Using Orthogonal Polynomials
5.5.1 General formulation
• Given f(x) (≠ a polynomial),
want to approximate f(x) by superposition of {i(x)} as:
n
r(x) =  ai i(x)
i =1

• L2-norm error in approximating f(x) by r(x) is

n
2
E2 = || f - r ||2 = ab w(x) [f(x) -  ai i(x)] dx
i =1

• Minimization of E2:
∂E2 n
b
= 0  -2 a w(x) [f(x) -  ai i(x)] j(x) dx = 0
∂aj i =1

n
 ab w(x) f(x) j(x) dx =  ai ab w(x) i(x)j(x)dx
i =1

n
=  ai(i, j) = aj || j ||2.
i =1

 aj = ab w(x) f(x) j(x)dx / || j ||2

i.e. aj = (f, j) / (j, j)

• Since we know the value of || j ||2 for j=0, 1, 2…, the only work
needed is the evaluation of (f, j) which can be evaluated using
standard numerical integration methods.

Notes organized by Prof. Renwei Mei, University of Florida 17

5.5.2 Applications
i) Using Legendre polynomials Pn(x).
• w(x) = 1 with [a, b] being arbitrary
but Pn(x) is only defined on [-1, 1].
1
• Transformation: x= [b+a +(b-a) t] (*)
2
so that t  [-1, 1]  x  [a, b].
1
• Substituting x = [b+a +(b-a) t] into f(x), we get
2
b+a+(b-a)t
f(x) = f( ) = F(t).
2
• L2-norm error in approximating f(x) by rn(x), a linear
combination of Pn(x), is
b 2
E2 = || f(x) - rn(x) ||2 =  [f(x) - rn(x)] dx
a

b−a 1 2
= 
2 −1
[F(t) - Rn(t)] dt

b+a+(b-a)t
where Rn(t) = rn( ) = rn(x).
2
n
• Rn(t) =  ai Pi(t), t  [-1, 1]
i =1
2i+1
 ai = (F, Pi) / (Pi, Pi) = 2 (F, Pi).

• The main work is the evaluation of (F, Pi).

Often, numerical integration is needed.

Notes organized by Prof. Renwei Mei, University of Florida 18

ii) Using Chebyshev polynomial
• If [a, b] ≠ [-1, 1], use the same transformation given by (*).
 ‚ n=0
• Since (Tn, Tn) =  
 2‚ n>0
Now construct Cn(x) to approximate f(x):
n
1
Cn(x) =  ' cj Tj(x) (' means taking 2c0, instead of c0, for j=0)
j =0

2 1 f(x)Tj(x)
 cj =  2 dx, for j = 0, 1,... n.
 −1 1-x

• To evaluate cj, use standard change of variables

x = cos (0≤  ≤ )
2 0 f(cos)cos(j)
 cj =  (-sin) d
 sin

2
=  cos(j) f() d
0
n
• Finally, Cn(x) = Cn(cos) =  ' cjcos(j).
j =1

• Example: f(x) = ex , x[-1, 1]

2 cos
Solution: cj =  cos(j) e d
0

Numerically evaluate the above (using trapezoidal rule)=>

c0 = 2.5321376, c1 = 1.13031812,
c2 = 0.27149534, c3 = 0.04433685,

Notes organized by Prof. Renwei Mei, University of Florida 19

 c4 = 0.00547424, c5 = 0.000542926
1
 C1(x) = c + c1x = 1.26607 + 1.13032x
2 0
1
C3(x) = 2 c0 + c1x + c2 (2x2-1) + c3(4x3-3x)

= 0.994571 + 0.997328x + 0.542991x2 + 0.177347x3

*
~ very close to q3(x) obtained from minimax process:
*
q3(x) = 0.994579 + 0.995668x + 0.542973 x2 + 0.179533x3

2.5

1.5 f(x)
C1(x)
C2(x)
1 C3(x)

0.5

0
-1.5 -1 -0.5 0 0.5 1 1.5

Notes organized by Prof. Renwei Mei, University of Florida 20

 3

2.5
f(x)
2
C4(x)

1.5 C3(x)

0.5

0
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 x

0.008
Error_q*3
0.006 Error_C3
Error_C4
0.004

0.002

-0.002

-0.004

-0.006

-0.008
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 x

* The errors in the approximations show that:

*
i) q3(x) and C3(x) are very close.

ii) C4(x) has much less error than C3(x).

Notes organized by Prof. Renwei Mei, University of Florida 21

* The dependence of coefficients ck (0k14) on k for the complete
set (until ck+1 is below machine error) are shown below:
1.E+01
c(k)
1.E-01

1.E-03

1.E-05

1.E-07

1.E-09

1.E-11

1.E-13

1.E-15
0 5 10 15 k

We observe an exponential decrease of the coefficient ck as k

increases for this continuous function f(x)=exp(-x).
* This type of behavior is called exponential convergence.
* It is typical for smooth functions.
* Question: what happens if f(x) has singularity in the interval?

Notes organized by Prof. Renwei Mei, University of Florida 22