GENG 300 NUMERICAL METHODS

Dr. Mohammad Aman Ullah

 CURVE FITTING AND INTERPOLATION

Topic 4

2
 Curve fitting: Objectives

• Familiarizing yourself with some basic descriptive

statistics and the normal distribution.
• Knowing how to compute the slope and intercept
of a best fit straight line with linear regression.
• Knowing how to compute and understand the
meaning of the coefficient of determination and
the standard error of the estimate.
• Understanding how to use transformations to
linearize nonlinear equations so that they can be
fit with linear regression.
 Statistics Review
Measure of Location
• Arithmetic mean: the sum of the individual data points (yi)
divided by the number of points n:

y
¦y i

• Median: the midpoint of a group of data.

• Mode: the value that occurs most frequently in a group of

data.
 Statistics Review
Measures of Spread
• Standard deviation:
St
sy
n 1
where St is the sum of the squares of the data residuals:
¦ y y
2
St i

and n-1 is referred to as the degrees of freedom.

• Variance:
¦ ¦ yi  ¦ yi / n
2 2
yi  y
2
2
sy
n 1 n 1
• Coefficient of variation:
sy
c.v. u100%
y
 Curve fitting
• Fit a curve to a given discrete data set that is the best
representation of the data set.

Two General Approaches:

Data exhibit a significant degree of scatter
Find a single curve that represents the
general trend of the data.
Data is very precise. Pass a curve(s) exactly
through each of the points.
 Linear Regression
• Fitting a straight line to a set of paired observations:
(x1, y1), (x2, y2),…,(xn, yn)
e = Error

yi : measured value
e : error

yi = a0 + a1 xi + e
e = yi - a0 - a1 xi
a1 : slope
a0 : intercept Line equation
y = a0 + a1 x
 Choosing Criteria For a “Best Fit”
• Minimize the sum of the residual errors
for all available data? Regression
n n Inadequate! line

¦e ¦(y
i 1
i
i 1
i  ao  a1 xi ) (see ÎÎÎ)

• Sum of the absolute values?

n n Inadequate!
¦e ¦ y
i 1
i
i 1
i  a0  a1 xi (see ÎÎÎ)

• How about minimizing the distance that

an individual point falls from the line?
This does not work either! see ÎÎÎ
 Linear Least-Squares Regression
• Linear least-squares regression is a method to
determine the “best” coefficients in a linear model for
given data set.
• “Best” for least-squares regression means
minimizing the sum of the squares of the estimate
residuals. For a straight line model, this gives:
n n
Sr ¦e ¦ y  a
2
i i 0  a1 xi
2

i 1 i 1

• This method will yield a unique line for a given set of

data.
 Least-Squares Fit of a Straight Line
n n
Minimize error : Sr ¦e
i 1
2
i ¦ i 0 1i
( y
i 1
 a  a x ) 2

wS r
2¦ ( yi  ao  a1 xi ) 0 Ÿ ¦ y ¦a ¦a x 0
wao
i 0 1 i

wS r
2¦ >( yi  ao  a1 xi ) xi @ 0 Ÿ ¦ y x ¦a x ¦a x 2
0
wa1
i i 0 i 1 i

Since ¦a 0 na0

(1) na0  ¦x i a1 ¦y i Normal equations which can

(2) ¦ xi a0  ¦x 2
i a1 ¦yx i i
be solved simultaneously
 Least-Squares Fit of a Straight Line
n n
Minimize error : Sr ¦e
i 1
2
i ¦ i 0 1i
( y 
i 1
a  a x ) 2

Normal equations which can be solved simultaneously

na0  ¦x i a1 ¦yi

¦x i a0  ¦x 2
i a1 ¦yx
i i

ª n
«
¦ x º» ªa º ª« ¦ y º»
i 0 i
«a »
¬¦ xi ¦ ¼ ¬ ¼ ¬¦ y x ¼
2
x i 1 i i

n¦ xi yi  ¦ xi ¦ yi Mean values
a1
n¦ x  ¦x
2 2
i i

using (1), a0 can be expressed as a0 y  a1 x

 Example
V F n¦ xi yi  ¦ xi ¦ yi 8 312850  360 5135
(m/s) (N) a1 19.47024
n¦ x  ¦x 8 20400  360
2 2
2
i i
i xi yi (xi)2 x iy i
a0 y  a1 x 641.875 19.47024 45 234.2857
1 10 25 100 250

2 20 70 400 1400 Fest 234.2857 19.47024v

3 30 380 900 11400

4 40 550 1600 22000

5 50 610 2500 30500

6 60 1220 3600 73200

7 70 830 4900 58100

8 80 1450 6400 116000

6 360 5135 20400 312850

 Quantification of Error
• Recall for a straight line, the sum of the squares of the
estimate residuals:

n n
Sr ¦e ¦ y  a
2
i i 0  a1 xi
2

i 1 i 1

• Standard error of the estimate:

Sr
s y/ x
n2
 Standard Error of the Estimate
• Regression data showing (a) the spread of data around the
mean of the dependent data and (b) the spread of the data
around the best fit line:

• The reduction in spread represents the improvement due to

linear regression.
 “Goodness” of our fit

• Sr = Sum of the squares of residuals around the regression line

• St = total sum of the squares around the mean
• (St – Sr) quantifies the improvement or error reduction due to describing
data in terms of a straight line rather than as an average value.

r : correlation coefficient
r 2 St  S r St ¦ i
( y  y ) 2

r2 : coefficient of determination St n
Sr ¦(y
i 1
i  a0  a1 xi ) 2

• For a perfect fit Sr=0 and r = r2 = 1

signifies that the line explains 100 percent of the variability of the data.
• For r = r2 = 0 Î Sr=St Î the fit represents no improvement
 Example
V F
(m/s) (N)
Fest 234.2857 19.47024v
i xi yi a0+a1xi (yi- ј)2 (yi-a0-a1xi)2 St ¦ yi  y
2
1808297
1 10 25 -39.58 380535 4171 Sr ¦ y a
i 0  a1 xi
2
216118
2 20 70 155.12 327041 7245
1808297
3 30 380 349.82 68579 911 sy 508.26
8 1
4 40 550 544.52 8441 30 216118
s y/ x 189.79
5 50 610 739.23 1016 16699 82
6 60 1220 933.93 334229 81837 1808297  216118
r2 0.8805
7 70 830 1128.63 35391 89180 1808297
8 80 1450 1323.33 653066 16044 88.05% of the original uncertainty
has been explained by the
6 360 5135 1808297 216118
linear model
 Nonlinear Relationships
• Linear regression is predicated on the fact that the
relationship between the dependent and independent
variables is linear - this is not always the case.

• Three common examples are:

exponential : y D1eE1 x

power : y D2 x E2
x
saturation - growth - rate : y D 3
E3  x
Transformation Examples
 Data to be fit to the power equation:

y D 2 x E2
Ÿ log y E 2 log x  log D 2

x y log x log y
1 0.5 0 -0.301
2 1.7 0.301 0.226
3 3.4 0.477 0.531
4 5.7 0.602 0.756
5 8.4 0.699 0.924

Linear Regression yields the

result:
log y = 1.75*log x – 0.300
After (log y)–(log x) plot is
obtained
Find D and E using: E2 = 1.75 log D2 = - 0.3Î D2 =
Slope E 2 intercept log D 2 0.5
 Polynomial Regression
• Some engineering data is poorly represented by a straight line. A curve
(polynomial) may be better suited to fit the data. The least squares method can
be extended to fit the data to higher order polynomials.
• As an example let us consider a second order polynomial to fit the data points:

y a0  a1 x  a2 x 2
n n
Minimize error : Sr ¦e
i 1
2
i ¦ i 0 1 i 2 i)
(
i 1
y  a  a x  a x 2 2

wS r
2¦ ( yi  ao  a1 xi  a x ) 0
2 na0  ¦x i a1  ¦x 2
i a2 ¦y i
wao
2 i

wS r
2¦ xi ( yi  ao  a1 xi  a2 xi2 ) 0 ¦ xi a0  ¦ i a1 
x 2
¦ i a2
x 3
¦x y
wa1 i i

wS r
2¦ xi2 ( yi  ao  a1 xi  a2 xi2 ) 0
wa2 ¦ i a0 
x 2
¦ i a1 
x 3
¦ i a2
x 4
¦ i yi
x 2

20
 Polynomial Regression
• To fit the data to an mth order polynomial, we need to solve the following
system of linear equations ((m+1) equations with (m+1) unknowns)

ª n ¦x i  ¦x º ª a0 º
m
i ª ¦ yi º
« »« » « »
« ¦ xi ¦x ¦x « ¦ xi yi »
2 m 1
i  » « a1 »
i
«     »«  » «  »
« mm » « » « »
«¬¦ i ¦ i ¦ xi »¼ ¬am ¼ «¬¦ xi yi »¼
m m 1 m
x x 

Matrix Form
Multiple Linear Regression
• A useful extension of linear regression is the case where y is a linear function
of two or more independent variables. For example:
y = ao + a1x1 + a2x2 + e
• For this 2-dimensional case, the regression line becomes a plane as shown
in the figure below.
Multiple Linear Regression
n n
Example (2 - vars) : Minimize error : Sr ¦e ¦(y
i 1
2
i
i 1
i  a0  a1 x1i  a2 x2i ) 2

wS r
2¦ ( yi  ao  a1 x1i  a2 x2i ) 0 na0  ¦x 1i a1  ¦x 2i a2 ¦y i
wao
wS r
wa1
2¦ x1i ( yi  ao  a1 x1i  a2 x2i ) 0 ¦x1i a0  ¦x 2
1i a1  ¦x x a2
1i 2 i ¦x 1i yi

wS r
2¦ x2i ( yi  ao  a1 x1i  a2 x2i ) 0
wa2 ¦x 2i a0  ¦x 1i x2 i a1  ¦x 2
2i a2 ¦x 2i yi

ª n
«
¦x 1i ¦x 2i º ª a0 º
»« »
ª ¦ yi º
« »
« ¦ x1i ¦x ¦x x « ¦ x1i yi »
2
1i 1i 2 i » « a1 »
« ¦ x2 i ¦x x ¦x 2
»« »
2 i ¼ ¬ a2 ¼
« ¦ x 2 i yi »
¬ 1i 2 i ¬ ¼
Multiple Linear Regression ª n
«
¦x 1i ¦x 2i º ª a0 º
»« »
ª ¦ yi º
« »
« ¦ x1i ¦x ¦x x « ¦ 1i i »
2
1i 1i 2 i » « a1 » x y
Example « ¦ x2 i ¦x x ¦x 2
»« »
2 i ¼ ¬ a2 ¼
« ¦ x 2 i yi »
¬ 1i 2 i ¬ ¼
The following data is calculated from the equation y = 5 + 4x1 - 3x2
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

Use multiple linear regression to fit this data.

Solution:
this system can be solved using
ª 6 16.5 14 º ªa0 º ª 54 º
«16.5 76.25 48» « a » «243.5» Gauss Elimination.
« »« 1 » « » The result is: a0=5 a1=4 and a2= -3
«¬ 14 48 54»¼ «¬a2 »¼ «¬ 100 »¼ y = 5 + 4x1 -3x2
 INTERPOLATION

Topic 4 continued…

1
 Polynomial Interpolation
• You will frequently have occasions to estimate
intermediate values between precise data points.
• The function you use to interpolate must pass
through the actual data points - this makes
interpolation more restrictive than fitting.
• The most common method for this purpose is
polynomial interpolation, where an (n-1)th order
polynomial is solved that passes through n data
points: f ( x) a  a x  a x 2  ...  a x n 1
1 2 3 n

MATLAB version :
f ( x) p1 x n 1  p 2 x n  2  ...  p n 1 x  p n
 f ( x) a1  a2 x  a3 x 2    an x n 1

• There is one and only one nth-order polynomial that fits n+1
points
 Polynomial Interpolation Problems
• One problem that can occur with solving for the coefficients of
a polynomial is that the system to be inverted is in the form:

ª x1n 1 x1n  2 ... x1 1º ­ p1 ½ ­ f x1 ½

« n 1 »° ° ° f x °
« x2 x 2n  2 ... x2 1» ° p 2 ° °° 2 °°
° °
«     » ®  ¾ ®  ¾
« n 1 »° °f x °
« x n 1 x nn12  x n 1 1» p n 1 °
° ° ° n 1
°
« x n 1 x nn  2  xn »
1¼ °¯ p n °¿ °¯ f x n °¿
¬ n

• Matrices such as that on the left are known as Vandermonde

matrices, and they are very ill-conditioned - meaning their
solutions are very sensitive to round-off errors.
 Determining Coefficients
• Since polynomial interpolation provides as many basis
functions as there are data points (n), the polynomial
coefficients can be found exactly using linear algebra.

• Two methods:
• The Newton polynomial
• The Lagrange polynomial

• MATLAB’s built in polyfit and polyval commands can also be

used - all that is required is making sure the order of the fit for
n data points is n-1.
 Newton Interpolating Polynomials
• Another way to express a polynomial interpolation is to use
Newton’s interpolating polynomial.

• The differences between a simple polynomial and Newton’s

interpolating polynomial for first and second order
interpolations are:

Order Simple Newton

1st f1 ( x) a1  a2 x f1 ( x) b1  b2 ( x  x1 )
2nd f 2 ( x) a1  a2 x  a3 x 2 f 2 ( x) b1  b2 ( x  x1 )  b3 ( x  x1 )( x  x2 )
Newton Interpolating Polynomials (cont..)
• The first-order Newton
interpolating polynomial may
be obtained from linear
interpolation and similar
triangles, as shown.
• The resulting formula based
on known points x1 and x2
and the values of the
dependent function at those
points is:
f x2  f x1
f1 x f x1  x  x1
x2  x1
Slope and a finite divided difference
approximation to 1st derivative
Newton Interpolating Polynomials (cont)
• The second-order Newton
interpolating polynomial
introduces some curvature to
the line connecting the points,
but still goes through the first
two points.
• The resulting formula based on
known points x1, x2, and x3 and
the values of the dependent
function at those points is:
f x 3  f x2 f x2  f x1

f x2  f x1 x 3  x2 x2  x1
f2 x f x1  x  x1  x  x1 x  x2
x2  x1 x3  x1
General Form of Newton’s Interpolating Polynomials
f n ( x) f ( x0 )  ( x  x0 ) f [ x1 , x0 ]  ( x  x0 )( x  x1 ) f [ x2 , x1 , x0 ]
   ( x  x0 )( x  x1 )  ( x  xn 1 ) f [ xn , xn 1 , , x0 ]
b0 f ( x0 )
b1 f [ x1 , x0 ]
b2 f [ x2 , x1 , x0 ]

bn f [ xn , xn 1 , , x1 , x0 ]
f ( xi )  f ( x j )
f [ xi , x j ] Bracketed function evaluations
xi  x j are finite divided differences
f [ xi , x j ]  f [ x j , xk ]
f [ xi , x j , xk ]
xi  xk

f [ xn , xn 1 ,  , x1 ]  f [ xn 1 , xn  2 ,  , x0 ]
f [ xn , xn 1 ,  , x1 , x0 ]
xn 18 x0
Chapter 10
 Divided Differences
• Divided difference are calculated as follows:
f xi  f x j
f >xi , x j @
xi  x j
f >xi , x j @ f >x j , xk @
f >xi , x j , xk @
xi  x k
f >xn , xn1 , , x2 @  f >xn1 , xn2 , , x1 @
f >xn , xn1 , , x2 , x1 @
xn  x1
• Divided differences are calculated using divided
difference of a smaller number of terms:
 Example
xi F(xi) xi F[xi] F[xi, xj] F[xi, xj, xk]

0 2 0 2 1 1

1 3 1 3 3

2 6 2 6

Data table 1st Order 2nd Order

difference difference
xi F[xi] F[xi, xj] F[xi, xj, xk]
f xi  f x j
0 2 1 1 >
f xi , x j @ xi  x j
1 3 3 3 2
1
2 6 1 0
xi F[ ] F[ , ] F[ , , ] f xi  f x j
>
f xi , x j @ xi  x j
0 2 1 1

1 3 3 63
3
2 6 2 1

>
f xi , x j , xk @ > @ >
f xi , x j  f x j , xk @
xi  xk
xi F[ ] F[ , ] F[ , , ]
0 2 1 1

1 3 3 3 1
1
2 6 20
xi F[ ] F[ , ] F[ , , ]
0 2 1 1

1 3 3

2 6

f 2 ( x) 2  1( x  0)  1( x  0)( x  1)

f 2 ( x) 2 x x  x2

x 2
2
Ordering the points should not affect the divided difference:

f [ x0 , x1 , x2 ] f [ x1 , x2 , x0 ] f [ x2 , x1 , x0 ]

Exercise: x f(x)

0 1

• Find a polynomial to 2 5
interpolate the data.
4 1

6 6
 Lagrange Interpolating Polynomials
• Another method that uses shifted value to express an
interpolating polynomial is the Lagrange interpolating
polynomial.
• The differences between a simple polynomial and
Lagrange interpolating polynomials for first and
second order polynomials is:
• Order Simple Lagrange
1st f1 (x) a1  a2 x f1 (x) L1 f x1  L2 f x2
2nd f2 (x) a1  a2 x  a3 x 2 f2 (x) L1 f x1  L2 f x2  L3 f x 3

where the Li are weighting coefficients that are

functions of x.
 Lagrange Interpolating Polynomials
• The first-order Lagrange
(cont…)
interpolating polynomial may
be obtained from a weighted
combination of two linear
interpolations, as shown.
• The resulting formula based on
known points x1 and x2 and the
values of the dependent
function at those points is:

f1 (x) L1 f x1  L2 f x2
x  x2 x  x1
L1 , L2
x1  x2 x2  x1
x  x2 x  x1
f1 (x) f x1  f x2
x1  x2 x2  x1
 Lagrange Interpolating Polynomials
(cont…)
In general, the Lagrange polynomial interpolation for n points
is: n
fn1 xi ¦ Li x f xi
i 1
n
x  xj
where Li is given by: Li x –x x
j 1 i j
jzi

x  x1 x  x0
f1 ( x ) f ( x0 )  f ( x1 )
x0  x1 x1  x0
x  x1 x  x2 x  x0 x  x2 x  x0 x  x1
f 2 ( x) f ( x0 )  f ( x1 )  f ( x2 )
x0  x1 x0  x 2 x1  x0 x1  x 2 x2  x0 x2  x1
The cardinals are n th order polynomials :
­0 i z j
Li ( x j ) ®
¯1 i j

f 2 ( x) f ( x0 ) L0 ( x)  f ( x1 ) L1 ( x)  f ( x 2 ) L2 ( x)
x f(x)
x  x1 x  x2 x2 x4 ( x  6 x  8)
2
L0 ( x)
x0  x1 x0  x 2 02 04 8
x  x0 x  x2 x0 x4 x 2  4x
0 1
L1 ( x) 
x1  x0 x1  x 2 20 24 4
x  x0 x  x1 x0 x2 x 2  2x 2 5
L2 ( x )
x 2  x0 x 2  x1 40 42 8
­ ( x 2  6 x  8) ½ ­ x 2  4 x ½ ­ x 2  2 x ½ 4 1
f 2 ( x) 1® ¾  5®  ¾  1® ¾
¯ 8 ¿ ¯ 4 ¿ ¯ 8 ¿
x 2  4x  1
 Exercise: x f(x)

Find a polynomial to interpolate the 0 1

data using Newton divided difference
method and Lagrange method. 2 5

4 1

6 6
 Inverse Interpolation
• Sometimes, it is useful to find the value of x for a given f(x) -
this is inverse interpolation.

• Rather than finding an interpolation of x as a function of f(x),

it may be useful to find an equation for f(x) as a function of x
using interpolation and then solve the corresponding roots
problem:
Objective: Find xi for which f(x)=yi

Soln: Get the value of interpolation function f(x) using the given
data set.

Solve for the root: f(x)=yi

Alternatively: x y y x
exchange xi with
0 1 1 0
yi and solve for x
as a function of y 2 5 5 2
4 1 1 4
6 6 6 6
 Spline Interpolation
• There are cases where
polynomials (esp. high order
polynomials) can lead to
erroneous results because of
round off error and overshoot.

• Alternative approach is to apply

lower-order polynomials to
subsets of data points. Such
connecting polynomials are
called spline functions.

EXAMPLE:
spline fits for a set of 4 points:
(a) linear
(b) quadratic
(c) cubic splines

• linear spline Î ÎÎ
is superior to higher-order
interpolating polynomials (parts
a, b, and c) 23
Quadratic Spline example
 Quadratic Splines - Equations
Given (n+1) points, there are 3(n) coefficients to find and hence, need to
determine 3n equations to solve:

1. Each polynomial must pass through the endpoints of the interval that
they are associated with. And these
conditions result in 2n equations:

ai xi2  bi xi  ci f ( xi ) i 1,2, , n
ai xi21  bi xi 1  ci f ( xi 1 ) i 1,2,, n
1. The first derivatives at the interior knots must be equal ((n-1)
equations):
2ai xi  bi 2ai 1 xi  bi 1 i 1,2,, n  1
2. So far, we are one equation short. Since the last equation will be
derived using only 2 points, it must be a line equation with only bn and
cn to be determined (an = 0)