Chapter 5 Curve Fitting
Chapter 5 Curve Fitting
CHAPTER-V
CURVE FITTING
5.1 Introduction
Data is often given for discrete values along a continuum. However estimates of points between these
discrete values may be required. One way to do this is to formulate a function to fit these values
approximately. This application is called curve fitting. There are two general approaches to curve fitting.
The first is to derive a single curve that represents the general trend of the data. One
method of this nature is the least-squares regression. The second approach is interpolation which is a
more precise one. The basic idea is to fit a curve or a series of curves that pass directly through each of
the points.
To determine the values of ao and a1, differentiate (4.1) with respect to each coefficient
Setting the derivatives equal to zero will result in a minimum Sr.The equations can then be
expressed as
∑ yi − ∑ ao − ∑ a1 xi = 0 (4.3)
∑ yi xi − ∑ ao xi − ∑ a1 xi2 = 0
Solving for ao and a1 simultaneously
n ∑ xi y i − ∑ xi ∑ y i
a1 = (4.4)
n∑ xi2 − (∑ xi )
2
_ _
a o = y − a1 x (4.5)
_ _
where y and x are the means of y and x, respectively
Sr
Sy/x = (4.6)
n−2
Sx/y quantifies the spread around the regression line. This concept can be used to quantify the
goodness of fit. This is particularly necessary when comparing several regressions. To do this,
we return to the original data and determine the total sum of squares around the mean of the
dependent variable i.e., y. This quantity is designated St and represents the magnitude of the
residual error associated with the dependent variable prior to the regression. Recall that Sr
characterizes the residual error that remains after the regression.
The difference between the two quantities, St – Sr, quantifies the improvement or error
reduction due to describing the data in terms of a straight line rather than an average value.
The normalized difference yields
St − S r
r2 = (4.7)
St
n∑ xi y i − (∑ xi )(∑ y i )
r= (4.8)
n∑ xi2 − (∑ xi ) 2 n∑ y i2 − (∑ yi ) 2
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 2
Numerical Methods (CENG 2084) Lecture Note
Caution!
Although the correlation coefficient provides a handy measure of goodness of fit, care should
be taken not to take for granted that the value of r close to one necessarily mean '"good fit". It
is always advised to plot the data along with the regression curve because it is possible to
obtain relatively high value of r when the relationship between y and x is not linear.
Linear regression provides a powerful technique for fitting a “best” line to a data. However it
is predicated on the fact that the relationship between the independent and dependent variables
is linear. But usually this is not the case. Visual inspection of the plot of the data will provide
useful information whether linear regression is acceptable. In situations where linear
regression is inadequate other methods such as polynomial regression are appropriate. For
others, transformations can be used to express the data in a form that is compatible with linear
regression.
The followings are examples of functions which can be linearized
i Exponential functions
This function can be linearized by taking the natural logarithm of both sides of the equation
ln y = ln a1 + b1 x ln e
ln y = ln a1 + b1 x
the plot of ln y versus ln x will yield a straight line with a slope of b1 and an intercept of ln a1.
ii Power functions
y = a2 x b2
where a2 and b2 are constant coefficients.
This equation can be linearized by taking its base 10 logarithm to give
log y = b2 log x + log a 2
the plot of logy versus logx will yield a straight line with a slope of b2 and an intercept of log
a2.
the plot of 1/y versus 1/x will be linear, with a slope of b3/a3 and an intercept of 1/a3.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 3
Numerical Methods (CENG 2084) Lecture Note
In their transformed forms, these models are fit using linear regression in order to evaluate the
constant coefficients. Then they can be transformed back to their original state and used for
predictive purposes.
Example Find the standard "least squares line" for the data points
.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 4
Numerical Methods (CENG 2084) Lecture Note
Some engineering data, although exhibiting marked pattern, is poorly represented by straight
line. For these cases a curve would be better suited to fit the data. One of the possible ways of
solving this kind of problems is to fit the data by a polynomial function. This is called
polynomial regression. The least squares method can be extended to fit data to a higher-order
polynomial. Suppose we want to fit a second order polynomial:
y = a o + a1 x + a 2 x 2 + e
S r = ∑ ( y i − a o − a1 xi −a 2 xi2 ) 2
Taking the derivative of the above equation with respect to each unknown coefficients of the
polynomial gives
∂S r
= −2∑ ( y i − a o − a1 xi − a 2 xi2 )
∂a o
∂S r
= −2∑ xi ( y i − a o − a1 xi − a 2 xi2 )
∂a1
∂S r
= −2∑ xi2 ( y i − a o − a1 xi − a 2 xi2 )
∂a o
Setting these equations equal to zero and rearranging to develop the following set of equations
nao + (∑ xi )a1 + (∑ xi2 )a 2 = ∑ yi
(∑ xi )a o + (∑ xi2 )a1 + (∑ xi3 )a 2 = ∑ xi yi
(∑ xi2 )a o + (∑ xi3 )a1 + (∑ xi4 )a 2 = ∑ xi2 y i
Solving for the coefficients of the quadratic regression is equivalent to solving three
simultaneous linear equations. The techniques for solving these problems are discussed in
chapter two.
This discussion can easily be extended to an mth order polynomial as
y = a o + a1 x + a 2 x 2 + ... + a m x m + e
Thus determination of the coefficients of an mth order polynomial is equivalent to solving a
system of m+1 simultaneous linear equations. For this case the standard error is formulated as
Sr
Sy/x =
n − (m + 1)
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 5
Numerical Methods (CENG 2084) Lecture Note
Example Find the standard "least squares parabola" for the data points
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 6
Numerical Methods (CENG 2084) Lecture Note
A useful extension of linear regression is the case where y is a linear function of more than
one variable, say x1 and x2,
y = a o + a1 x1 + a 2 x 2 + e
Such an equation is useful when fitting experimental data where the variable being studied is a
function of two other variables.
In the same manner as the previous cases the best values of the coefficients are determined by
setting the sum of the squares of the residuals to a minimum.
S r = ∑ ( yi − a o − a1 x1i − a 2 x 2i ) 2
Sr
Sy/x =
n − (m − 1)
Multiple linear regression has utility in the derivation of power equations of the general form
Transformation of this form of equation can be achieved by taking the logarithm of the
equation.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 7
Numerical Methods (CENG 2084) Lecture Note
In the preceding discussions we have seen three types of regression: linear, polynomial and
multiple linear. All these belong to the general linear least squares model given by
y = a o z o + a1 z1 + a 2 z 2 + ... + a m z m + e (4.9)
where zo, z1, z2, .....,zm are m+1 different functions. For multiple linear regression
zo = 1, z1 =x1, z2 = x2, ...., zm = xm. For polynomial regression, the z's are simple monomials as
in zo= 1, z1 = x, z2 = x2 , ..., zm = xm.
The terminology linear refers only to the model's dependence on its parameters i.e., the a's.
Equation (4.9) can be expressed in a matrix form as
z o1 z11 .. .. z m1
z z12 .. .. z m 2
o2
. . . . .
. . . . .
z on z1n . . z mn
where m is the number of variables in the model and n is the number of data points. Because
n ≥m+1 , most of the time [z] is not a square matrix.
The column vector {y} contains the observed values of the dependent variable.
{y}T = y1 , y 2 , ..., y n
The column vector {A} contains the unknown coefficients
{A}T= a o , a1 ,..., a m
the column vector {E} contains the residuals
{E}T = e1 , e2 ,..., en
The sum of the squares of the residuals can be defined as
2
n m
S r = ∑ yi − ∑ a j z ji
i =1 j =0
This quantity can be minimized by taking its partial derivative with respect to each of the
coefficients and setting the resulting equation equal to zero. The outcome of this process can
be expressed in matrix form as
[[Z ] [Z ]]{A} = {[Z ] {y}}
T T
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 8
Numerical Methods (CENG 2084) Lecture Note
4.3 Interpolation
In many engineering applications it is required to estimate intermediate values between
precise data points. The most common method used is the polynomial interpolation. For n+1
data points, there is a unique polynomial of order n that passes through all the points.
f(x) = ao + a1 x + a2 x2 + ... +an xn
Polynomial interpolation involves the determination of that unique nth order polynomial that
fits n+1 data points. This formula can then be used to estimate intermediate values. Although
there is only one nth order polynomial that fits n+1 points, there are a variety of mathematical
formats in which this polynomial can be expressed. The most widely used alternatives are the
Newton and Lagrange interpolating polynomials.
4.3.1 Newton's Divided Difference Interpolating Polynomials
The simplest form of interpolation is to connect two data points with a straight line. This
technique is called linear interpolation. Using the concept of similar triangles,
f 1 ( x) − f ( xo ) f ( x1 ) − f ( xo )
=
x − xo x1 − xo
This is the linear interpolation formula. The notation f1(x) indicates that this is a first order
polynomial. The term [f(x1)-f(xo)]/(x1-x0) is the finite difference approximation of the first
derivative. The smaller the interval between the data points, the better the approximation.
Approximation of a curve with a straight line can result in significant errors. A better
approximation can be achieved if some curvature is introduced into the line connecting the
points. If three points are available, a second order polynomial can be plotted through the data
points. This is called a quadratic interpolating polynomial. A convenient form for this purpose
is
f2(x) = bo + b1(x-xo) + b2(x-x0) (x-x1)
bo can be solved for by setting x=x0, in which case bo = f2(xo)
then f2(x) can be evaluated at x=x1 to solve for b1
f ( x1 ) − f ( xo )
b1 =
x1 − xo
f ( x 2 ) − f ( x1 ) f ( x1 ) − f ( xo )
−
x 2 − x1 x1 − xo
evaluating f2(x) at x=x2 gives b2 =
x2 − xo
Note that b1 represents the slope of the line between xo and x1 and b2 is similar to the finite
difference approximation of the second derivative.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 9
Numerical Methods (CENG 2084) Lecture Note
The preceding analysis can be generalized to fit an nth order polynomial to n+1 data points.
The nth order polynomial is
fn(x) = b0 + b1(x-xo) + b2(x-xo)(x-x1)+ .... + bn(x-xo)(x-x1)....(x-xn-1)
The data points can be used to evaluate the coefficients bo, b1, b2, ...., bn.
b0 = f(xo)
b1 = f[x1,xo]
b2 = f[x2,x1,xo]
.
.
bn = f[xn,xn-1, ..., x1,xo]
where the bracketed function evaluations are finite divided differences. For example the first
finite divided difference is given by
f ( xi ) − f ( x j )
f [ xi , x j ] =
xi − x j
The second finite divided difference is expressed by
f [ xi , x j ] − f [ x j , x k ]
f [ xi , x j , x k ] =
xi − x k
Similarly the n-th finite divided difference is
f [ x n , x n −1 ,..., x1 ] − f [ x n −1 , x n− 2 ,..., xo ]
f [ xn , x n −1 ,..., x 1 , xo ] =
x n − xo
These differences can be used to evaluate the coefficients and then can be substituted to yield
the interpolating polynomial
This is called the Newton’s divided difference interpolating polynomial. It does not require
that the data be put in ascending order or the interval between data points to be equal.
Example 1
Form the Newton polynomials of degree n = 1,2, 3, 4, and 5 for the function
over the interval using equally spaced nodes selected from the following list
a)
b)
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 10
Numerical Methods (CENG 2084) Lecture Note
c)
d)
e)
After simplification
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 11
Numerical Methods (CENG 2084) Lecture Note
f ( n +1) (ξ )
Rn = ( xi +1 − xi ) n +1
(n + 1)!
where ξ is somewhere in the interval xi to xi+1. For the nth order interpolating polynomial, an
analogous relationship for error is
f ( n +1) (ξ )
Rn = ( x − xo )( x − x1 )...( x − x n )
(n + 1)!
where ξ is somewhere in the interval containing the unknown and the data. An alternative
formulation which uses a finite difference to approximate the (n+1)th derivative is
Rn = f[x,xn,xn-1, ..., xo](x-xo)(x-x1)...(x-xn)
where f[x,xn,xn-1,...,xo] is the (n+1)th finite divided difference. The above equation can be
solved if an additional point f(xn+1) is available in which case
Rn ≅ f[xn+1,xn,xn-1,....,xo](x-xo)(x-x1)...(x-xn)
The error estimate for the nth order polynomial is equivalent to the difference between the
(n+1)th order and the n-th order prediction.
Rn = fn+1 (x) - fn(x)
The validity of this approach lies on the fact that the series is strongly convergent. For such
situations the (n+1)th order prediction would be closer to the true value than the nth order
prediction.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 12
Numerical Methods (CENG 2084) Lecture Note
Example 2 Investigate the error for the Newton polynomial approximations Example1.
a)
b)
c)
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 13
Numerical Methods (CENG 2084) Lecture Note
d)
e)
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 14
Numerical Methods (CENG 2084) Lecture Note
Thus if an additional point at x=x+1 is available an error estimate can be obtained. But
because the finite differences are not employed as part of the Lagrange operation, this is rarely
done. For cases where the order of the polynomial is not determined prior to interpolation, the
Newton method has advantages because of the insight it provides into the behavior of the
different order formulas.
Higher order polynomials tend to be ill conditioned; i.e. they are highly sensitive to round-off
error. The same order goes for higher order polynomial regression.
Special care should be taken when interpolating polynomials are to be used for extrapolation.
This arises because the point lies outside the base points used of the interpolation. The best
results of interpolation are obtained when the unknown lies near the center of the base points.
Subsequently when the point lies outside the range of the data set, the error in extrapolation
can be very large.
Example Construct three interpolating polynomials of degree n=1 for the function
over [0,1]. Use the following sets of interpolation nodes.
a)
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 15
Numerical Methods (CENG 2084) Lecture Note
b)
c)
Note that the three polynomials of degree n = 1 are different. The error in
approximating f[x] will also be different.
Example 2. Investigate the error for the Lagrange polynomial approximations of degree n =
2, 3, 4, and 5 in Exercise 1.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 16
Numerical Methods (CENG 2084) Lecture Note
i. Linear splines
The simplest connection between two points is a straight line. The first order splines for a
group of ordered data points can be defined as a set of linear functions,
f(x) = f(xo) + mo(x-x0) x0≤ x ≤ x1
f(x) = f(x1) + m1(x-x1) x1≤ x ≤ x2
.
.
.
f(x) = f(xn-1) + mn-1(x-xn-1) xn-1≤ x ≤ xn
where mi is the slope of the straight line connecting the points:
f ( xi +1 ) − f ( xi )
mi =
xi +1 xi
The disadvantage of first order splines is that they are not smooth. The slope changes abruptly
at the knots i.e., the first derivative of the function is discontinuous at these points.
To ensure the mth order derivatives are continuous at the knots, a spline of atleast m+1 order
must be used. Cubic splines ensure that the first and second derivatives are continuous at the
knots. Third and higher derivatives may be discontinuous but cannot be detected visually and
consequently are ignored.
The derivation of cubic splines can be facilitated by understanding the concepts of second
order interpolating polynomial (quadratic spline).
The objective in quadratic splines is to derive a second order polynomial for each interval
between the data points. The polynomial for each interval can be represented generally as
for n+1 data points there are n intervals and consequently 3n unknown coefficients to
evaluate. There fore, 3n equations are required to solve the problem. These equations are
obtained from:
1. The function values of adjacent polynomials must be equal at the interior knots:
ai −1 xi2−1 + bi −1 xi −1 + ci −1 = f ( xi −1 )
ai xi2−1 + bi xi −1 + ci = f ( xi −1 )
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 17
Numerical Methods (CENG 2084) Lecture Note
for i= 2 to n. Because only interior points are used, this provides 2n-2 conditions.
2. The first and last functions must pass through the end points. This gives two additional
equations:
a1 x02 + b1 xo + c1 = f ( xo )
a n x n2 + bn x n + c n = f ( x n )
2ai −1 xi −1 + bi −1 = 2ai xi −1 + bi
f i ( x) = ai x 3 + bi x 2 + ci x + d i
For n+1 data points there are n intervals and therefore 4n unknowns. The required 4n
conditions are obtained from:
1. the function values at the interior knots must be equal (2n-2 conditions)
2. the first and last functions must pass through the end points ( 2 conditions)
3. The third derivatives at the interior knots must be equal (n-1 conditions)
4. the second derivatives at the interior knots must be equal ( n-1 conditions)
5. The second derivatives at the end knots are zero ( 2 conditions)
This equation can be integrated twice to yield an expression for fi(x). By applying the
condition that the function values at the interior knots must be the same the following cubic
equation results:
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 18
Numerical Methods (CENG 2084) Lecture Note
This equation contains only two unknowns-the second derivatives at the end of each interval.
The second derivatives can be evaluated by invoking the condition that the first derivatives at
the knots must be continuous. The above equation can be differentiated to give an expression
for the first derivative. Setting the first derivatives for the (i-1)th and the ith intervals equal , we
have
If this equation is written for all the interior knots, n-1 simultaneous equations result with n-1
unknowns.
__________________________________________________________________________
AAiT ADDIS ABABA INSTITITUTE OF TECHNOLOGY 19