0% found this document useful (0 votes)
270 views32 pages

Lecture07 - Least Squares Regression

This document discusses linear regression and least squares regression. It introduces linear regression, polynomial regression, and multiple linear regression. It explains that regression is used to fit curves to noisy experimental data to characterize the trend without passing through every point. The objective is to find the curve that minimizes the sum of the squares of the residuals between the data points and the curve. This is done by taking derivatives of the residual sum with respect to the curve parameters and setting them equal to zero.

Uploaded by

Na2ry
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
270 views32 pages

Lecture07 - Least Squares Regression

This document discusses linear regression and least squares regression. It introduces linear regression, polynomial regression, and multiple linear regression. It explains that regression is used to fit curves to noisy experimental data to characterize the trend without passing through every point. The objective is to find the curve that minimizes the sum of the squares of the residuals between the data points and the curve. This is done by taking derivatives of the residual sum with respect to the curve parameters and setting them equal to zero.

Uploaded by

Na2ry
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 32

MTH2212 – Computational Methods and

Statistics

Curve Fitting and Interpolation

Lecture 7:
Least Squares Regression
Objectives

 Introduction
 Linear regression
 Polynomial regression
 Multiple linear regression
 General linear least squares
 Nonlinear regression

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 2


Curve Fitting

 Experimentation
 Data available at discrete points or times
 Estimates are required at points between the discrete values
 Curves are fit to data in order to estimate the intermediate values

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 3


Curve Fitting

 Two methods - depending on error in data


120

 Interpolation 100

- Precise data 80

Temperature (deg F)
- Force through each data point 60

40

20

0
0 1 2 3 4 5
Time (s)

 Regression 6

5
- Noisy data f(x)
4

- Represent trend of the data 3

0
0 2 4 6 8 10 12 14 16
x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 4


Least Squares Regression

 Experimental Data
 Noisy (contains errors or inaccuracies)
 x values are accurate, y values are not

 Find relationship between x and y = f(x)


 Fit general trend without matching individual points
 Derive a curve that minimizes the discrepancy between the data
points and the curve  Least-squares regression

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 5


Linear Regression: Definition

Noisy Data From 8

Experiment 7

6
x y
2.10 2.90 5

y  a0  a1 x
f(x)

6.22 3.83 4

7.17 5.98
3
10.5 5.71
2
13.7 7.74
1

0
0 2 4 6 8 10 12 14 16
x

Straight line characterizes trend without


passing through particular point
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 6
Linear Regression: criteria for a “best” fit

 How do we measure goodness of fit of the line to the data?


9

y5
8

7
Data points
y3
6
e3
y4
5
f(x)

e2
4 Residual
y1 y2 e = y - (a 0 + a 1x )

Regression Model
2 y = a 0 + a 1x

0
0 2 4 6 8 10 12 14 16
x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 7


Linear Regression: criteria for a “best” fit

 Use the curve that minimizes the residual between the data
points and the line 9

y5

Model: y  a0  a1 x
8

7
Data points
y3
6
x y e3
y4

yi  a0  a1xi
5
2.10 2.90

f(x)
e2
6.22 3.83 4 Residual
y1 y2 e = y - (a 0 + a 1x )

7.17 5.98
ei = yi  a0  a1 xi
3

Regression Model
10.5 5.71 2 y = a 0 + a 1x

13.7 7.74 1

n 2 n 0

S r =  ei =   yi  a0  a1xi  2
0 2 4 6 8 10 12 14 16
x

i=1 i=1
 Find the values of a0 and a1 that minimize Sr
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 8
Linear Regression: Finding a0 and a1

S r  n 2
 Minimize Sr by taking =    yi  a0  a1 xi  
a0 a0  i=1 
derivatives WRT n

a0 and a1,    2
i=1 a0
 First a0
n
  
  2   
i=1  a0 
n
  2 y
i=1
i  a0  a1 xi (  1 )
 Finally  0
n  n
na0    xi  a1   yi
i=1  i=1
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 9
Linear Regression: Finding a0 and a1

S r  n 2
 Minimize Sr by taking = 
 i 0 1 i 
y  a  a x 
a1 a1  i=1
derivatives WRT
a0 and a1 n

   2

 Second a1 i=1 a1

n
 
  2   
i=1  a1 
n
  2 y
i=1
i  a0  a1 xi (  xi )
 Finally  0
n   n 2 n
  xi  a0    xi  a1   xi yi
i=1  i=1  i=1

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 10


Linear Regression: Normal equations

 Set of two simultaneous linear equations with two unknowns


( a0 and a1):

n  n
na0    xi  a1   yi
i=1  i=1

n   n 2 n
  xi  a0    xi  a1   xi yi
i=1  i=1  i=1

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 11


Linear Regression: Solution of normal equations

 The normal equations can be solved simultaneously for:

1 n n 2 1 n n
 yi  xi   xi  xi yi
n n i=1 i=1
a0  i=1 i=1
n 2 1  n 2
 xi    xi 
i=1 n i=1 
n 1 n n
 xi yi   xi  yi
i=1 n i=1 i=1
a1 
n 2 1  n 2
 xi    xi 
i=1 n i=1 

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 12


Example 1

 Fit a straight line to the values in the following table

x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 13


Example 1 - Solution

 The intercept and the slope can be calculated with:


1 n n 2 1 n n
 yi  xi   xi  xi yi
n i=1 i=1 n i=1 i=1
a0 
n 2 1  n 2 i xi yi xi2 xiyi
 xi    xi  1 2.10 2.90 4.41 6.09
i=1 n i=1 
2 6.22 3.83 38.69 23.82
n 1 n n
 xi yi   xi  yi 3 7.17 5.98 51.41 42.88
i=1 n i=1 i=1 4 10.5 5.71 110.25 59.96
a1 
n 2 1  n 2 5 13.7 7.74 187.69 106.04
 xi    xi  Σ 39.69 26.16 392.32 238.74
i=1 n i=1 

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 14


Example 1 - Solution

 The values of the intercept and the slope:


1 1
(26.16)(392.3)  (39.69)( 238.7)
a0  5 5  2.038
1
392.3   39.69 2
5
1
238.7  (39.69)( 26.16)
a1  5  0.4023
1
392.3   39.69 2
5

 The equation of the straight line linear regression


y  2.038  0.4023x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 15


Linear Regression: Quantification of error

 Suppose we have data points (xi, yi) and modeled (or


predicted) points (xi, ŷi) from the model ŷ = f(x).
 Data {yi} have two types of variations;
(i) variation explained by the model and
(ii) variation not explained by the model.
 Residual sum of squares: variation not explained by the
model n
S r   ( yi  yˆ i ) 2
i 1

 Regression sum of squares: variation explained by the


model n
St   ( y  yˆ i ) 2
i 1
St  S r
 The coefficient of determination r2 r2 
St
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 16
Linear Regression: Quantification of error

 For a perfect fit


Sr=0 and r=r2=1, signifying that the line explains 100% of the
variability of the data.
 For r=r2=0, Sr=St, the fit represents no improvement.
y2

y1
x1 x2
Total variation in y = Variation explained by the model + Unexplained variation (error)

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 17


Linear Regression: Another measure of fit

 In addition to r2, r
 Define Sr
S y| x 
n2
= standard error of the estimate
- Represents the distribution of the residuals around the
regression line
- Large Sy|x􀃆large residuals
- Small Sy|x􀃆small residuals

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 18


Example 2

 Compute the total standard deviation, the standard error of


the estimate, and the correlation coefficient for the data in
Example 1.

x y
2.10 2.90
6.22 3.83
7.17 5.98
10.5 5.71
13.7 7.74

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 19


Example 2 - Solution

 The standard deviation is


St 14.4819
Sy    1.9028
n 1 5 1
 The standard error of the estimate is
Sr 1.9643
S y| x    0.8092 i xi yi (yi-̅y)2 (yi-a0-a1xi)2
n2 52 1 2.1 2.9 5.4382 0.0003
 The correlation coefficient r is 2 6.22 3.83 1.9656 0.5045
3 7.17 5.98 0.5595 1.1183
2St  S r 14.4819  1.9643
r    0.8644 4 10.5 5.71 0.2285 0.3049
St 14.4819 5 13.7 7.74 6.2901 0.0363
 Σ 39.69 26.16 14.4819 1.9643
r  0.8644  0.9297

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 20


Linearization of Nonlinear Relationships

 If relationship is an exponential function y  ae bx


To make it linear, take logarithm of both sides
ln (y)  ln (a) + bx
Now it’s a linear relation between ln(y) and x

 If relationship is a power function y  ax b

To make linear, take logarithm of both sides


ln (y)  ln (a) + b ln (x)

Now it’s a linear relation between ln(y) and ln(x)

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 21


Linearization of Nonlinear Relationships

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 22


Polynomial Regression

 Minimize the residual between the data points and the


curve -- least-squares regression

Linear yi  a0  a1xi

Quadratic yi  a0  a1 xi  a2 xi2

Cubic yi  a0  a1 xi  a2 xi2  a3 xi3

General yi  a0  a1 xi  a2 xi2  a3 xi3    am xim

Must find values of a0 , a1, a2, … am

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 23


Polynomial Regression

 Residual

ei = yi  (a0  a1 xi  a2 xi2  a3 xi3    am xim )

 Sum of squared residuals


n 2 n
S r =  ei =  [ y  (a0  a1 x  a2 x 2  a3 x 3    am x m )]2
i=1 i=1

 Minimize by taking derivatives

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 24


Polynomial Regression

 Normal Equations

 n n
2
n
m   n 
 n  xi  xi   xi    y i 
 n i=1 i=1 i=1   a0   ni=1 
n n n
 x  xi
2 3
 xi   xi   a1 
m 1  x y 
 i=1 i i=1 i=1 i=1    i=1 i i 
 n    n 2 
m  2  a2 
n n n
  xi2  xi
3 4
 xi   xi     xi yi 
 i=1 i=1 i=1 i=1     i=1 
       am    
n m n
m 1
n
m 2
n
2m  n m 
  xi  xi  xi   xi    xi yi 
i=1 i=1 i=1 i=1  i=1 

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 25


Example 3

 Fit a third-order polynomial to the data given in the Table


below

x 0 1.0 1.5 2.3 2.5 4.0 5.1 6.0 6.5 7.0 8.1 9.0
y 0.2 0.8 2.5 2.5 3.5 4.3 3.0 5.0 3.5 2.4 1.3 2.0
x 9.3 11.0 11.3 12.1 13.1 14.0 15.5 16.0 17.5 17.8 19.0 20.0
y -0.3 -1.3 -3.0 -4.0 -4.9 -4.0 -5.2 -3.0 -3.5 -1.6 -1.4 -0.1

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 26


Example 3 - Solution

 n n
2
n
3  n 
 n  xi  xi  i
x   i 
y
 n i=1 i=1 i=1   ni=1 
4  0 
n n n a
 x  xi
2 3
 xi  xi  a   x y 
 i=1 i i=1 i=1 i=1   1   i=1 i i   a0    0.3593
n   n 
n n n
5 a2  a   2.3051 
  xi2  xi
3 4
 xi  xi   
2
  xi yi  1
    
i=1 i=1 i=1 i=1   a3  i=1  a2   0.3532
n 3 n
4
n
5
n
6 n 3  a   0.0121 
  xi  xi  xi  i
x   xi yi   3  
i=1 i=1 i=1 i=1  i=1 

 24 229.6 3060.2 46342.8   a0    1.30 


 229.6 3060.2 46342.8 752835.2   a1    316.9 
     
 3060.2 46342.8 752835.2 12780147.7  a 2    6037.2 
46342.8 752835.2 12780147.7 223518116.8  a   9943.36
  3   

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 27


Example 3 - Solution

Regression Equation
y = - 0.359 + 2.305x - 0.353x2 + 0.012x3
6

2
f(x)

-2

-4

-6
0 5 10 15 20 25
x

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 28


Multiple Linear Regression

 y = a0 + a1x1 + a2x2 + e
 Again very similar.
 Minimize e
 Polynomial and
multiple regression fall
within definition of
General Linear Least
Squares.

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 29


General Linear Least Squares

y  a0 z0  a1 z1  a2 z 2    am z m  e
z0 , z1,  , zm are m  1 basis functions
Y    Z  A   E
 Z   matrix of the calculated values of the basis functions
at the measured values of the independent variable
 Y  observed valued of the dependent variable
 A  unknown coefficients
 E  residuals
2 Minimized by taking its partial
n  m  derivative w.r.t. each of the
S r    yi   a j z ji  coefficients and setting the
i 1  j 0  resulting equation equal to zero
Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 30
Nonlinear Regression

 Not all equations can be broken down into General Linear


ax
Least Squares model i.e. y  a0 (1  e 1 )  e
 Solve with nonlinear least squares using iterative methods
like Gauss-Newton method
 Equation could possibly be transformed into linear form
 Caveat: when fitting transformed data you minimize the residuals of the
data you are working with
 May not give you the exact same fit as nonlinear regression on
untransformed data

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 31


Nonlinear Regression

y  a0 (1  e a1 x )  e

Dr. M. Hrairi MTH2212 - Computational Methods and Statistics 32

You might also like