Numerical Methods

Lecture 08
7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 1
 Regression
❑ Regression
– There are various methods of curve fitting for data points of well-defined
functions.
– There are also various methods of curve fitting for experimental data.

❑ Why is it necessary?
– In many applications, it often becomes necessary to establish a mathematical relationship
between experimental values. This relationship may be used for either testing existing
mathematical models or establishing new ones.
– The mathematical equation can also be used to predict or forecast values of the
dependent variable.
– The process of establishing such relationships in the form of a mathematical equation is
known as regression analysis or curve fitting.
– Suppose the values of y for the different values of x are given. If we want to know the
effect of x on y, then we may write a functional relationship
y = f(x)
– The variable y is called the dependent variable and x the independent variable. The
relationship may be either linear or nonlinear.
7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 2
 Regression
❑ Least Squares Regression
– Least square regression technique is used to fit the data under the following
situations:
• Relationship is linear.
• Relationship is transcendental.
• Relationship is polynomial.
• Relationship involves two or more independent variables.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 3

 Regression
❑ Fitting Linear Equations
– Fitting a straight line is the simplest approach of regression analysis.
– Lets us consider the mathematical equation for a straight line to describe the
data.
y = a + bx = f(x)
a is the intercept of the line and b its slope.
– Consider a point (xi, yi) as shown in Fig. The vertical distance of this point from
the line f(x) = a + bx is the error qi. Then,
qi = yi – f(xi)
= yi – a - bxi

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 4

 Regression
❑ Fitting Linear Equations
– There are various approaches that could be tried for fitting a ‘’best ’’ line
through the data. They include:
• Minimize the sum of errors,
 qi =  ( yi − a − bxi )
• Minimize the sum of absolute values of errors
 qi =  ( yi − a − bxi )
• Minimize the sum of squares of errors
q =  ( y i − a − bxi )
2 2
i

– First two strategies do not yield a unique line for a given set of data.
– The third strategy overcomes this problem and guarantees a unique line.
– The technique of minimizing the sum of squares of errors is known as least
squares regression.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 5

 Regression
❑ Least Squares Regression
– Let us consider the mathematical equation for a straight line to describe the
data.
y = a + bx = f(x)
a is the intercept of the line and b its slope.
– Consider a point (xi, yi) as shown in Fig. The vertical distance of this point from
the line f(x) = a + bx is the error qi. Then,
qi = yi – f(xi)
Line f(x) = a + bx
= yi – a - bxi
Error, qi
Point (xi, yi)

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 6

 Regression
❑ Least Squares Regression
– Let the sum of squares of individual errors be expressed as
n n
Q =  q =   y i − f ( xi )
2 2
i
i =1 i =1

n
=  ( y i − a − bxi )
2

i =1

– Now choose a and b such that Q is minimum. Since Q depends on a and b, a

necessary condition for Q to be minimum is

Q Q
=0 and =0
a b

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 7

 Regression
❑ Least Squares Regression
– Then, Q = −2 n ( y − a − bx ) = 0
 i
a i
i =1

  y i − na − b xi = 0

 a=
y i
−
b xi
= y − bx
n n
Q n

and = −2 xi ( y i − a − bxi ) = 0

b i =1

  xi y i − a  xi − b xi2 = 0
  y i b xi 
  xi y i −  −  xi − b xi2 = 0
 n n 

 n xi y i − ( y i − b xi ) xi − nb xi2 = 0
 
 b n xi2 − ( xi ) = n xi y i −  xi  y i
2

n xi y i −  xi  y i
b =
n xi2 − ( xi )
2

Where x and y are the averages of x values and y values respectively.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 8
 Regression
❑ Least Squares Regression: Algorithm

1. Read data values.

2. Compute sum of powers and products

 i  i  i ,  xi yi
x , y , x 2

3. Check whether the denominator of the equation for b is zero.

4. Compute b and a.
5. Print out the equation.
6. Interpolate data, if required.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 9

 Regression
❑ Example: Fit a straight line to the following set of data:
x 1 2 3 4 5
y 3 4 5 6 8

Solution:
The various summations are given as follows:

xi yi xi2 xiyi
1 3 1 3
2 4 4 8
3 5 9 15
4 6 16 24
5 8 25 40
∑ 15 26 55 90

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 10

 Regression
❑ Example: Fit a straight line to the following set of data:
x 1 2 3 4 5
y 3 4 5 6 8

Solution:
Calculation for b and a: 8
7
5  90 − 15  26 6
b= = 1.20 y = 1.6 + 1.2x
5  55 − 15 2 y 5
4
26 15
a= − 1.20  = 1.60
3
2
5 6
1

Therefore, the linear equation is 1 2 3 4 5 6 x

y = 1.6 + 1.2 x Fig.: Plot of the data and regression line.

The regression line along with the data is shown in Fig. above.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 11

 Regression
❑ FITTING TRANSCENDENTAL EQUATIONS
– The relationship between the dependent and independent variables is not
always linear. See the following Fig.

x
Fig. Data would fit a nonlinear curve better than a linear one.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 12

 Regression
❑ FITTING TRANSCENDENTAL EQUATIONS
– The nonlinear relationship between them may exist in the form of
transcendental equations. For example, the familiar equation for population
growth is given by
P = p0ekt
Where p0 is the initial population, k is the rate of growth and t is time.
– Another example of nonlinear model is the gas low relating to the pressure and
volume, as given by
p=a
– This type of problems can be solved by using the method of least squares.
– Let us rewrite the equation using the conventional variable x and y as:
y = axb
– Take logarithm on both the sides:
ln y = ln a + b ln x
– This equation is similar in form to the linear equation (Least Square
Regression) and therefore, using the same procedure we can evaluate the
7/17/2023 parameters a and b : Md. Golam Moazzam, Dept. of CSE, JU 13
 Regression
❑ FITTING TRANSCENDENTAL EQUATIONS

n  ln xi ln yi −  ln xi  ln yi
b= y = a + bx
n  (ln xi ) − ( ln xi )
2 2

n  xi yi −  xi  yi
b=
n  xi2 − ( xi )
2

ln a = R =
1
( ln yi − b ln xi )
y b xi
n
a= i
− = y − bx
n n
So, a = eR

ln y = ln a + b ln x
n  ln xi ln yi −  ln xi  ln yi
b=
n  (ln xi ) − ( ln xi )
2 2

ln a = R =
1
( ln yi − b ln xi )
n

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 14

 Regression
❑ Example: For the following data table, fit a power-function model of the form,
y = axb
x 1 2 3 4 5
y 0.5 2 4.5 8 12.5
Solution:
xi yi ln xi ln yi (ln xi)2 (ln xi)( ln yi)
1 0.5 0 -0.6931 0 0
2 2 0.6931 0.6931 0.4805 0.4804
3 4.5 1.0986 1.5041 1.2069 1.6524
4 8 1.3863 2.0794 1.9218 2.8827
5 12.5 1.6094 2.5257 2.5903 4.0649
Sum 4.7874 6.1092 6.1995 9.0804

n  ln xi ln yi −  ln xi  ln yi
b= ln a = R =
1
( ln yi − b ln xi )
n  (ln xi ) − ( ln xi )
2 2
n

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 15

 Regression
❑ Example: For the following data table, fit a power-function model of the form,
y = axb
x 1 2 3 4 5
y 0.5 2 4.5 8 12.5
Solution:
(5)( 9.0804) − (4.7874)( 6.1092)
Calculation: b= = 1.9998
(n )( 6.1995) − (4.7874) 2

6.1092 − (1.9998)( 4.7874)

ln a = = −0.6929
5
So, a = 0.5001
Therefore, the power function equation is y = 0.5001 x1.9998

x2
Actual equation is y =
2
The discrepancy is due to round-off errors.

7/17/2023 Md. Golam Moazzam, Dept. of CSE, JU 16