Principle of Least Square

The principle of least square is used to fit a curve of the form:

𝑦 = 𝑓(𝑥, 𝑎0 , 𝑎1 , … … … , 𝑎𝑛 ),

where 𝑎𝑖 ′𝑠 are unknown parameters, to a set of n sample observations (𝑥𝑖 , 𝑦𝑖 ), 𝑖 = 1, 2, … … … , 𝑛 from a

bivariate population. It consists in minimizing the sum of squares of residuals:

𝐸 = ∑𝑛𝑖=1[𝑦𝑖 − 𝑌𝑖 ]2 , where 𝑌𝑖 = 𝑓(𝑥𝑖 , 𝑎0 , 𝑎1 , … … … , 𝑎𝑛 ) is the expected value at 𝑥 = 𝑥𝑖 (1)

Subject to variations in 𝑎0 , 𝑎1 , … … … , 𝑎𝑛 .

The normal equations for eliminating 𝑎0 , 𝑎1 , … … … , 𝑎𝑛 are given by:

𝜕𝐸
𝜕𝑎𝑖
= 0; 𝑖 = 1, 2, … … … , 𝑛. (2)

## Fitting of curves is also useful in the study of correlation and regression; and the line of regression can
be considered as fitting of linear curves to the given bivariate distribution.

Fitting of a Parabola:

Fig. Fitting a parabola

Suppose we are given a set of values (𝑥𝑖 , 𝑦𝑖 ), 𝑖 = 1, 2, … … … , 𝑛, where 𝑥 is independent variable and 𝑦 the
dependent variable. Such values can be located as points in the xy-plane.

Suppose such points are closely scattered around a parabola:

𝑌 = 𝑎 + 𝑏𝑋 + 𝑐𝑋 2 (3)

Substituting 𝑥𝑖 for X in (3), we get

𝑌𝑖 = 𝑎 + 𝑏𝑥𝑖 + 𝑐𝑥𝑖 2 ; 𝑖 = 1, 2, … … … , 𝑛 (4)

The quantity 𝑌𝑖 is called the expected value of 𝑦 corresponding to 𝑋 = 𝑥𝑖 and the value 𝑦𝑖 is the observed
value of 𝑦 corresponding to the same value 𝑥𝑖 of X. The two values 𝑌𝑖 and 𝑦𝑖 are , in general, different. The
difference (𝑦𝑖 − 𝑌𝑖 ) is called the residual corresponding to 𝑋 = 𝑥𝑖 . According to the principle of least
squares, we choose the constants 𝑎, 𝑏 and 𝑐 such that the sum of squares of residual is minimum.

Now 𝐸 = ∑𝑛𝑖=1[𝑦𝑖 − 𝑌𝑖 ]2 = ∑𝑛𝑖=1[𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 − 𝑐𝑥𝑖 2 ]2 (5)

The normal equations are

𝜕𝐸 𝜕𝐸 𝜕𝐸
𝜕𝑎
= 0; 𝜕𝑏
= 0 and 𝜕𝑐
=0 (6)

From equations (5) and (6), we get

𝜕𝐸
= −2 ∑𝑛𝑖=1[𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 − 𝑐𝑥𝑖 2 ] = 0,
𝜕𝑎
𝜕𝐸
= −2 ∑𝑛𝑖=1 𝑥𝑖 [𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 − 𝑐𝑥𝑖 2 ] = 0, (7)
𝜕𝑏
𝜕𝐸
= −2 ∑𝑛𝑖=1 𝑥𝑖 2 [𝑦𝑖 − 𝑎 − 𝑏𝑥𝑖 − 𝑐𝑥𝑖 2 ] = 0. }
𝜕𝑐

Hence the normal equations for fitting a parabola 𝑌 = 𝑎 + 𝑏𝑋 + 𝑐𝑋2 are

∑𝑛𝑖=1 𝑦𝑖 = 𝑛𝑎 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 + 𝑐 ∑𝑛𝑖=1 𝑥𝑖 2 ,

∑𝑛𝑖=1 𝑥𝑖 𝑦𝑖 = 𝑎 ∑𝑛𝑖=1 𝑥𝑖 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 2 + 𝑐 ∑𝑛𝑖=1 𝑥𝑖 3 , (8)

∑𝑛𝑖=1 𝑥𝑖 2 𝑦𝑖 = 𝑎 ∑𝑛𝑖=1 𝑥𝑖 2 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 3 + 𝑐 ∑𝑛𝑖=1 𝑥𝑖 4 . }

For the given data, we can calculate ∑ 𝑥 , ∑ 𝑦 , ∑ 𝑥𝑦 , ∑ 𝑥 2 𝑦 , ∑ 𝑥 2 , ∑ 𝑥 3 , ∑ 𝑥 4 , and the constants

𝑎, 𝑏 and 𝑐 can be determined by solving equations (8).

Fitting a Straight Line: By applying the similar process and putting 𝑐 = 0, the normal equations
for fitting a straight line 𝑌 = 𝑎 + 𝑏𝑋 are

∑𝑛𝑖=1 𝑦𝑖 = 𝑛𝑎 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 ,
ቋ (9)
∑𝑛𝑖=1 𝑥𝑖 𝑦𝑖 = 𝑎 ∑𝑛𝑖=1 𝑥𝑖 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 2 .

For the given data, we can calculate ∑ 𝑥 , ∑ 𝑦 , ∑ 𝑥𝑦 , ∑ 𝑥 2 , and the constants 𝑎 and 𝑏 can be
determined by solving equations (9).

Fitting of more general curves:

Curves: 𝑌 = 𝑎𝑋 𝑏 𝒐𝒓 𝑌 = 𝑎𝑏 𝑋 𝒐𝒓 𝑌 = 𝑎𝑒 𝑋 to a set of 𝑛 points (10)

We apply the same process for all the above curves. (Now we take the first curve)
Taking logarithm of each side, we get

log 𝑌 = log 𝑎 + 𝑏 log 𝑋 ⟹ 𝑈 = 𝐴 + 𝑏𝑉, (11)

where 𝑈 = log 𝑌 , 𝐴 = log 𝑎 , 𝑉 = log 𝑋.

This is a linear equation in U and V. Therefore, the normal equations for estimating A and b are:

∑𝑛𝑖=1 𝑉𝑖 = 𝑛𝐴 + 𝑏 ∑𝑛𝑖=1 𝑈𝑖 ,
ቋ (12)
∑𝑛𝑖=1 𝑈𝑖 𝑉𝑖 = 𝐴 ∑𝑛𝑖=1 𝑈𝑖 + 𝑏 ∑𝑛𝑖=1 𝑈𝑖 2 .

Example 1. Fit a straight line to that data

x 0 1 2 3 4

y 1 1.8 3.3 4.5 6.3

Solution: Suppose the required line is 𝑦 = 𝑎 + 𝑏𝑥

The normal equations are

∑𝑛𝑖=1 𝑦𝑖 = 𝑛𝑎 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 ,
ቋ 𝑤𝑖𝑡ℎ 𝑛 = 5
∑𝑛𝑖=1 𝑥𝑖 𝑦𝑖 = 𝑎 ∑𝑛𝑖=1 𝑥𝑖 + 𝑏 ∑𝑛𝑖=1 𝑥𝑖 2 .

x y xy 𝑥2

0 1 0 0

1 1.8 1.8 1

2 3.3 6.6 4

3 4.5 13.5 9

4 6.3 25.2 16

∑𝒏𝒊=𝟏 𝒙𝒊 = 𝟏𝟎 ∑𝑛𝑖=1 𝑦𝑖 = 16.9 ∑𝑛𝑖=1 𝑥𝑖 𝑦𝑖 = 47.1 ∑𝑛𝑖=1 𝑥𝑖 2 = 30

16.9 = 5𝑎 + 10,
The normal equations become ൠ (1)
47.1 = 10𝑎 + 30𝑏.

Solving eqn. (1), we get 𝑎 = 0.72 𝑎𝑛𝑑 𝑏 = 1.33

Hence the equation of the line of best fit is 𝑦 = 0.72 + 1.33𝑥.

Example 2. If 𝑃 is the pull required to lift a load 𝑊 by means of a pulley block, find a linear law
of the form 𝑃 = 𝑚𝑊 + 𝑐 connecting 𝑃 𝑎𝑛𝑑 𝑊, using the following data:

𝑃 12 15 21 25

𝑊 50 70 100 120

where 𝑃 𝑎𝑛𝑑 𝑊 are taken in kg-wt. Compute 𝑃 𝑤ℎ𝑒𝑛 𝑊 = 150 𝑘𝑔. 𝑤𝑡.

∑ 𝑃 = 4𝑐 + 𝑚 ∑ 𝑊 ,
Solution: The normal equations are ൠ (1)
∑ 𝑊𝑃 = 𝑐𝑊 + 𝑚 ∑ 𝑊 2 .

W P WP 𝑊2

50 12 600 2500

70 15 1050 4900

100 21 2100 10000

120 25 3000 14400

4 6.3 25.2 16

∑ 𝑾 = 340 ∑ 𝑃 = 73 ∑ 𝑊𝑃 = 6750 ∑ 𝑊 2 = 31800

Therefore, equations (1) become

365 = 4𝑐 + 340𝑚,
ൠ (2)
6750 = 340𝑐 + 31800𝑚.

Solving eqn. (2), we get 𝑐 = 2.28 𝑎𝑛𝑑 𝑚 = 0.19

Hence the equation of the line of best fit is 𝑃 = 2.28 + 0.19𝑊.

When 𝑊 = 150 𝑘𝑔. 𝑤𝑡, 𝑃 = 2.28 + 0.19 × 150 = 30.78 𝑘𝑔. 𝑤𝑡.

Example 3. Fit a parabola of second degree to the following data:

 𝑥 0 1 2 3 4

𝑦 1 1.8 1.3 2.5 6.3

Solution:

𝒙 𝒚 𝒙𝟐 𝒙𝟑 𝒙𝟒 𝒙𝒚 𝒙𝟐 𝒚

0 1 0 0 0 0 0

1 1.8 1 1 1 1.8 1.8

2 1.3 4 8 16 2.6 5.2

3 2.5 9 27 81 7.5 22.5

4 6.3 16 64 256 25.2 100.8

∑ 𝒙 = 𝟏𝟎 ∑ 𝑦 = 12.9 ∑ 𝑥 2 = 30 ∑ 𝑥 3 = 100 ∑ 𝑥 4 = 354 ∑ 𝑥𝑦 = 37.1 ∑ 𝑥 2 𝑦 = 130.3

Let the required parabola be 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑥 2 .

The normal equations for the fitting of a parabola are:

∑ 𝑦 = 𝑛𝑎 + 𝑏 ∑ 𝑥 + 𝑐 ∑ 𝑥 2 ,

∑ 𝑥𝑦 = 𝑎 ∑ 𝑥 + 𝑏 ∑ 𝑥 2 + 𝑐 ∑ 𝑥 3 , (1)

∑ 𝑥2𝑦 = 𝑎 ∑ 𝑥2 + 𝑏 ∑ 𝑥3 + 𝑐 ∑ 𝑥4 . }

Therefore, the normal equations (1) become

12.9 = 5𝑎 + 10𝑏 + 30𝑐,

37.1 = 10𝑎 + 30𝑏 + 100𝑐 (2)

130.3 = 30𝑎 + 100𝑏 + 354𝑐.}

Solving equations (2), we get

𝑎 = 1.42, 𝑏 = −1.07, 𝑐 = 0.55

Hence the required equation of parabola is 𝑦 = 1.42 − 1.07𝑥 + 0.55𝑥 2 .

Exercises:

1. Show that the best fitting linear function for points (𝑥𝑖 , 𝑦𝑖 ), 𝑖 = 1, 2, … … … , 𝑛, may be expressed
𝑥 𝑦 1
in the form [ ∑ 𝑥𝑖 ∑ 𝑦𝑖 𝑛 ] = 0. Show that the line passes through the mean point (𝑥̅ , 𝑦̅).
∑ 𝑥𝑖2 ∑ 𝑥𝑖 𝑦𝑖 ∑ 𝑥𝑖
2. Fit a straight line to the following data taking 𝑦 as the dependent variable:

𝑥 1 2 3 4 5 6 7 8 9

𝑦 9 8 10 12 11 13 14 16 15

Ans. 𝑦 = 8 − 0.5𝑥.
3. Example 3. Fit a parabola of second degree to the following data:

𝑥 0 1 2 3 2

𝑦 1 5 10 22 38

Ans. 𝑦 = 1.42 + 0.26𝑥 + 2.21𝑥 2 .

4. Fit the exponential curve of the form 𝑦 = 𝑎𝑏 𝑥 to the following data:

𝑥 1 2 3 4 5 6 7 8

𝑦 1 1.2 1.8 2.5 3.6 4.7 6.6 9.1

Ans. 𝑦 = 0.6821(1.38)𝑥 .