Appendix 2 The Method of Least Squares

The method of least squares, or linear regression, is important in all branches of science, business, and
other areas. The software is quite often built into calculators. The mathematical basis of this method is
involved and goes deep into the theory of statistics. We can cover here only those aspects which you need
to know in order to use it in the form of a program on your calculator or on a computer.

In general terms, the method of least squares enables you to fit a polynomial to a set of data and to
calculate the polynomial’s coefficients. In the case of the polynomial being a line, the coefficients are the
slope and intercept. To see how this works consider the data shown in Figure 1.

Fitted line:
y y = mx + b
( x2 , y2 )
Results:
m±∆m
D2 b ± ∆b

D3

m x1 + b

D1 ( x3 , y3 )

( x1 , y1 ) x

Figure 1. Three data points and their deviations. (Error bars have been omitted for convenience.)

Suppose that we wish to fit the line y = mx + b to the data and find m and b in the process. Consider
the first point (x1 , y1 ). The ordinate y1 differs from the value mx1 + b which is given by the value of the
function at x1 by the amount

D1 = y1 – ( mx1 + b) .

D1 is called the deviation of the point (x1 , y1 ) from the line. Some deviations are positive while others are
negative. In order to work with positive quantities we square the deviations. For the first point this is

D12 = ( y1 – mx1 – b) .
2

The sum of the squares of the deviations of all the points, also called the residual sum of squares, is

C2-1
Appendix 2. The Method of Least Squares
n n
D2 = ∑ Di2 = ∑ (mxi + b – yi ) .
2
…[1]
i=1 i=1

By definition, the line of best fit is the line specified by those values of m and b which minimize D2
—hence the term least squares. These values can be found by taking the partial derivatives of eq[1] with
respect to m and b, setting them to zero, and then solving for m and b. Taking the partial derivative with
respect to m yields
∂D2 n
= ∑ 2 (mxi + b – yi ) xi = 0
∂m i = 1

so that m ∑ x 2i + b∑ xi – ∑ yi xi = 0 .

Taking the partial derivative with respect to b gives

n n

∑ yi = nb + m∑ xi .
i =1 i =1

You should see that the last two equations are simultaneous equations in m and b. Solving them gives

n
 n  n 
n∑ xi yi –  ∑ xi   ∑ yi 
 i = 1   i=1 
m= i=1
2 …[2a]
n
 n 
n∑ xi2 –  ∑ xi 
i =1  i=1 

 n  n   n  n 
–  ∑ xi   ∑ xi yi  +  ∑ x i2  ∑ yi 
 i =1   i =1   i =1   i =1 
and b= 2 . …[2b]
n
 n 
n∑ xi2 –  ∑ xi 
i =1  i =1 

You could evaluate eqs[2] long hand for a typical file of data by substituting points (x1 ,y 1 ), ( x2 ,y 2 ) etc.
one after the other. And this would be a lot of work! Fortunately, your calculator, if equipped with linear
regression software, will evaluate them automatically (if you enter the proper keystrokes of course). For
instructions you will have to consult the manual for your calculator. (Note that on some calculators m and
b are called b and a respectively. The quantity b, the slope of the line, is called the regression coefficient.)

You should also know what the uncertainties in m and b are. Estimates of these uncertainties are
given by the standard deviations of m and b, denoted σ(m) and σ(b). The calculations require the
correlation coefficient R and the standard error of estimate , denoted σy.x. These two quantities will be defined
below. In what follows it will be assumed for simplicity that the x is and the y is each have the same error.

The correlation coefficient is defined as

C2-2
 Appendix 2. The Method of Least Squares

 n
  
n
 n
n ∑ x i yi  –  ∑ xi   ∑ yi 
 i =1   i =1   i=1 
R= . …[3]
 n 2  n  2   n 2  n 2 
n∑ xi –  ∑ xi   n ∑ yi –  ∑ yi  
 i=1  i=1    i = 1  i =1  

R is a measure of how well y depends on x, i.e., on how well y and x are correlated. For instance, if R = ± 1
then the dependance is direct, that is, the data points lay on a straight line of the form y = ± mx + b. If R
= 0 (or very small) then x and y are unrelated (like apples and oranges). These calculations are normally
performed using a program like proFit. You will never have to evaluate eqs[2] or [3] explicitly, either by
hand or on a calculator.

The residual sum of squares, eq[1], is a measure of the goodness of fit. If the data points lay exactly on
a straight line then D2 = 0. The quantity
D2
σ y ,x = ,
n– 2

is called the standard error of estimate and is a measure of the scatter of the y values about the regression
line of y on x. Obviously, the less scatter the better. It can be shown that

σ y ,x = σ y 1– R2 , …[4]

where σy is a standard deviation of the sample of y values. If you feel energetic you can obtain both σy
and R on your calculator, and hence also σy.x.

The standard deviations of m and b (the errors in m and b) are given by the following expressions

nσ y, x
σ(m) = 2
,
n
 n 
n ∑ xi2 –  ∑ x i 
i =1  i =1 

∑x 2
s
i y, x
i=1
and σ(b) = 2
. …[5]
n
  n
n ∑ xi2 –  ∑ x i 
i =1  i =1 

Thus the expressions given by eqs[2], [3] and [5] represent a complete solution to the regression problem.

Few calculators of the present generation will yield eqs[5] directly. If you insist on using a calculator
at all, you’ll have to find σy.x first from eq[4], plug it into eqs[5], and then evaluate the sums of squares.
This would be a most tedious experience for the most dedicated physicist. Fortunately, you will never
have to do this yourself or perform such calculations manually; proFit will do these calculations for you.

C2-3