Lecture 3

Method of Ordinary Least Squares

Obid Khakimov

Westminster International University in Tashkent

Econometrics, 5ECON012C WIUT 1/1

In the previous lecture..
SRF

Weekly consumption expenditure, $

ûi
ui

PRF

Yi

Ŷi

E(Y |Xi )

Weekly income, $

Econometrics, 5ECON012C WIUT 2/1

The critical question was:

If our primary objective in regression analysis is to estimate the PRF

Yi = β1 + β2 + ui
on the basis of the SRF
Yi = β̂1 + β̂2 Xi + ûi
and granted that the SRF is an approximation of the PRF, can we
devise a rule or a method that will make this approximation as
”close” as possible?

Econometrics, 5ECON012C WIUT 3/1

Ordinary Least Squares (OLS)
This is one of many methods of econometric estimation (MLE, IV,
2SLS, GMM. . . )

Attributed to Carl F. Gauss

Features:
attractive statistical properties
simple
intuitive

Econometrics, 5ECON012C WIUT 4/1

How it works?

To understand this method, we first explain the least squares

principle.
Recall the two-variable PRF:
Y i = β1 + β2 Xi + u i
The PRF is not directly observable. We estimate it from the SRF:
Yi = β̂1 + β̂2 Xi + ûi
= Ŷi + ûi
where Ŷi is the estimated (conditional mean) value of Yi .

Econometrics, 5ECON012C WIUT 5/1

How it works?

But how is the SRF itself determined? First,

ûi = Yi − Ŷi = Yi − β̂1 − β̂2 Xi
Now given n pairs of observations on Y and X, we would like to
determine the SRF in such a manner that it is as close as possible to
the actual Y . To this end, we may adopt the following criterion:
Choose the SRF in such
 a way that the sum of the residuals
P P
ûi = Yi − Ŷi is as small as possible.

Econometrics, 5ECON012C WIUT 6/1

How it works?
But this is not a very good
Y
criterion. If we adopt the
criterion of minimizing ûi . the
residuals û2 and û3 as well as SRF
the residuals û3 and û4 receive û1
û2
the same weight in the sum
û3
(û1 + û2 + û3 + û4 ). A
û4
consequence of this is that it is
quite possible that the
algebraic sum of the hatui is
small (even zero) although the
ûi are widely scattered about X

the SRF.

Econometrics, 5ECON012C WIUT 7/1

How it works?

To see this, let û1 , û2 , û3 , and û4 take the values of +10, -2, +2,
and -10, respectively. The algebraic sum of these residuals is zero
although û1 and û4 are scattered more widely around the SRF than
û2 and û3 .
We can avoid this problem if we adopt the least-squares criterion,
which states that the SRF can be fixed in such a way that
P 2 P 2 P  2
ûi = Yi − Ŷi = Yi − β̂1 − β̂2 Xi
where û2i are the squared residuals.

Econometrics, 5ECON012C WIUT 8/1

How it works?

By squaring ûi , this method Y

gives more weight to

residuals such as û1 and û4
SRF
than the residuals û2 and û3 . û1
û2
A further justification for the û3
least-squares method lies in û4
the fact that the estimators
obtained by it have some
very desirable statistical
properties, as we shall see
shortly. X

Econometrics, 5ECON012C WIUT 9/1

Experimental Determination of the SRF

It is obvious Hypothetical Data

 that 
P 2 Yi Xi Ŷ1i û1i û21i Ŷ2i û2i û22i
ûi = f β̂1 , β̂2
4 1 2.929 1.071 1.147 4 0 0
5 4 7.000 -2.000 4.000 7 -2 4
In this experiment:
7 5 8.357 -1.357 1.841 8 -1 1
Ŷ1i = 1.572 + 1.357Xi
12 6 9.714 2.286 5.226 9 3 9
Ŷ2i = 3.000 + 1.000Xi sum 28 16 0.0 12.214 0 14

Now which sets of β̂ values should we choose? Obviously the β̂’s of the first
experiment are the “best” values. But we can make endless experiments and
P 2
then choosing that set of β̂ values that gives us the least possible value of ûi .
Fortunately, the method of least squares provides us with unique estimates of β1
and β2 without trial-and-error process.

Econometrics, 5ECON012C WIUT 10 / 1

Derivation of Least-Squares Estimates
P 2  
Differentiating ûi = f β̂1 , β̂2 partially respect to β̂1 and β̂2 , we
obtain
P
∂ ûi P  P
= −2 Yi − β̂1 − β̂2 Xi = −2 ûi
∂ β̂1
P
∂ ûi P  P
= −2 Yi − β̂1 − β̂2 Xi Xi = −2 ûi Xi
∂ β̂2
Setting these equations to zero, after algebraic simplification and
manipulation, gives the least-squares estimators
P


Xi − X̄ Yi − Ȳ
β̂1 = Ȳ − β̂2 X̄ β̂2 =
2
Xi − X̄

Econometrics, 5ECON012C WIUT 11 / 1

Numerical Properties of Estimators

Note the following numerical properties of estimators obtained by the

method of OLS:
I The OLS estimators are expressed solely in terms of the
observable (i.e., sample) quantities (i.e., X and Y ). Therefore,
they can be easily computed.
II They are point estimators; that is, given the sample, each
estimator will provide only a single (point, not interval) value of
the relevant population parameter.
III Once the OLS estimates are obtained from the sample data, the
sample regression line can be easily obtained.

Econometrics, 5ECON012C WIUT 12 / 1

Properties of Regression line
1. It passes through the sample means of Y and X.
Y
X̄
SRF

Ȳ

Econometrics, 5ECON012C WIUT 13 / 1

Properties of Regression line

2. The mean value of the estimated Y is equal to the mean value of

the actual Y.

Ŷi = β̂1 + β̂2 Xi

 
= Ȳ − β̂2 X̄ + β̂2 Xi


= Ȳ − β̂2 Xi − X̄
Ȳˆ = Ȳ

Econometrics, 5ECON012C WIUT 14 / 1

Properties of Regression line

3. The mean value of the residuals û¯i is zero.

X 
−2 Yi − β̂1 − β̂2 Xi = 0
X
−2 ûi = 0
X
ûi = 0

Econometrics, 5ECON012C WIUT 15 / 1

Properties of Regression line
4. Regression in deviation form:
(
Yi = β̂1 + β̂2 Xi + û
Ȳ = β̂1 + β̂2 X̄i

Now we subtract both equations above, then

Yi − Ȳ = β̂2 Xi − X̄ + ûi
yi = β̂2 xi + ûi

where
yi = Yi − Ȳi
xi = Xi − X̄i
ŷi = β̂2 xi
Econometrics, 5ECON012C WIUT 16 / 1
Properties of Regression line

P
5. The residuals ûi are uncorrelated with Xi : ûi Xi = 0
X X
ŷi ûi = β̂2 xi ûi
X
= β̂2 xi (yi − β̂2 xi )
X X
= β̂2 xi yi − β̂i2 x2i
X X
= β̂i2 x2i − β̂i2 x2i
=0

Econometrics, 5ECON012C WIUT 17 / 1

OLS Assumpations

In regression analysis our objective is not only to obtain β̂1 and β̂2
but also to draw inferences about the true β1 and β2 .
Look at the PRF (Yi = β1 + β2 Xi + ui ). It shows that Yi depends on
both Xi and ui . The assumptions made about the Xi variable(s) and
the error term are extremely critical to the valid interpretation of the
regression estimates.
The Classical Linear Regression Model has 10 assumptions.

Econometrics, 5ECON012C WIUT 18 / 1

Assumption 1: Linear regression model

The regression model is linear in the parameters

Yi = β1 + β2 Xi + ui

Keep in mind that the regressand Y and the regressor X themselves

may be nonlinear.

Econometrics, 5ECON012C WIUT 19 / 1

Assumption 2: X values are fixed in repeated
sampling

Values taken by the regressor X are considered fixed in repeated

samples. More technically, X is assumed to be non-stochastic.

This means that our regression analysis is conditional regression

analysis, that is, conditional on the given values of the regressor(s) X.

Econometrics, 5ECON012C WIUT 20 / 1

Assumption 3: Zero mean value of disturbance ui

o E(Y |X) - Conditional mean

Given the value of X,
the mean, or expected,
value of the random E(Y |Xi )

disturbance term ui is
+ui
zero. Technically, the
conditional mean value −ui

of ui is zero.

E(ui |Xi ) = 0
X
X1 X2 X3

Econometrics, 5ECON012C WIUT 21 / 1

Assumption 3: Homoscedasticity or equal variance

Given the value of X, the

variance of ui is the same
for all observations. That f (u)
y
is, the conditional
variances of ui are
identical. x1 β1 + β2 Xi

x2

var(ui |Xi ) = E (ui − E (ui |Xi ))2 xi

x

= E u2i |Xi

= σ2

Econometrics, 5ECON012C WIUT 22 / 1

Assumption 3: Homoscedasticity or equal variance

Heteroscedasticity f (u)
y

u2i |Xi 2


var(ui |Xi ) = E ̸= σ
β1 + β2 Xi
x1
or x2
xi
E u2i |Xi = σi2


x

The likelihood is that the Y observations coming from the population with
X = X1 would be closer to the PRF than those coming from populations
corresponding to X = X2 , X = X3 , and so on. In short, not all Y values
corresponding to the various X’s will be equally reliable, reliability being
judged by how closely or distantly the Y values are distributed around
their means.

Econometrics, 5ECON012C WIUT 23 / 1

Assumption 4: No Autocorrelation
Given any two X values, Xi
and Xj (i ̸= j), the correlation Positive Autocorrelation Negative Autocorrelation
uj
between two ui and uj (i ̸= j) uj

is zero. ui ui

The disturbances ui and uj are

uncorrelated, i.e., no serial
No Autocorrelation
correlation. This means that, uj

given Xi , the deviations of

ui
any two Y values from their
mean value do not exhibit
patterns.
cov (ui , uj |Xi , Xj ) = E ([ui − E(ui )] |Xi ) ([uj − E(uj )] |Xj )
= E (ui |Xi ) (uj |Xj )
=0
Econometrics, 5ECON012C WIUT 24 / 1
Assumption 6: Zero covariance between ui and Xi;
or E(uiXi) = 0

cov(ui , Xi ) = E [ui − E(ui )] [Xi − E(Xi )]

= E [ui (Xi − E(Xi ))] since E(ui ) = 0
= E(ui Xi ) − E(Xi )E(ui ) since E(Xi ) − non-stochastic
= E(ui Xi ) since E(ui ) = 0
= 0 by assumption

The PRF assumes that X and u (which may represent the influence of all
the omitted variables) have separate (and additive) influence on Y . But if
X and u are correlated, it is not possible to assess their individual effects
on Y . In other words, it is difficult to isolate the influence of X and u on
Y.

Econometrics, 5ECON012C WIUT 25 / 1

Assumption 7: The # of βs < the # of obs
The number of observations n must be greater than the number of
parameters to be estimated. Alternatively, the number of observations
n must be greater than the number of explanatory variables.

Imagine that we had only the first pair of observations on Y and X.

From this single observation there is no way to estimate the two
unknowns, β1 and β2 . We need at least two pairs of observations to
estimate the two unknowns

Econometrics, 5ECON012C WIUT 26 / 1

Assumption 8: Variability in X values

The X values in a given sample must not all be the same.

Thechnically, var(X) must be a finite positive number.

This assumption too is not so innocuous as it looks. If all the X

values are identical, then Xi = X̄i and the denominator of that
equation will be zero, making it impossible to estimate β2 and
therefore β1 .

Econometrics, 5ECON012C WIUT 27 / 1

Assumption 9: Correct specification
There is no specification bias or
error in the model used in Yi = β1 + β2 Xi + β3 Xi2
empirical analysis. Some
important questions are

Marginal cost of production

Yi = α1 + α2 Xi
(1) What variables should be
included in the model?
(2) What is the functional form
of the model? Is it linear in the
parameters, the variables, or
both?
(3) What are the probabilistic
assumptions made about the Yi ,
the Xi , and the ui entering the
Output
model?

Econometrics, 5ECON012C WIUT 28 / 1

Assumption 10: No perfect multicollinearity

That is, there are no perfect linear relationships among explanatory

variables. We will discuss this assumption in Chapter 7, where we
discuss multiple regression models.

Econometrics, 5ECON012C WIUT 29 / 1

Reference
Gujarati, D.N., and Porter, D.C., (2009), Basic Econometrics, 5th
edition. Chapter 3

Econometrics, 5ECON012C WIUT 30 / 1