1/29

EC114 Introduction to Quantitative Economics

15. The Classical Two-Variable Regression Model I
Marcus Chambers
Department of Economics
University of Essex
14/16 February 2012
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
2/29
Outline
1
Introduction
2
Assumptions of the Classical Regression Model
3
Properties of the OLS Estimators
Reference: R. L. Thomas, Using Statistics in Economics,
McGraw-Hill, 2005, sections 12.1 and 12.2.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Introduction 3/29
We have dealt with the problem of estimating the unknown
population parameters, and , in the linear regression
model
Y = + X + ,
where Y is the dependent variable, X is the regressor, and
is a random disturbance that causes Y to deviate from its
expected value, E(Y) = + X.
In order to estimate and we assume we have a sample
of n observations on Y and X, which satisfy
Y
i
= + X
i
+
i
, i = 1, . . . , n,
i.e. the model holds at each point in the sample.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Introduction 4/29
The Ordinary Least Squares (OLS) estimators of and
are
b =

x
i
y
i

x
2
i
, a =

Y b

X,
respectively, where

Y and

X are the sample means of Y
and X respectively, y
i
= Y
i

Y and x
i
= X
i

X.
The tted model is
Y
i
= a + bX
i
+ e
i
, i = 1, . . . , n,
where e
i
denotes the residual i.e. the deviation of Y
i
from
the tted value

Y
i
= a + bX
i
.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Introduction 5/29
We measure the goodness-of-t by the coefcient of
determination, 0 R
2
1, where
R
2
=
b
2

x
2
i

y
2
i
= 1
SSR
SST
,
SSR =

e
2
i
and SST =

y
2
i
.
A low R
2
indicates a poor t, while a high R
2
indicates a
good t.
But can we say anything more about our sample
regression? For example:
What are the properties of the estimators a and b?
Are they good estimators of and ?
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Introduction 6/29
We have seen (Lecture 14) that two useful properties of an
estimator, Q, of a parameter, , are:
unbiasedness i.e. E(Q) = ; and
efciency i.e. no other estimator of has smaller
variance.
In addition, a linear estimator is said to be BLUE (Best
Linear Unbiased Estimator) if it is:
linear (L);
unbiased (U); and
no other linear unbiased estimator (LUE) has smaller
variance, so it is best (B).
Do the OLS estimators, a and b, of and , have such
properties?
If we wish to answer this question, we need to make some
more assumptions about X and . . .
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 7/29
The Classical Linear Regression Model (CLRM) consists of
a regression equation and a set of assumptions concerning
the properties of the regressor X and the disturbance .
With two variables, Y and X, the regression equation is
Y
i
= + X
i
+
i
, i = 1, . . . , n,
where n denotes the number of observations (sample size).
We shall focus on nite sample (or small sample)
properties and will not be concerned with large sample (or
asymptotic) properties that hold when n gets bigger and
bigger (i.e. n ).
Hence n is nite (n < ) but needs to be greater than 2
(the number of unknown parameters).
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 8/29
The rst set of assumptions concern the explanatory
variable, X:
Assumptions concerning the explanatory variable X
IA (non-random X): X is non-stochastic (non-random);
IB (xed X): The values of X are xed in repeated
samples.
Assumption IA (non-random X) implies that X is not
determined by chance but is entirely non-random.
One way of interpreting IA is that X is chosen by the
investigator.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 9/29
But IA is hard to justify in Economics in which the X values
are not chosen by the researcher in an experiment but are
observed in the real world.
It is, however, a useful starting point for our analysis of the
OLS estimators.
It is interesting to note that the origins of IA lie in the
Classical model having been developed for the physical
sciences in which the researcher can choose the X values
in an experiment and then see what the resulting values for
Y are.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 10/29
Assumption IB (xed X) implies that if we could obtain
more than one sample then the same values of X would be
found in each sample.
Again, this is something that may be possible in
experiments in the physical sciences, but in Economics we
typically only have one sample.
Hence, given the xed, non-random nature of X, the only
source of randomness determining Y is , about which we
shall make the following assumptions:
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 11/29
Assumptions concerning the disturbances
IIA (zero mean): E(
i
) = 0, for all i;
IIB (constant variance): V(
i
) =
2
= constant for all i;
IIC (zero covariance): Cov(
i
,
j
) = 0 for all i = j;
IID (normality): each
i
is normally distributed.
These assumptions govern the properties of the random
part of the model.
Given that X is xed they govern the variation in Y in
repeated samples, as illustrated in the next diagram:
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 12/29

The vertical lines correspond to the xed values of X.
The dots on the lines show the values of Y that occur in
repeated samples these differ because different values of
occur each time due to their randomness.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 13/29
Assumption IIA (zero mean) implies that the average
effect of in repeated samples is zero.
Another way of interpreting this is that, on average, the
expected value of Y is + X i.e.
E(Y
i
) = E( + X
i
+
i
)
= + X
i
+ E(
i
)
= + X
i
, i = 1, . . . , n,
because E(
i
) = 0 under IIA.
Note that E(Y
i
) is not the same for each i but depends on
X
i
which is not constant throughout the sample.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 14/29
Assumption IIB (constant variance) tells us that all
disturbances come from distributions with the same
variance,
2
.
This implies that the variance of Y around E(Y) (= + X)
is the same at all points in the sample.
Note that this Assumption does not say anything about the
form of the distribution of the
i
, just that the variances are
equal across the sample.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 15/29
Assumption IIC (zero covariance) ensures that there is no
systematic tendency for
i
to be related to
j
(j = i).
This is often a strong assumption for time series where
there can be a high degree of correlation from one period
to the next.
Recall that represents the deviation of Y from its average
value E(Y), and can be regarded as the unexpected
component of Y.
For example, a sequence of unexpectedly cold months
might lead to an increased demand for gas for heating
purposes.
This unexpected rise in demand would be reected in a
sequence of consecutive positive shocks (the s) implying
a positive covariance structure, which is ruled out by IIC.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 16/29
Assumption IID (normality) builds on the previous
assumptions by specifying that the distribution of each
i
is
normal.
Combining IIA (zero mean), IIB (constant variance) and IID
(normality) gives

i
N(0,
2
), i = 1, . . . , n.
Note that Y
i
E(Y
i
) = Y
i
X
i
=
i
; this implies that
V(Y
i
) = E (Y
i
E(Y
i
))
2
= E(
2
i
) = V(
i
) =
2
which in turn implies that
Y
i
N( + X
i
,
2
), i = 1, . . . , n.
We can illustrate these ideas in a diagram:
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Assumptions of the Classical Regression Model 17/29

Note that the distribution has the same shape (normal) at
all points but the mean changes with the value of X:
E(Y
i
) = + X
i
, i = 1, . . . , n.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 18/29
Under IA (non-random X) and IB (xed X) the OLS
estimators are linear, meaning that they can be written as
a linear function of the Y
i
.
For b this means that
b =

w
i
Y
i
= w
1
Y
1
+ w
2
Y
2
+ . . . + w
n
Y
n
,
where the w
i
are constants. In fact,
b =

x
i
y
i

x
2
i
=

x
i
(Y
i

Y)

x
2
i
=

x
i
Y
i

x
2
i
because

Y

x
i
= 0
=

w
i
Y
i
where w
i
= x
i
/

x
2
i
.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 19/29
Under IA (non-random X), IB (xed X) and IIA (zero mean),
the OLS estimators are unbiased:
E(a) = and E(b) = .
This means that the means of the sampling distributions of
a and b coincide with the population parameters and ,
respectively.
Put another way, if we had repeated samples then a and b
would, on average, be equal to the population parameters.
In practice, of course, we only have a single sample, but it
is a useful property for the distributions of the estimators to
be centred at the population parameter values.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 20/29
It is also possible to show that the OLS estimators are best
out of all possible linear unbiased estimators i.e. the OLS
estimators are BLUE (best linear unbiased estimators).
For this we need IA, IB, IIA, IIB and IIC.
The proof of this result also known as the Gauss-Markov
Theorem is a bit complicated!
The variances of a and b which are the smallest out of all
LUEs are given by
V(a) =
2
a
=

2

X
2
i
n

x
2
i
and V(b) =
2
b
=

2

x
2
i
.
These are the variances of the sampling distributions of a
and b; their square roots (
a
and
b
) are the standard
errors.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 21/29
If we now add Assumption IID (normality) it can be shown
that the OLS estimators are efcient among all unbiased
estimators (not just linear ones).
The normality assumption, IID, also ensures that a and b
are normally distributed:
a N(,
2
a
), b N(,
2
b
).
These results concerning the distributions of a and b
enable us to conduct hypothesis tests more on this in
Lecture 16.
However, there is one parameter that remains unknown,
which is
2
.
We therefore need to estimate
2
, but how?
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 22/29
Recall that
2
is the variance of each
i
.
Our usual sample variance estimator, for a sample
X
1
, . . . , X
n
, is calculated as the sum of squared deviations
of observations from their mean, divided by n 1:
s
2
X
=

(X
i

X)
2
n 1
.
Problem: we dont observe the
i
so cant compute such an
estimator.
However, we can use the residuals e
i
instead, the obvious
estimator being
s
2
=

(e
i
e)
2
n 1
.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 23/29
But this estimator is biased:
E(s
2
) =

n 2
n 1

2
<
2
.
Instead, we use the unbiased estimator
s
2
=

n 1
n 2

s
2
=

(e
i
e)
2
n 2
.
Note that the denominator involves n 2 and not n 1.
This is because in constructing the e
i
we have had to
estimate two parameters, and , by a and b and have
therefore used up two degrees of freedom.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 24/29
Recall that the OLS normal equations imply that

e
i
= 0
and, hence
e =

e
i
n
=
0
n
= 0.
We can therefore write s
2
as
s
2
=

e
2
i
n 2
.
It turns out that s
2
is an unbiased estimator of
2
i.e. it is
possible to show that
E(s
2
) =
2
.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 25/29
In order to compute s
2
we need to nd

e
2
i
(=SSR).
One way is to calculate e
i
for each i = 1, . . . , n.
There is, however, a simpler method based on quantities
that are calculated for the OLS estimators:

e
2
i
=

y
2
i
b

x
i
y
i
where x
i
= X
i

X and y
i
= Y
i

Y as usual.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 26/29
The estimated variances of a and b are then
s
2
a
=
s
2

X
2
i
n

x
2
i
, s
2
b
=
s
2

x
2
i
.
The estimated standard errors are then given by s
a
and s
b
;
these are the values computed by Stata when running a
linear regression.
For example, the regression output for the cross-section of
30 countries money supply and GDP is as follows:
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 27/29
. regress m g
Source | SS df MS Number of obs = 30
-------------+------------------------------ F( 1, 28) = 94.88
Model | 20.3862321 1 20.3862321 Prob > F = 0.0000
Residual | 6.01600434 28 .214857298 R-squared = 0.7721
-------------+------------------------------ Adj R-squared = 0.7640
Total | 26.4022364 29 .910421946 Root MSE = .46353
------------------------------------------------------------------------------
m | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
g | .1748489 .0179502 9.74 0.000 .1380795 .2116182
_cons | .0212579 .1157594 0.18 0.856 -.2158645 .2583803
------------------------------------------------------------------------------
The numbers in the column headed Std. Err. are calculated
using the formulae on the previous slide.
As we shall see next week they are used in the
construction of t-statistics for hypothesis tests concerning
and .
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Properties of the OLS Estimators 28/29
We can summarise the results for the OLS estimators as
follows:
Property Assumptions
Linearity IA, IB
Unbiasedness IA, IB, IIA
BLUness IA, IB, IIA, IIB, IIC
Efciency IA, IB, IIA, IIB, IIC, IID
Normality IA, IB, IIA, IIB, IIC, IID
Notice that more assumptions are required in order to
obtain stronger results this is a common situation.
We shall now look at how these results for a and b can be
used for making inferences about the population
parameters and .
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I
Summary 29/29
Summary
Assumptions of the Classical Model
Properties of the OLS estimators
Next week:
Inference in the Classical Model.
EC114 Introduction to Quantitative Economics 15. The Classical Two-Variable Regression Model I