LECTURE 2

Elementary Regression Theory

Regression and Conditional Expectations

Let x and y be a pair of random variables with a well-defined joint proba-
bility density function f (x, y). If x is unknown, then the best predictor of y is
its unconditional expectation which is defined by
Z Z
E(y) = yf (x, y)dxdy
y x
(52) Z
= yf (y)dy.
y

If the value of x is know, then the best predictor is the conditional expectation
of y given x which is defined as
Z
f (x, y)
E(y|x) = y dy
y f (x)
(53) Z
= yf (y|x)dy,
y

where f (y|x) is the conditional probability density function of y given x. The

marginal and the conditional expectations are related to each other by the
following identity:
Z
(54) E(y) = E(y|x)f (x)dx.
x

In some cases, it is reasonable to make the assumption that the conditional

expectation E(y|x) is a linear function of x:

(55) E(y|x) = α + xβ.

15
 D.S.G. POLLOCK: INTRODUCTORY ECONOMETRICS

This function is described as a linear regression equation. The error from

predicting y by its conditional expectation can be denoted by ε = y − E(y|x);
and therefore we have

y = E(y|x) + ε
(56)
= α + xβ + ε.

Our object is to express the parameters α and β as functions of the mo-

ments of the joint probability distribution of x and y. Usually the moments of
the distribution can be estimated in a straightforward way from a set of obser-
vations on x and y. Using the relationship which exits between the parameters
and the theoretical moments, we should be able to find estimates for α and β
corresponding to the estimated moments.
We begin by multiplying equation (55) throughout by f (x), and by inte-
grating with respect to x. This gives the equation

(57) E(y) = α + βE(x),

whence

(58) α = E(y) − βE(x).

Equation (57) shows that the regression line passes through the point E(x, y) =
{E(x), E(y)} which is the expected value of the joint distribution.
By putting (58) into (55), we find that
© ª
(59) E(y|x) = E(y) + β x − E(x) ,

which shows how the conditional expectation of y differs from the unconditional
expectation in proportion to the error of predicting x by taking its expected
value.
Now let us multiply (55) by x and f (x) and then integrate with respect to
x to provide

(60) E(xy) = αE(x) + βE(x2 ).

Multiplying (57) by E(x) gives

© ª2
(61) E(x)E(y) = αE(x) + β E(x) ,

whence, on taking (61) from (60), we get

h © ª2 i
(62) E(xy) − E(x)E(y) = β E(x2 ) − E(x) ,

16
 2: ELEMENTARY REGRESSION

which implies that

E(xy) − E(x)E(y)
β= © ª2
E(x2 ) − E(x)
h© ª© ªi
E x − E(x) y − E(y)
(63) = h© ª2 i
E x − E(x)

C(x, y)
= .
V (x)

Thus we have expressed α and β in terms of the moments E(x), E(y), V (x)
and C(x, y) of the joint distribution of x and y.
It should be recognised that the prediction error ε = y−E(y|x) = y−α−xβ
is uncorrelated with the variable x. This is shown by writing
h© ª i
(64) E y − E(y|x) x = E(yx) − αE(x) − βE(x2 ) = 0,

where the final equality comes from (60). This result is readily intelligible; for,
if the prediction error were correlated with the value of x, then we should not
be using the information of x efficiently in predicting y.

Empirical Regressions
Imagine that we have a sample of T observations on x and y which are
(x1 , y1 ), (x2 , y2 ), . . . , (xT , yT ). Then we can calculate the following empirical or
sample moments:

1X
T
(65) x̄ = xt ,
T t=1

1X
T
(66) ȳ = yt ,
T t=1

1X 1X 1X 2
T T T
(67) Sx2 = (xt − x̄) =
2
(xt − x̄)xt = x − x̄2 ,
T t=1 T t=1 T t=1 t

1X 1X 1X
T T T
(68) Sxy = (xt − x̄)(yt − ȳ) = (xt − x̄)yt = xt yt − x̄ȳ.
T t=1 T t=1 T t=1

It seems reasonable that, in order to estimate α and β, we should replace

the moments in the formulae of (58) and (63) by the corresponding sample

17
 D.S.G. POLLOCK: INTRODUCTORY ECONOMETRICS

moments. Thus the estimates of α and β are

α̂ = ȳ − β̂ x̄,
(69) P
(xt − x̄)(yt − ȳ)
β̂ = P .
(xt − x̄)2

The justification of this estimation procedure, which is know as the method

of moments, is that, in many of the circumstances under which the sample is
liable to be generated, we can expect the sample moments to converge to the
true moments of the bivariate distribution, thereby causing the estimates of
the parameters to converge likewise to their true values.
Often there is insufficient statistical regularity in the processes generating
the variable x to justify our postulating a joint probability density function for
x and y. Sometimes the variable is regulated in pursuit of an economic policy in
such a way that it cannot be regarded as random in any of the senses accepted
by statistical theory. In such cases, we may prefer to derive the estimators of
the parameters α and β by methods which make fewer statistical assumptions
about x.
When x is a nonstochastic variable, the equation

(70) y = α + xβ + ε

is usually regarded as a functional relationship between x and y which is subject

to the effects of a random disturbance term ε. It is commonly assumed that,
in all instances of this relationship, the disturbance has a zero expected value
and a variance which is finite and constant. Thus

(71) E(ε) = 0 and V (ε) = E(ε2 ) = σ 2 .

Also it is assumed that the movements in x are unrelated to those of the

disturbance term.
The principle of least squares suggests that we should estimate α and β
by finding the values which minimise the quantity

X
T
S= (yt − ŷt )2
t=1
(72)
X
T
= (yt − α − xt β)2 .
t=1

This is the sum of squares of the vertical distances—measured parallel to the

y-axis—of the data points from an interpolated regression line.

18
 2: ELEMENTARY REGRESSION

Differentiating the function S with respect to α and setting the results to

zero for a minimum gives
X
−2 (yt − α − βxt ) = 0, or, equivalently,
(73)
ȳ − α − β x̄ = 0.

This generates the following estimating equation for α:

(74) α(β) = ȳ − β x̄.

Next, by differentiating with respect to β and setting the result to zero, we get
X
(75) −2 xt (yt − α − βxt ) = 0.

On substituting for α from (74) and eliminating the factor −2, this becomes
X X X
(76) xt yt − xt (ȳ − β x̄) − β x2t = 0,

whence we get
P
xt yt − T x̄ȳ
β̂ = P 2
xt − T x̄2
(77) P
(xt − x̄)(yt − ȳ)
= P .
(xt − x̄)2

This expression is identical to the one under (69) which we have derived by the
method of moments. By putting β̂ into the estimating equation for α under
(74), we derive the same estimate α̂ for the intercept parameter as the one to
be found under (69).
It is notable that the equation (75) is the empirical analogue of the equation
(64) which expresses the condition that the prediction error is uncorrelated with
the values of x.
The method of least squares does not automatically provide an estimate
of σ = E(ε2t ). To obtain an estimate, we may invoke the method of moments
2

which, in view of the fact that the regression residuals et = yt −α̂−β̂xt represent
estimates of the corresponding values of εt , suggests an estimator in the form
of
1X 2
(78) σ̃ 2 = et .
T
In fact, this is a biased estimator with
¡ ¢ © ª
(79) E T σ̃ 2 = T − 2 σ 2 ;

19
 D.S.G. POLLOCK: INTRODUCTORY ECONOMETRICS

so it is common to adopt the unbiased estimator

P 2
2 et
(80) σ̂ = .
T −2

The Regression Equation with Two Explanatory Variables

In order to facilitate the treatment of the regression model via matrix
algebra, it is useful to recall the algebra of the regression model with two
explanatory variables.
Consider the equation
(81) y = α + x1 β1 + x2 β2 + ε,
and imagine that there are T observations on y, x1 and x2 which are indexed
by t = 1, . . . , T . Compared with the former notation, we are using lower-case
letters rather than capitals to denote the observations.
According to the principle of least squares, the parameters α, β1 and β2
should be estimated by finding the values which minimise the function
X
T
(82) S= (yt − α − xt1 β1 − xt2 β2 )2 .
t=1

The first-order conditions for the minimisation are obtained by differentiating

S = S(α, β1 , β2 ) in respect of its arguments and setting the results to zero.
After some trivial simplifications this leads to
X
(83) 0= (yt − α − xt1 β1 − xt2 β2 ),
t
X
(84) 0= xt1 (yt − α − xt1 β1 − xt2 β2 ),
t
X
(85) 0= xt2 (yt − α − xt1 β1 − xt2 β2 ).
t

On dividing the first of these equations by T are rearranging it, we get the
estimating equation for α:
(86) α(β1 , β2 ) = ȳ − x̄1 β1 − x̄2 β2 ,
−1
P −1
P
where x̄1 = T t xt1 and x̄ 2 = T t xt2 . When this is substituted into
the equations (84) and (85) they become
X n o
(87) 0= xt1 (yt − ȳ) − (xt1 − x̄1 )β1 − (xt2 − x̄2 )β2 ,
t
X n o
(88) 0= xt2 (yt − ȳ) − (xt1 − x̄1 )β1 − (xt2 − x̄2 )β2 .
t

20
 2: ELEMENTARY REGRESSION

We can now avail ourselves of a few definitions:

1X 1X
T T
(89) S11 = (xt1 − x̄1 )2 = (xt1 − x̄1 )xt1 ,
T t=1 T t=1

1X 1X
T T
(90) S22 = (xt2 − x̄2 ) =
2
(xt2 − x̄2 )xt2 ,
T t=1 T t=1

1X 1X
T T
(91) S12 = (xt1 − x̄1 )(xt2 − x̄2 ) = (xt1 − x̄1 )xt2 ,
T t=1 T t=1

1X 1X
T T
(92) S1y = (xt1 − x̄1 )(yt − ȳ) = (xt1 − x̄1 )yt ,
T t=1 T t=1

1X 1X
T T
(93) S2y = (xt2 − x̄2 )(yt − ȳ) = (xt2 − x̄2 )yt .
T t=1 T t=1

In these terms, the pair of equations under (87) and (88) become

(94) S11 β1 + S12 β2 = S1y ,

(95) S21 β1 + S22 β2 = S2y ,

wherein S21 = S12 . Using simple algebraic manipulations, a solution may be

obtained in the form of

S1y − S12 β̂2

(96) β̂1 = ,
S11
S11 S2y − S12 S1y
(97) β̂2 = ,
S11 S22 − S12
2

Alternatively, we may write the equations in a matrix format as

· ¸· ¸ · ¸
S11 S12 β1 S1y
(98) = .
S21 S22 β2 S2y

Using the formula for the inverse of a matrix of order 2 × 2, we get

· ¸ · ¸· ¸
β1 1 S22 −S12 S1y
(99) = .
β2 S11 S22 − S12
2 −S21 S11 S2y

On multiplying the vector and the matrix on the RHS we get

S22 S1y − S12 S2y
(100) β̂1 = .
S11 S22 − S12
2

21
 D.S.G. POLLOCK: INTRODUCTORY ECONOMETRICS

together with the expression for β̂2 of (97). The estimate of α, which comes
from substituting β̂1 and β̂2 into equation (86), is

(101) α̂ = ȳ − x̄1 β̂1 − x̄2 β̂2 .

The Multiple Regression Model in Matrices

Consider the regression equation

(102) y = β0 + β1 x1 + · · · + βk xk + ε,

and imagine that T observations on the variables y, x1 , . . . , xk are available

which are indexed by t = 1, . . . , T . Then we can write the T realisations of the
relationship in the following form:
      
y1 1 x11 ... x1k β0 ε1
 y2   1 x21 ... x2k   β1   ε2 
(103)  . =. .. ..   . + . .
 .   .. . .   .   .. 
. .
yT 1 xT 1 ... xT k βk εT

This can be represented in summary notation by

(104) y = Xβ + ε.

Our object is to derive an expression for the ordinary least-squares es-

timates of the elements of the parameter vector β = [β0 , β1 , . . . , βk ]0 . The
criterion is to minimise a sum of squares of residuals which can be written
variously as

S(β) = ε0 ε
= (y − Xβ)0 (y − Xβ)
(105)
= y 0 y − y 0 Xβ − β 0 X 0 y + β 0 X 0 Xβ
= y 0 y − 2y 0 Xβ + β 0 X 0 Xβ.

Here, to reach the final expression, we have used the identity β 0 X 0 y = y 0 Xβ

which comes from the fact that the transpose of a scalar—which may be con-
strued as a matrix of order 1 × 1—is the scalar itself.
To find the first-order conditions, we differentiate the function with respect
to the vector β and we set the result to zero. According to the rules of matrix
differentiation, which are easily verified, the derivative is

∂S
(106) = −2y 0 X + 2β 0 X 0 X.
∂β

22
 2: ELEMENTARY REGRESSION

Setting this to zero gives 0 = β 0 X 0 X − y 0 X, which is transposed to provide the

so-called normal equations:

(107) X 0 Xβ = X 0 y.

On the assumption that the inverse matrix exists, the equations have a unique
solution which is the vector of ordinary least-squares estimates:

(108) β̂ = (X 0 X)−1 X 0 y.

The Partitioned Regression Model

Consider taking the regression equation of (104) in the form of
· ¸
β1
(109) y = [ X1 X2 ] + ε = X1 β1 + X2 β2 + ε.
β2

Here, [X1 , X2 ] = X and [β10 , β20 ]0 = β are obtained by partitioning the matrix
X and vector β in a conformable manner. The normal equations of (107) can
be partitioned likewise. Writing the equations without the surrounding matrix
braces gives

(110) X10 X1 β1 + X10 X2 β2 = X10 y,

(111) X20 X1 β1 + X20 X2 β2 = X20 y.

From (110), we get the equation X10 X1 β1 = X10 (y − X2 β2 ) which gives an

expression for the leading subvector of β̂ :

(112) β̂1 = (X10 X1 )−1 X10 (y − X2 β̂2 ).

To obtain an expression for β̂2 , we must eliminate β1 from equation (111). For
this purpose, we multiply equation (110) by X20 X1 (X10 X1 )−1 to give

(113) X20 X1 β1 + X20 X1 (X10 X1 )−1 X10 X2 β2 = X20 X1 (X10 X1 )−1 X10 y.

When the latter is taken from equation (111), we get

n o
(114) X20 X2 − X20 X1 (X10 X1 )−1 X10 X2 β2 = X20 y − X20 X1 (X10 X1 )−1 X10 y.

On defining

(115) P1 = X1 (X10 X1 )−1 X10 ,

23
 D.S.G. POLLOCK: INTRODUCTORY ECONOMETRICS

can we rewrite (114) as

n o
(116) X20 (I − P1 )X2 β2 = X20 (I − P1 )y,

whence
n o−1
(117) β̂2 = X20 (I − P1 )X2 X20 (I − P1 )y.

The Matrix Form for Simple Regression

Now consider again the equations

(118) yt = α + xt β + εt , t = 1, . . . , T

which comprise T observations of the simple regression model. To represent

these in a matrix form, we must define the following vectors:

y = [y1 , y2 , . . . , yT ]0 ,
x = [x1 , x2 , . . . , xT ]0 ,
(119)
ε = [ε1 , ε2 , . . . , εT ]0 ,
i = [1, 1, . . . , 1]0 .

Here the vector i = [1, 1, . . . , 1]0 , which consists of T units, is described alter-
natively as the dummy vector or the summation vector.
In terms of the vector notation, the equation of (118) can be written as

(120) y = iα + xβ + ε,

which can be construed as a case of the partitioned regression equation of (109).

By setting X1 = i and X2 = x and by taking β1 = α, β2 = β in equations
(112) and (117), we derive the following expressions for the estimates of the
parameters α, β:

(121) α̂ = (i0 i)−1 i0 (y − xβ̂),

© ª−1 0
β̂ = x0 (I − Pi )x x (I − Pi )y, with
(122) 1 0
Pi = i(i0 i)−1 i0 = ii .
T

24
 2: ELEMENTARY REGRESSION

To understand the effect of the operator Pi in this context, consider the follow-
ing expressions:

X
T
0
iy= yt ,
t=1

1X
(123) T
0 −1 0
(i i) iy= yt = ȳ,
T t=1
Pi y = i(i0 i)−1 i0 y = [ȳ, ȳ, . . . , ȳ]0 .

Here Pi y = [ȳ, ȳ, . . . , ȳ]0 is simply a column vector containing T repetitions of

the sample mean. From the expressions above, it can be be understood that,
if x = [x1 , x2 , . . . xT ]0 is vector of T elements, then

X
T X
T X
T
0
(124) x (I − Pi )x = xt (xt − x̄) = (xt − x̄)xt = (xt − x̄)2 .
t=1 t=1 t=1
P P
The final equality depends upon the fact that (xt − x̄)x̄ = x̄ (xt − x̄) = 0.
On using the results under (123) and (124) in the equations (121) and
(122), we find that

(125) α̂ = ȳ − x̄β̂,

P P
(xt − x̄)yt (xt − x̄)(yt − ȳ)
(126) β̂ = P t
= tP ,
t (xt − x̄)xt t (xt − x̄)
2

which are the formulae to be found under (69).

The Regression Model in Deviation Form

The estimator for β under (126) comprises the deviations of the original
observations x1 , . . . , xT from their sample mean x̄. Also, we are free to replace
the observations y1 , . . . , yT by their deviations from the corresponding sample
mean ȳ. It follows that the estimate of β is precisely the value which would
be obtained by applying the technique of least-squares regression to a meta-
equation

(127) yt − ȳ = (xt − x̄)β + (εt − ε̄),

which lacks an intercept term. The estimate for the intercept term can be
recovered from the equation (125) once the value for β̂ is available.

25
 D.S.G. POLLOCK: INTRODUCTORY ECONOMETRICS

This approach is applicable to equations with any number of explanatory

variables. Consider replacing the equation of (103) by the equation
      
y1 − ȳ x11 − x̄1 ... x1k − x̄k β1 ε1 − ε̄
 y2 − ȳ   x21 − x̄1 ... x2k − x̄k   .   ε2 − ε̄ 
(128)  . = .. ..  .  +  . .
 .   . .   .   .. 
.
yT − ȳ xT 1 − x̄1 ... xT k − x̄k βk εT − ε̄

If we define the matrix X = [xtj − x̄j ] and the vectors y = [yt − ȳ] and
ε = [εt − ε̄], then we can retain the summary notation y = Xβ + ε which now
denotes equation (128) instead of equation (103).
As an example of this device, let us consider the equation

(129) yt = α + xt1 β1 + xt2 β2 + εt , t = 1, . . . , T,

which was displayed, in slightly different notation, in the lecture of November

24th. Compared with the former notation, we are now now setting α = β0 and
we are using lower-case letters rather than capitals to denote the observations.
In the former notation, lower-case letters were used to denote deviations.
The present equation gives rise to the following deviation form:

(130) yt − ȳ = (xt1 − x̄1 )β1 + (xt2 − x̄2 )β2 + (εt − ε̄), t = 1, . . . , T.

Let us define the corresponding vectors:

y = [y1 − ȳ, . . . , yT − ȳ]0 ,

x1 = [x11 − x̄1 , . . . , xT 1 − x̄1 ]0 ,
(131)
x2 = [x12 − x̄2 , . . . , xT 2 − x̄2 ]0 ,
ε = [ε1 − ε̄, . . . , εT − ε̄]0 .

Then the summary notation for the equation (130) is just

(132) y = x1 β1 + x2 β2 + ε,

which is equation (109) with X1 = x1 and X2 = x2 and with β1 , β2 as scalars

rather than vectors. It follows that equations (112) and (117) provide the
appropriate means of estimating the regression parameters.
With P1 = x1 (x01 x1 )−1 x01 , we get

x02 (1 − P1 )x2 = x02 x2 − x02 x1 (x01 x1 )−1 x01 x2

(133) © −1
ª
= T S22 − S21 S11 S12 ,

26
 2: ELEMENTARY REGRESSION

where S21 = S12 , since these are scalars. It follows that

β̂1 = (x01 x1 )−1 x01 (y − x2 β̂2 )

(134) −1
© ª
= S11 S1y − S12 β̂2 ,

and that
© ª−1 0
β̂2 = x02 (1 − P1 )x2 x2 (1 − P1 )y
(135) © −1
ª−1 © −1
ª
= S22 − S21 S11 S12 S2y − S21 S11 S1y .

These are the matrix versions of the formulae which have already appeared
under (96) and (97).

27