Introduction to Econometrics

Econ-2061
Sied Hassen (PhD)

Chapter Two

THE CLASSICAL REGRESSION ANALYSIS

[The Simple Linear Regression Model]

Economic theories are mainly concerned with the relationships among various
economic variables. These relationships, when phrased in mathematical terms, can
predict the effect of one variable on another. The functional relationships of these
variables define the dependence of one variable upon the other variable (s) in the
specific form. The specific functional forms may be linear, quadratic, logarithmic,
exponential, hyperbolic, or any other form.

In this chapter we shall consider a simple linear regression model, i.e. a

relationship between two variables related in a linear form. We shall first discuss
two important forms of relation: stochastic and non-stochastic, among which we
shall be using the former in econometric analysis.

2.1. Stochastic and Non-stochastic Relationships

A relationship between X and Y, characterized as Y = f(X) is said to be

deterministic or non-stochastic if for each value of the independent variable (X)
there is one and only one corresponding value of dependent variable (Y). On the
other hand, a relationship between X and Y is said to be stochastic if for a
particular value of X there is a whole probabilistic distribution of values of Y. In
such a case, for any given value of X, the dependent variable Y assumes some

13
specific value only with some probability. Let’s illustrate the distinction between
stochastic and non stochastic relationships with the help of a supply function.

Assuming that the supply for a certain commodity depends on its price (other
determinants taken to be constant) and the function being linear, the relationship
can be put as:
Q  f ( P)    P                            (2.1)

The above relationship between P and Q is such that for a particular value of P,
there is only one corresponding value of Q. This is, therefore, a deterministic
(non-stochastic) relationship since for each price there is always only one
corresponding quantity supplied. This implies that all the variation in Y is due
solely to changes in X, and that there are no other factors affecting the dependent
variable.

If this were true all the points of price-quantity pairs, if plotted on a two-
dimensional plane, would fall on a straight line. However, if we gather
observations on the quantity actually supplied in the market at various prices and
we plot them on a diagram we see that they do not fall on a straight line.

The deviation of the observation from the line may be attributed to several factors.
a. Omission of variables from the function
b. Random behavior of human beings
c. Imperfect specification of the mathematical form of the model

14
 d. Error of aggregation
e. Error of measurement
In order to take into account the above sources of errors we introduce in
econometric functions a random variable which is usually denoted by the letter ‘u’
or ‘  ’ and is called error term or random disturbance or stochastic term of the
function, so called because u is supposed to ‘disturb’ the exact linear relationship
which is assumed to exist between X and Y. By introducing this random variable
in the function the model is rendered stochastic of the form:
Yi     X  u i ……………………………………………………….(2.2)

Thus a stochastic model is a model in which the dependent variable is not only
determined by the explanatory variable(s) included in the model but also by others
which are not included in the model.
2.2. Simple Linear Regression model.
The above stochastic relationship (2.2) with one explanatory variable is called
simple linear regression model.

The true relationship which connects the variables involved is split into two parts:
a part represented by a line and a part represented by the random term ‘u’.

15
The scatter of observations represents the true relationship between Y and X. The
line represents the exact part of the relationship and the deviation of the
observation from the line represents the random component of the relationship.

- Were it not for the errors in the model, we would observe all the points on the
line Y1' , Y2' ,......, Yn' corresponding to X 1 , X 2 ,...., X n . However because of the random

disturbance, we observe Y1 , Y2 ,......, Yn corresponding to X 1 , X 2 ,...., X n . These

points diverge from the regression line by u1 , u 2 ,...., u n .
Yi     xi  ui
 
  
the dependent var iable the regression line random var iable

- The first component in the bracket is the part of Y explained by the changes
in X and the second is the part of Y not explained by X, that is to say the
change in Y is due to the random influence of u i .

2.2.1 Assumptions of the Classical Linear Stochastic Regression Model.

The classicals made important assumption in their analysis of regression .The most
importat of these assumptions are discussed below.

1. The model is linear in parameters.

The classicals assumed that the model should be linear in the parameters
regardless of whether the explanatory and the dependent variables are linear or
not. This is because if the parameters are non-linear it is difficult for estimation..
Example 1. Y    x  u is linear in both parameters and the variables,
2. ln Y     ln x  u is linear only in the parameters.
Check yourself whether the following models satisfy the above assumption.
a. ln Y 2     ln X 2  U i

b. Yi    X i  U i

16
 2. U i is a random real variable

This means that the value which u may assume in any one period depends on
chance; it may be positive, negative or zero. Every value has a certain probability
of being assumed by u in any particular instance.

2. The mean value of the random variable(U) in any particular period is

zero
This means that for each value of x, the random variable(u) may assume
various values, some greater than zero and some smaller than zero, but if we
considered all the possible and negative values of u, for any given value of X,
they would have on average value equal to zero. In other words the positive
and negative values of u cancel each other.

Mathematically, E (U i )  0 ………………………………..….(2.3)

3. The variance of the random variable(U) is constant in each period (The

assumption of homoscedasticity)

For all values of X, the u’s will show the same dispersion around their mean.
In Fig.2.c this assumption is denoted by the fact that the values that u can
assume lie with in the same limits, irrespective of the value of X. For X 1 , u
can assume any value with in the range AB; for X 2 , u can assume any value
with in the range CD which is equal to AB and so on.
Graphically;

17
Mathematically;

Var (U i )  E[U i  E (U i )]2  E (U i ) 2   2 (Since E (U i )  0 ).This constant variance is

called homoscedasticity assumption and the constant variance itself is called

homoscedastic variance.

4. The random variable (U) has a normal distribution

This means the values of u (for each x) have a bell shaped symmetrical
distribution about their zero mean and constant variance  2 , i.e.
U i  N (0,  2 ) ………………………………………..……2.4

5. The random terms of different observations U i ,U j are independent.

(The assumption of no autocorrelation)

This means the value which the random term assumed in one period does not
depend on the value which it assumed in any other period.

18
 Algebraically,
 
Cov(u i u j )   [(u i  (u i )][u j  (u j )]

 E (u i u j )  0 …………………………..….(2.5)

6. The X i are a set of fixed values in the hypothetical process of repeated

sampling which underlies the linear regression model.
- This means that, in taking large number of samples on Y and X, the X i
values are the same in all samples, but the u i values do differ from sample
to sample, and so of course do the values of y i .

7. The random variable (U) is independent of the explanatory variables.

This means there is no correlation between the random variable and the
explanatory variable. If two variables are unrelated their covariance is
zero.
Hence Cov( X i , U i )  0 ………………………………………..….(2.6)
Proof:-
cov( X , U )  [( X i  ( X i )][U i  (U i )]

 [( X i  ( X i )(U i )] given E (U i )  0

 ( X iU i )  ( X i )(U i )

  ( X iU i )

 X i (U i ) , given that the xi are fixed

0
8. The explanatory variables are measured without error
- U absorbs the influence of omitted variables and possibly errors of
measurement in the y’s. i.e., we will assume that the regressors are error
free, while y values may or may not include errors of measurement.

19
We can now use the above assumptions to derive the following basic concepts.

A. The dependent variable Yi is normally distributed.

i.e  Yi ~ N(  x i ),  2 ………………………………(2.7)

Proof:
Mean: (Y )     xi  u i 
   X i Since (u i )  0

Variance: Var (Yi )  Yi  (Yi ) 2

    X i  u i  (  X i ) 
2

 (u i ) 2

  2 (since (u i ) 2   2 )

 var(Yi )   2 ……………………………………….(2.8)

The shape of the distribution of Yi is determined by the shape of the distribution of

u i which is normal by assumption 4. Since  and  , being constant, they don’t

affect the distribution of y i . Furthermore, the values of the explanatory variable,

xi , are a set of fixed values by assumption 5 and therefore don’t affect the shape of

the distribution of y i .

 Yi ~ N(   x i ,  2 )

B. successive values of the dependent variable are independent, i.e

Cov(Yi , Y j )  0

Proof:
Cov(Yi , Y j )  E{[Yi  E (Yi )][Y j  E (Y j )]}

 E{[  X i  U i  E (  X i  U i )][  X j  U j  E (  X j  U j )}

(Since Yi    X i  U i and Y j     X j  U j )

20
 = E[(  X i  Ui    X i )(  X j  U j    X j )] ,Since (u i )  0

 E (U iU j )  0 (from equation (2.5))

Therefore, Cov(Yi ,Y j )  0 .

2.2.2 Methods of estimation

Specifying the model and stating its underlying assumptions are the first stage of
any econometric application. The next step is the estimation of the numerical
values of the parameters of economic relationships. The parameters of the simple
linear regression model can be estimated by various methods. Three of the most
commonly used methods are:
1. Ordinary least square method (OLS)
2. Maximum likelihood method (MLM)
3. Method of moments (MM).
2.2.2.1 The ordinary least square (OLS) method
The model Yi    X i  U i is called the true relationship between Y and X
because Y and X represent their respective population value, and  and  are
called the true parameters since they are estimated from the population value of Y
and X But it is difficult to obtain the population value of Y and X because of
technical or economic reasons. So we are forced to take the sample value of Y and
X. The parameters estimated from the sample value of Y and X are called the
estimators of the true parameters  and  and are symbolized as ˆ and ˆ .

The model Yi  ˆ  ˆX i  ei , is called estimated relationship between Y and X since

ˆ and ˆ are estimated from the sample of Y and X and ei represents the sample

counterpart of the population random disturbance U i .

21
Estimation of  and  by least square method (OLS) or classical least square

(CLS) involves finding values for the estimates ˆ and ˆ which will minimize the
sum of square of the squared residuals (  ei2 ).

From the estimated relationship Yi  ˆ  ˆX i  ei , we obtain:

ei  Yi  (ˆ  ˆX i ) ……………………………(2.6)

e 2
i   (Yi  ˆ  ˆX i ) 2 ……………………….(2.7)

To find the values of ˆ and ˆ that minimize this sum, we have to partially

differentiate e 2
i with respect to ˆ and ˆ and set the partial derivatives equal to

zero.
  ei2
1.  2 (Yi  ˆ  ˆX i )  0.......................................................(2.8)
ˆ

Rearranging this expression we will get: Y i  n  ˆX i ……(2.9)

If you divide (2.9) by ‘n’ and rearrange, we get

ˆ  Y  ˆX ..........................................................................(2.10)

  ei2
2.  2 X i (Yi  ˆ  ˆX )  0..................................................(2.11)
ˆ

Note: at this point that the term in the parenthesis in equation 2.8and 2.11 is the
residual, e  Yi  ˆ  ˆX i . Hence it is possible to rewrite (2.8) and (2.11) as

 2 ei  0 and  2 X i ei  0 . It follows that;

e i  0 and X e i i  0............................................(2.12)

If we rearrange equation (2.11) we obtain;

Y X i i  ˆX i  ˆX i2 ……………………………………….(2.13)

Equation (2.9) and (2.13) are called the Normal Equations. Substituting the
values of ̂ from (2.10) to (2.13), we get:

22
 Y Xi i  X i (Y  ˆX )  ˆX i2

 Y X i  ˆXX i  ˆX i2

Y Xi i  Y X i  ˆ (X i2  XX i )

XY  nXY = ̂ ( X i2  nX 2)

 XY  n X Y
ˆ  ………………….(2.14)
 X i2  n X 2

Equation (2.14) can be rewritten in somewhat different way as follows;

( X  X )(Y  Y )  ( XY  XY  XY  XY )

 XY  Y X  XY  nXY

 XY  nY X  nXY  nXY

( X  X )(Y  Y )  XY  n X Y              (2.15)

( X  X ) 2  X 2  nX 2                  (2.16)

Substituting (2.15) and (2.16) in (2.14), we get

( X  X )(Y  Y )
ˆ 
( X  X ) 2

Now, denoting ( X i  X ) as xi , and (Yi  Y ) as y i we get;

 xi yi
ˆ  ……………………………………… (2.17)
 x i2

The expression in (2.17) to estimate the parameter coefficient is termed is the

formula in deviation form.

2.2.2.2 Estimation of a function with zero intercept

Suppose it is desired to fit the line Yi    X i  U i , subject to the restriction

  0. To estimate ̂ , the problem is put in a form of restricted minimization

problem and then Lagrange method is applied.

23
 n
We minimize: ei2   (Yi  ˆ  ˆX i ) 2
i 1

Subject to: ˆ  0
The composite function then becomes
Z   (Yi  ˆ  ˆX i ) 2  ˆ , where  is a Lagrange multiplier.

We minimize the function with respect to ˆ , ˆ , and 

Z
 2(Yi  ˆ  ˆX i )    0        (i )
ˆ
Z
 2(Yi  ˆ  ˆX i ) ( X i )  0        (ii )

ˆ

z
 2  0                    (iii )

Substituting (iii) in (ii) and rearranging we obtain:
X i (Yi  ˆX i )  0

Yi X i  ˆX i  0
2

X i Yi
ˆ  ……………………………………..(2.18)
X i2

This formula involves the actual values (observations) of the variables and not
their deviation forms, as in the case of unrestricted value of ̂ .

2.2.2.3. Statistical Properties of Least Square Estimators

There are various econometric methods with which we may obtain the estimates of
the parameters of economic relationships. We would like to an estimate to be as
close as the value of the true population parameters i.e. to vary within only a small
range around the true parameter. How are we to choose among the different
econometric methods, the one that gives ‘good’ estimates? We need some criteria
for judging the ‘goodness’ of an estimate.

24
‘Closeness’ of the estimate to the population parameter is measured by the mean
and variance or standard deviation of the sampling distribution of the estimates of
the different econometric methods. We assume the usual process of repeated
sampling i.e. we assume that we get a very large number of samples each of size
‘n’; we compute the estimates ̂ ’s from each sample, and for each econometric
method and we form their distribution. We next compare the mean (expected
value) and the variances of these distributions and we choose among the
alternative estimates the one whose distribution is concentrated as close as
possible around the population parameter.

PROPERTIES OF OLS ESTIMATORS

The ideal or optimum properties that the OLS estimates possess may be
summarized by well known theorem known as the Gauss-Markov Theorem.
Statement of the theorem: “Given the assumptions of the classical linear regression model, the
OLS estimators, in the class of linear and unbiased estimators, have the minimum variance, i.e. the OLS
estimators are BLUE.

According to the this theorem, under the basic assumptions of the classical linear
regression model, the least squares estimators are linear, unbiased and have
minimum variance (i.e. are best of all linear unbiased estimators). Some times the
theorem referred as the BLUE theorem i.e. Best, Linear, Unbiased Estimator. An
estimator is called BLUE if:
a. Linear: a linear function of the a random variable, such as, the
dependent variable Y.
b. Unbiased: its average or expected value is equal to the true population
parameter.
c. Minimum variance: It has a minimum variance in the class of linear
and unbiased estimators. An unbiased estimator with the least variance
is known as an efficient estimator.

25
According to the Gauss-Markov theorem, the OLS estimators possess all the
BLUE properties. The detailed proof of these properties are presented below
a. Linearity: (for ̂ )

Proposition: ˆ & ˆ are linear in Y.

Proof: From (2.17) of the OLS estimator of ̂ is given by:

xi y i xi (Y  Y ) xi Y  Y xi
ˆ    ,
xi2 xi2 xi2

(but xi   ( X  X )   X  nX  nX  nX  0 )

x Y xi
 ˆ  i 2 ; Now, let  Ki (i  1,2,.....n)
x i xi2

 ˆ  K i Y                          (2.19)

 ˆ  K 1Y1  K 2Y2  K 3Y3       K nYn

 ̂ is linear in Y

Check yourself question:

Show that ̂ is linear in Y? Hint: ̂  1 n  Xk i Yi . Derive this relationship
between ̂ and Y.

b. Unbiasedness:
Proposition: ˆ & ˆ are the unbiased estimators of the true parameters  & 

From your statistics course, you may recall that if ˆ is an estimator of  then
E (ˆ)    the amount of bias and if ˆ is the unbiased estimator of  then bias =0

i.e. E (ˆ)    0  E (ˆ)  

In our case, ˆ & ˆ are estimators of the true parameters  &  .To show that they
are the unbiased estimators of their respective parameters means to prove that:
( ˆ )   and (ˆ )  

26
  Proof (1): Prove that ̂ is unbiased i.e. ( ˆ )   .

We know that ˆ  kYi  k i (   X i  U i )

 k i   k i X i  k i u i ,

but k i  0 and k i X i  1

xi (X  X ) X  nX nX  nX
ki     0
xi2
xi2
 x i2 xi2

  k i  0 …………………………………………………………………(2.20)

xi X i ( X  X ) Xi
k i X i  
xi2 xi2

X 2  XX X 2  nX 2
  1
X 2  nX 2 X 2  nX 2
  k i X i  1............................. ……………………………………………(2.21)

ˆ    k i u i  ˆ    k i u i                          (2.22)

( ˆ )  E (  )  k i E (u i ), Since ki are fixed

( ˆ )   , since (u i )  0

Therefore, ˆ is unbiased estimator of  .

 Proof(2): prove that ̂ is unbiased i.e.: (ˆ )  
From the proof of linearity property under 2.2.2.3 (a), we know that:
̂  1 n  Xk i Yi

 1 n  Xk i    X i  U i  , Since Yi    X i  U i

  1
n X i  1 n u i  Xk i   Xk i X i  Xk i u i

   1 n u i  X k i u i ,  ˆ    1
n u i  X k i u i

   1 n  Xk i )u i ……………………(2.23)

(ˆ )    1 n (u i )  Xk i (u i )

(ˆ )                                (2.24)

27
 ̂ is an unbiased estimator of  .

c. Minimum variance of ˆ and ˆ

Now, we have to establish that out of the class of linear and unbiased estimators
of  and  , ˆ and ˆ possess the smallest sampling variances. For this, we shall

first obtain variance of ˆ and ˆ and then establish that each has the minimum
variance in comparison of the variances of other linear and unbiased estimators
obtained by any other econometric methods than OLS.
a. Variance of ̂

var( )  ( ˆ  ( ˆ )) 2  ( ˆ   ) 2 ……………………………………(2.25)

Substitute (2.22) in (2.25) and we get

var(ˆ )  E ( k i u i ) 2

 [k12 u12  k 22 u 22  ............  k n2 u n2  2k1 k 2 u1u 2  .......  2k n 1 k n u n 1u n ]

 [k12 u12  k 22 u 22  ............  k n2 u n2 ]  [2k1 k 2 u1u 2  .......  2k n 1 k n u n 1u n ]

 ( k i2 u i2 )  (k i k j u i u j ) i j

 k i2 (u i2 )  2k i k j (u i u j )   2 k i2 (Since (ui u j ) =0)

x i xi2 1
k i  , and therefore,  k 2
  2
x i ( x i ) x i
2 i 2 2

2
 var(ˆ )   2 k i2  ……………………………………………..(2.26)
xi2

b. Variance of ̂

var(ˆ )  (ˆ  ( ) 

2

 ˆ                             (2.27)
2

Substituting equation (2.23) in (2.27), we get


var(ˆ )   1 n  Xk i  u i2
2


28
   1 n  Xk i  (u i ) 2
2

  2  ( 1 n  Xk i ) 2

  2 ( 1 n 2  2 n Xk i  X 2 k i2 )

  2 ( 1 n  2 X n k i  X 2 k i2 ) , Since  k i  0

  2 ( 1 n  X 2 k i2 )

1 X2 xi2 1
 2(  ) , Since  k 2
  2
n  xi ( x i ) x i
2 i 2 2

Again:
1 X 2 xi2  nX 2  X 2 
    
n xi2 nxi2  nxi
2


 X2   X i2 
 var(ˆ )   2  1 n  2    2
  n x 2
 …………………………………………(2.28)

 x i   i 
We have computed the variances OLS estimators. Now, it is time to check
whether these variances of OLS estimators do possess minimum variance property
compared to the variances other estimators of the true  and  , other than

ˆ and ˆ .

To establish that ˆ and ˆ possess minimum variance property, we compare their

variances with that of the variances of some other alternative linear and unbiased
estimators of  and  , say  * and  * . Now, we want to prove that any other
linear and unbiased estimator of the true population parameter obtained from any
other econometric method has larger variance that that OLS estimators.
Lets first show minimum variance of ̂ and then that of ̂ .

1. Minimum variance of ̂
Suppose:  * an alternative linear and unbiased estimator of  and;
Let  *  w i Y i ......................................... ………………………………(2.29)
where , wi  k i ; but: wi  k i  ci

29
  *  wi (   X i  u i ) Since Yi    X i  U i

 wi   wi X i  wi u i

 (  *)  wi   wi X i ,since (u i )  0

Since  * is assumed to be an unbiased estimator, then for  * is to be an unbiased

estimator of  , there must be true that wi  0 and wi X  1 in the above
equation.
But, wi  k i  ci
wi  (k i  ci )  k i  ci

Therefore, ci  0 since k i  wi  0

Again wi X i  (k i  ci ) X i  k i X i  ci X i
Since wi X i  1 and k i X i  1  ci X i  0 .

From these values we can drive ci xi  0, where xi  X i  X

ci xi   ci ( X i  X ) ci X i  Xci

Since ci xi  1 ci  0  ci xi  0

Thus, from the above calculations we can summarize the following results.
wi  0, wi xi  1, ci  0, ci X i  0

To prove whether ̂ has minimum variance or not lets compute var( *) to

compare with var(ˆ ) .

var( *)  var(wi Yi )

 wi var(Yi )
2

 var( *)   2 wi2 since Var (Yi )   2

But, wi 2  (k i  ci ) 2  k i2  2k i ci  ci2

c i x i
 wi2  k i2  ci2 Since k i ci  0
xi2

Therefore, var( *)   2 (k i2  ci2 )   2 k i2   2 ci2

30
 var(  *)  var( ˆ )   2  c i2

Given that ci is an arbitrary constant,  2 ci2 is a positive i.e it is greater than zero.

Thus var( *)  var(ˆ ) . This proves that ̂ possesses minimum variance property.
In the similar way we can prove that the least square estimate of the constant
intercept ( ̂ ) possesses minimum variance.

2. Minimum Variance of ̂
We take a new estimator  * , which we assume to be a linear and unbiased
estimator of function of  . The least square estimator ̂ is given by:
ˆ  ( 1 n  Xk i )Yi

By analogy with that the proof of the minimum variance property of ̂ , let’s use
the weights wi = ci + ki Consequently;
 *  ( 1 n  Xwi )Yi

Since we want  * to be on unbiased estimator of the true  , that is, ( *)   ,

we substitute for Y    xi  u i in  * and find the expected value of  * .

 *  ( 1 n  Xwi )(  X i  u i )

 X ui
 (    Xwi  XX i wi  Xwi u i )
n n n
 *    X   ui / n  Xwi   Xwi X i  Xwi u i

For  * to be an unbiased estimator of the true  , the following must hold.

 ( wi )  0, ( wi X i )  1 and  ( wi u i )  0

i.e., if wi  0, and wi X i  1 . These conditions imply that ci  0 and ci X i  0 .

As in the case of ̂ , we need to compute Var (  * ) to compare with var( ̂ )

var( *)  var( 1 n  Xwi )Yi 

 ( 1 n  Xwi ) 2 var(Yi )

  2 ( 1 n  Xwi ) 2

31
   2 ( 1 n 2  X 2 wi  2 1 n Xwi )
2

  2 ( n n 2  X 2 wi  2 X wi )
2 1
n

var( *)   2 
1
n  X 2 wi
2
 ,Since wi  0

but wi 2  k i2  ci2

 var( *)   2 1 n  X 2 (k i2  ci2 

1 X2 
var( *)   2   2    2 X 2 ci2

 n x i 

 X i2 
  2     2 X 2 ci2
 nxi
2

The first term in the bracket it var(ˆ ) , hence

var( *)  var(ˆ )   2 X 2 ci2

 var( *)  var(ˆ ) , Since  2 X 2 ci2  0

Therefore, we have proved that the least square estimators of linear regression
model are best, linear and unbiased (BLU) estimators.

The variance of the random variable (Ui)

You may observe that the variances of the OLS estimates involve  2 , which is the
population variance of the random disturbance term. But it is difficult to obtain the
population data of the disturbance term because of technical and economic
reasons. Hence it is difficult to compute  2 ; this implies that variances of OLS
estimates are also difficult to compute. But we can compute these variances if we
take the unbiased estimate of  2 which is ˆ 2 computed from the sample value of
the disturbance term ei from the expression:
ei2
ˆ u2  …………………………………..2.30
n2

32
To use ˆ 2 in the expressions for the variances of ˆ and ˆ , we have to prove

e   
2

whether ˆ is the unbiased estimator of  , i.e., E (ˆ )  E

2 2 2 i 2

n2

e
2
To prove this we have to compute i from the expressions of Y,

Yˆ , y, yˆ and ei .

Proof:
Yi  ˆ  ˆX i  ei

Yˆ  ˆ  ˆx

 Y  Yˆ  ei ……………………………………………………………(2.31)

 ei  Yi  Yˆ ……………………………………………………………(2.32)

Summing (2.31) will result the following expression

Yi  y i  ei

Yi  Yˆi sin ce (ei )  0

Dividing both sides the above by ‘n’ will give us

Y Yˆi
  Y  Yˆ                    (2.33)
n n
Putting (2.31) and (2.33) together and subtract
Y  Yˆ  e

Y  Yˆ

 (Y  Y )  (Yˆ  Yˆ )  e

 y i  yˆ i  e ………………………………………………(2.34)

From (2.34):
ei  y i  yˆ i ………………………………………………..(2.35)

Where the y’s are in deviation form.

Now, we have to express y i and yˆ i in other expression as derived below.
From: Yi    X i  U i

33
 Y    X  U

We get, by subtraction
y i  (Yi  Y )   i ( X i  X )  (U i  U )  xi  (U  U )

 y i  x  (U  U ) …………………………………………………….(2.36)

Note that we assumed earlier that , (u )  0 , i.e in taking a very large number
samples we expect U to have a mean value of zero, but in any particular single
sample U is not necessarily zero.
Similarly: From;
Yˆ  ˆ  ˆx

Y  ˆ  ˆx

We get, by subtraction

Yˆ  Yˆ  ˆ ( X  X )

 yˆ  ̂x …………………………………………………………….(2.37)

Substituting (2.36) and (2.37) in (2.35) we get

ei   xi  (u i  u )  ˆxi

 (u i  u )  ( ˆi   ) xi

The summation over the n sample values of the squares of the residuals over the
‘n’ samples yields:
ei2  [(u i  u )  ( ˆ   ) xi ] 2

 [(u i  u ) 2  ( ˆ   ) 2 xi  2(u i  u )( ˆ   ) xi ]
2

 (u i  u ) 2  ( ˆ   ) 2 xi  2[( ˆ   )xi (u i  u )]

2

Taking expected values we have:

(ei2 )  [(u i  u ) 2 ]  [( ˆ   ) 2 xi ]  2[( ˆ   )xi (u i  u )] ……………(2.38)
2

The right hand side terms of (2.38)may be rearranged as follows

a. [(u  u ) 2 ]  (u i2  u u i )

34
  ( u i ) 2 
  u i2  

 n 
1
 (u i2 )  (u ) 2
n
 n 2  1n (u1  u 2  .......  u i ) 2 since (u i2 )   u2

 n 2  1n (u i2  2u i u j )

 n 2  1n ((u i2 )  2u i u j ) i  j

 n 2  1n n u2  n2 (u i u j )

 n u2   u2 ( given (u i u j )  0)

  u2 (n  1) ……………………………………………..(2.39)

b. [( ˆ   ) 2 xi 2 ]  xi2 .( ˆ   ) 2

Given that the X’s are fixed in all samples and we know that
1
( ˆ   ) 2  var(ˆ )   u2 2
x
1
Hence xi2 .( ˆ   ) 2  xi2 .  u2
x 2

 x i2 . ( ˆ   ) 2   u2 ……………………………………………(2.40)

c. -2 [( ˆ   )xi (u i  u )]  2[( ˆ   )(xi u i  u xi )]

= -2 [( ˆ   )(xi u i )] , sin ce  xi  0

But from (2.22) , ( ˆ   )  k i u i and substitute it in the above expression,

we will get:
-2 [( ˆ   )xi (u i  u )  2(k i u i )(xi u i )]

  x i u i  
= -2    ( x i u i ) xi
2  ,since k i 
   x i   x
2
i

 (x u ) 2 
 2  i 2i 
 xi 

35
  xi 2 u i 2  2xi x j u i u j 
 2  
xi
2
 
 x 2 (u i 2 )  2( xi x j )(u i u j ) 
 2  i j
 xi x i
2 2


x 2 (u i )
2

 2 ( given (u i u j )  0)
x i
2

 2(u i2 )  2 2 …………………………………………………….(2.41)

Consequently, Equation (2.38) can be written interms of (2.39), (2.40) and (2.41)
as follows: ei2   n  1 u2   2  2 u2  (n  2) u2 ………………………….(2.42)
From which we get
 e 2 
 i   E (ˆ u2 )   u2 ………………………………………………..(2.43)
n2
ei2
Since ˆ u2 
n2
ei2
Thus, ˆ 2  is unbiased estimate of the true variance of the error term(  2 ).
n2
The conclusion that we can drive from the above proof is that we can substitute
ei2
ˆ 2  for (  2 ) in the variance expression of ˆ and ˆ , since E (ˆ 2 )   2 .
n2

Hence the formula of variance of ˆ and ˆ becomes;

ˆ 2 ei2
Var (  )  2 =
ˆ ……………………………………(2.44)
xi (n  2) xi 2

 X i2   ei  X i
2 2

Var (ˆ )  ˆ 
2
  ……………………………(2.45)
 nxi  n(n  2) xi
2 2

e can be computed as  ei 2   y i  ̂  xi y i .
2 2
Note: i

2.2.2.4. Statistical test of Significance of the OLS Estimators

(First Order tests)

36
After the estimation of the parameters and the determination of the least square
regression line, we need to know how ‘good’ is the fit of this line to the sample
observation of Y and X, that is to say we need to measure the dispersion of
observations around the regression line. This knowledge is essential because the
closer the observation to the line, the better the goodness of fit, i.e. the better is the
explanation of the variations of Y by the changes in the explanatory variables.
We divide the available criteria into three groups: the theoretical a priori criteria,
the statistical criteria, and the econometric criteria. Under this section, our focus
is on statistical criteria (first order tests). The two most commonly used first order
tests in econometric analysis are:
i. The coefficient of determination (the square of the
correlation coefficient i.e. R2). This test is used for judging
the explanatory power of the independent variable(s).
ii. The standard error tests of the estimators. This test is used
for judging the statistical reliability of the estimates of the
regression coefficients.

1. TESTS OF THE ‘GOODNESS OF FIT’ WITH R2

r2 shows the percentage of total variation of the dependent variable that can be
explained by the changes in the explanatory variable(s) included in the model. To
elaborate this let’s draw a horizontal line corresponding to the mean value of the
dependent variable Y . (see figure‘d’ below). By fitting the line Yˆ  ˆ 0  ˆ1 X we
try to obtain the explanation of the variation of the dependent variable Y produced
by the changes of the explanatory variable X.

37
 .Y
Y = e  Y  Yˆ
Y Y = Ŷ Yˆ  ˆ 0  ˆ1 X

= Yˆ  Y
Y.

X
Figure ‘d’. Actual and estimated values of the dependent variable Y.
As can be seen from fig.(d) above, Y  Y represents measures the variation of the
sample observation value of the dependent variable around the mean. However
the variation in Y that can be attributed the influence of X, (i.e. the regression line)
is given by the vertical distance Yˆ  Y . The part of the total variation in Y about
Y that can’t be attributed to X is equal to e  Y  Yˆ which is referred to as the
residual variation.
In summary:
ei  Yi  Yˆ = deviation of the observation Yi from the regression line.

y i  Y  Y = deviation of Y from its mean.

yˆ  Yˆ  Y = deviation of the regressed (predicted) value ( Ŷ ) from the mean.

Now, we may write the observed Y as the sum of the predicted value ( Yˆ ) and the
residual term (ei.).
Yi  Yˆ  ei
 predicted Yi

Observed Yi Re sidual

From equation (2.34) we can have the above equation but in deviation form
y  yˆ  e . By squaring and summing both sides, we obtain the following
expression:
y 2  ( yˆ 2  e) 2

38
 y 2  ( yˆ 2  ei2  2 yei)

 y i  ei2  2yˆ ei
2

But ŷei = e(Yˆ  Y )  e(ˆ  ˆxi  Y )

 ˆei  ˆexi  Yˆei

(but  e i  0 ,  ex i  0 )

  yˆ e  0 ………………………………………………(2.46)

Therefore;
y i2  
 yˆ 2  ei2 ………………………………...(2.47)
 
Total Explained Un exp lained
var iation var iation var ation

OR,
Total sum of Explained sum Re sidual sum
 
square of square of square
     
TSS ESS RSS

i.e
TSS  ESS  RSS ……………………………………….(2.48)
Mathematically; the explained variation as a percentage of the total variation is
explained as:
ESS yˆ 2
 ……………………………………….(2.49)
TSS y 2

From equation (2.37) we have yˆ  ̂x . Squaring and summing both sides give us

yˆ 2  ˆ 2 x 2                        (2.50)

We can substitute (2.50) in (2.49) and obtain:

ˆ 2 x 2
ESS / TSS  …………………………………(2.51)
y 2

 xy  xi x y
2 2

 2  , Since ˆ  i 2 i
 x  y x i
2

39
 xy xy
 ………………………………………(2.52)
x 2 y 2

Comparing (2.52) with the formula of the correlation coefficient:

r = Cov (X,Y) / x2x2 = xy / nx2x2 = xy / ( x 2 y 2 )1/2 ………(2.53)

Squaring (2.53) will result in: r2 = ( xy )2 / ( x 2 y 2 ). ………….(2.54)

Comparing (2.52) and (2.54), we see exactly the expressions. Therefore:

xy xy
ESS/TSS  = r2
x 2 y 2

From (2.48), RSS=TSS-ESS. Hence R2 becomes;

TSS  RSS RSS e 2
R2   1  1  i2 ………………………….…………(2.55)
TSS TSS y

From equation (2.55) we can drive;

RSS  ei2  y i2 (1  R 2 )                            (2.56)

The limit of R2: The value of R2 falls between zero and one. i.e. 0  R 2  1 .

Interpretation of R2
Suppose R 2  0.9 , this means that the regression line gives a good fit to the
observed data since this line explains 90% of the total variation of the Y value
around their mean. The remaining 10% of the total variation in Y is unaccounted
for by the regression line and is attributed to the factors included in the disturbance
variable u i .

40
Check yourself question:
a. Show that 0  R 2  1 .
b. Show that the square of the coefficient of correlation is equal to ESS/TSS.

Exercise:
Suppose rxy  is the correlation coefficient between Y and X and is give by:

xi yi

 x i2  y i2

And let ry2 yˆ  the square of the correlation coefficient between Y and Ŷ , and is

(yyˆ ) 2
given by: ry2 yˆ 
y 2 yˆ 2

Show that: i) ry2 yˆ  R 2 ii) ryy  ryx

2. TESTING THE SIGNIFICANCE OF OLS PARAMETERS

To test the significance of the OLS parameter estimators we need the following:
 Variance of the parameter estimators
 Unbiased estimator of  2
 The assumption of normality of the distribution of error term.
We have already derived that:
ˆ 2
 var(ˆ ) 
x 2
ˆ 2 X 2
 var(ˆ ) 
nx 2
e 2 RSS
 ˆ 2  
n2 n2
For the purpose of estimation of the parameters the assumption of normality is not
used, but we use this assumption to test the significance of the parameter

41
estimators; because the testing methods or procedures are based on the assumption
of the normality assumption of the disturbance term. Hence before we discuss on
the various testing methods it is important to see whether the parameters are
normally distributed or not.

We have already assumed that the error term is normally distributed with mean
zero and variance  2 , i.e. U i ~ N(0, 2 ) . Similarly, we also proved

that Yi ~ N[(  x),  2 ] . Now, we want to show the following:

 2 
1. ˆ ~ N  , 2 
 x 

  2 X 2 
2. ˆ ~ N  , 
 nx 2 

To show whether ˆ and ˆ are normally distributed or not, we need to make use of
one property of normal distribution. “........ any linear function of a normally
distributed variable is itself normally distributed.”
ˆ  k i Yi  k1Y1  k 2 Y2i  ....  k n Yn

ˆ  wi Yi  w1Y1  w2 Y2i  ....  wn Yn

Since ˆ and ˆ are linear in Y, it follows that

 2    2 X 2 
 ~ N  , 2
ˆ  ;  ~ N  ,
ˆ 
 x   nx 2 

The OLS estimates ˆ and ˆ are obtained from a sample of observations on Y and
X. Since sampling errors are inevitable in all estimates, it is necessary to apply
test of significance in order to measure the size of the error and determine the
degree of confidence in order to measure the validity of these estimates. This can
be done by using various tests. The most common ones are:
i) Standard error test ii) Student’s t-test iii) Confidence interval

42
All of these testing procedures reach on the same conclusion. Let us now see these
testing methods one by one.
i) Standard error test
This test helps us decide whether the estimates ˆ and ˆ are significantly different
from zero, i.e. whether the sample from which they have been estimated might
have come from a population whose true parameters are zero.
  0 and / or   0 .
Formally we test the null hypothesis
H 0 :  i  0 against the alternative hypothesis H 1 :  i  0

The standard error test may be outlined as follows.

First: Compute standard error of the parameters.

SE ( ˆ )  var(ˆ )

SE (ˆ )  var(ˆ )

Second: compare the standard errors with the numerical values of ˆ and ˆ .
Decision rule:
 If SE ( ˆi )  1
2 ˆi , accept the null hypothesis and reject the alternative

hypothesis. We conclude that ̂ i is statistically insignificant.

 If SE ( ˆi )  1
2 ˆi , reject the null hypothesis and accept the alternative

hypothesis. We conclude that ̂ i is statistically significant.

The acceptance or rejection of the null hypothesis has definite economic meaning.
Namely, the acceptance of the null hypothesis   0 (the slope parameter is zero)
implies that the explanatory variable to which this estimate relates does not in fact
influence the dependent variable Y and should not be included in the function,
since the conducted test provided evidence that changes in X leave Y unaffected.
In other words acceptance of H0 implies that the relation ship between Y and X is
in fact Y    (0) x   , i.e. there is no relationship between X and Y.

43
Numerical example: Suppose that from a sample of size n=30, we estimate the
following supply function.
Q  120  0.6 p  ei
SE : (1.7) (0.025)

Test the significance of the slope parameter at 5% level of significance using the
standard error test.
SE ( ˆ )  0.025

( ˆ )  0.6

1
2 ˆ  0.3

This implies that SE ( ˆi )  1

2 ˆi . The implication is ̂ is statistically significant at

5% level of significance.
Note: The standard error test is an approximated test (which is approximated from
the z-test and t-test) and implies a two tail test conducted at 5% level of
significance.
ii) Student’s t-test
Like the standard error test, this test is also important to test the significance of the
parameters. From your statistics, any variable X can be transformed into t using
the general formula:
X 
t , with n-1 degree of freedom.
sx

Where  i  value of the population mean

s x  sample estimate of the population standard deviation

( X  X ) 2
sx 
n 1
n  sample size
We can derive the t-value of the OLS estimates

44
 ˆi   
t ˆ  
SE ( ˆ ) 
 with n-k degree of freedom.
ˆ   
tˆ 
SE (ˆ ) 

Where:
SE = is standard error
k = number of parameters in the model.

Since we have two parameters in simple linear regression with intercept different
from zero, our degree of freedom is n-2. Like the standard error test we formally
test the hypothesis: H 0 :  i  0 against the alternative H 1 :  i  0 for the slope
parameter; and H0 :  0 against the alternative H 1 :   0 for the intercept.

To undertake the above test we follow the following steps.

Step 1: Compute t*, which is called the computed value of t, by taking the value of
 in the null hypothesis. In our case   0 , then t* becomes:

ˆ  0 ˆ
t*  
SE ( ˆ ) SE ( ˆ )

Step 2: Choose level of significance. Level of significance is the probability of

making ‘wrong’ decision, i.e. the probability of rejecting the hypothesis when it is
actually true or the probability of committing a type I error. It is customary in
econometric research to choose the 5% or the 1% level of significance. This
means that in making our decision we allow (tolerate) five times out of a hundred
to be ‘wrong’ i.e. reject the hypothesis when it is actually true.
Step 3: Check whether there is one tail test or two tail test. If the inequality sign
in the alternative hypothesis is  , then it implies a two tail test and divide the
chosen level of significance by two; decide the critical rejoin or critical value of t
called tc. But if the inequality sign is either > or < then it indicates one tail test and

45
there is no need to divide the chosen level of significance by two to obtain the
critical value of to from the t-table.
Example:
If we have H 0 :  i  0
against: H1 :  i  0

Then this is a two tail test. If the level of significance is 5%, divide it by two to
obtain critical value of t from the t-table.

Step 4: Obtain critical value of t, called tc at  and n-2 degree of freedom for two
2

tail test.
Step 5: Compare t* (the computed value of t) and tc (critical value of t)
 If t*> tc , reject H0 and accept H1. The conclusion is ̂ is statistically
significant.
 If t*< tc , accept H0 and reject H1. The conclusion is ̂ is statistically
insignificant.
Numerical Example:
Suppose that from a sample size n=20 we estimate the following consumption
function:
C  100  0.70  e
(75.5) (0.21)

The values in the brackets are standard errors. We want to test the null hypothesis:
H 0 :  i  0 against the alternative H 1 :  i  0 using the t-test at 5% level of

significance.
a. the t-value for the test statistic is:
ˆ  0 ˆ 0.70
t*   =  3.3
SE ( ˆ ) SE ( ˆ ) 0.21

46
 b. Since the alternative hypothesis (H1) is stated by inequality sign (  ) ,it is a
two tail test, hence we divide 
2  0.05 2  0.025 to obtain the critical value of

‘t’ at  =0.025 and 18 degree of freedom (df) i.e. (n-2=20-2). From the
2

t-table ‘tc’ at 0.025 level of significance and 18 df is 2.10.

c. Since t*=3.3 and tc=2.1, t*>tc. It implies that ̂ is statistically significant.

iii) Confidence interval

Rejection of the null hypothesis doesn’t mean that our estimate ˆ and ˆ is the
correct estimate of the true population parameter  and  . It simply means that
our estimate comes from a sample drawn from a population whose parameter  is
different from zero.

In order to define how close the estimate to the true parameter, we must construct
confidence interval for the true parameter, in other words we must establish
limiting values around the estimate with in which the true parameter is expected to
lie within a certain “degree of confidence”. In this respect we say that with a
given probability the population parameter will be with in the defined confidence
interval (confidence limits).

We choose a probability in advance and refer to it as confidence level (interval

coefficient). It is customarily in econometrics to choose the 95% confidence level.
This means that in repeated sampling the confidence limits, computed from the
sample, would include the true population parameter in 95% of the cases. In the
other 5% of the cases the population parameter will fall outside the confidence
interval.
In a two-tail test at  level of significance, the probability of obtaining the specific
t-value either –tc or tc is  at n-2 degree of freedom. The probability of obtaining
2

47
 ˆ  
any value of t which is equal to at n-2 degree of freedom is
SE ( ˆ )

1   2   2  i.e. 1   .

i.e. Pr t c  t*  t c   1   …………………………………………(2.57)

ˆ  
but t*  …………………………………………………….(2.58)
SE ( ˆ )

Substitute (2.58) in (2.57) we obtain the following expression.

 ˆ   
Pr  t c   t c   1   ………………………………………..(2.59)
 SE ( ˆ ) 

 
Pr  SE ( ˆ )t c  ˆ    SE ( ˆ )t c  1        by multiplying SE ( ˆ )

Pr  ˆ  SE ( ˆ )t      ˆ  SE ( ˆ )t   1        by subtracting ˆ
c c

Pr  ˆ  SE ( ˆ )    ˆ  SE ( ˆ )t   1        by multiplyin g by  1
c

Pr ˆ  SE ( ˆ )t    ˆ  SE ( ˆ )t   1        int erchanging

c c

The limit within which the true  lies at (1   )% degree of confidence is:

[ ˆ  SE ( ˆ )t c , ˆ  SE ( ˆ )t c ] ; where t c is the critical value of t at 

2 confidence
interval and n-2 degree of freedom.
The test procedure is outlined as follows.
H0 :   0

H1 :   0

Decision rule: If the hypothesized value of  in the null hypothesis is within the

confidence interval, accept H0 and reject H1. The implication is that ̂ is

statistically insignificant; while if the hypothesized value of  in the null

hypothesis is outside the limit, reject H0 and accept H1. This indicates ̂ is
statistically significant.
Numerical Example:

48
Suppose we have estimated the following regression line from a sample of 20
observations.
Y  128.5  2.88 X  e
(38.2) (0.85)

The values in the bracket are standard errors.

a. Construct 95% confidence interval for the slope of parameter
b. Test the significance of the slope parameter using constructed confidence
interval.
Solution:
a. The limit within which the true  lies at 95% confidence interval is:

ˆ  SE ( ˆ )t c

ˆ  2.88

SE ( ˆ )  0.85

t c at 0.025 level of significance and 18 degree of freedom is 2.10.

 ˆ  SE ( ˆ )t c  2.88  2.10(0.85)  2.88  1.79.

The confidence interval is:

(1.09, 4.67)
b. The value of  in the null hypothesis is zero which implies it is out side the
confidence interval. Hence  is statistically significant.

2.2.3 Reporting the Results of Regression Analysis

The results of the regression analysis derived are reported in conventional formats.
It is not sufficient merely to report the estimates of  ’s. In practice we report
regression coefficients together with their standard errors and the value of R2. It
has become customary to present the estimated equations with standard errors
placed in parenthesis below the estimated parameter values. Sometimes, the

49
estimated coefficients, the corresponding standard errors, the p-values, and some
other indicators are presented in tabular form.
These results are supplemented by R2 on ( to the right side of the regression
equation).
Y  128 . 5  2 . 88 X
Example: , R2 = 0.93. The numbers in the
( 38 . 2 ) ( 0 . 85 )

parenthesis below the parameter estimates are the standard errors. Some
econometricians report the t-values of the estimated coefficients in place of the
standard errors.
Review Questions

Review Questions
1. Econometrics deals with the measurement of economic relationships which are stochastic
or random. The simplest form of economic relationships between two variables X and Y
can be represented by:
Yi   0  1 X i  U i ; where  0 and  1  are regression parameters and

U i  the stochastic disturbance term

What are the reasons for the insertion of U-term in the model?
2. The following data refers to the demand for money (M) and the rate of interest (R) in for
eight different economics:
M (In billions) 56 50 46 30 20 35 37 61
R% 6.3 4.6 5.1 7.3 8.9 5.3 6.7 3.5
a. Assuming a relationship M    R  U i , obtain the OLS estimators of
 and 
b. Calculate the coefficient of determination for the data and interpret its value
c. If in a 9th economy the rate of interest is R=8.1, predict the demand for money(M) in
this economy.
3. The following data refers to the price of a good ‘P’ and the quantity of the good supplied,
‘S’.
P 2 7 5 1 4 8 2 8
S 15 41 32 9 28 43 17 40

50
 a. Estimate the linear regression line ( S )     P

b. Estimate the standard errors of ˆ and ˆ

c. Test the hypothesis that price influences supply
d. Obtain a 95% confidence interval for 
4. The following results have been obtained from a sample of 11 observations on the values
of sales (Y) of a firm and the corresponding prices (X).
X  519.18
Y  217.82
 X  3,134,543
i
2

 X Y  1,296,836
i i

 Y  539,512
i
2

i) Estimate the regression line of sale on price and interpret the results
ii) What is the part of the variation in sales which is not explained by the
regression line?
iii) Estimate the price elasticity of sales.
5. The following table includes the GNP(X) and the demand for food (Y) for a country over
ten years period.
year 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Y 6 7 8 10 8 9 10 9 11 10
X 50 52 55 59 57 58 62 65 68 70
a. Estimate the food function
b. Compute the coefficient of determination and find the explained and unexplained
variation in the food expenditure.
c. Compute the standard error of the regression coefficients and conduct test of
significance at the 5% level of significance.
6. A sample of 20 observation corresponding to the regression model Yi     X i  U i

gave the following data.

 Y  21.9  Y  Y   86.9
2
i i

 X  186.2   X  X   215.4
2
i i

  X  X Y i i  Y   106.4

a. Estimate  and 

51
 b. Calculate the variance of the estimates
c. estimate the conditional mean of Y corresponding to a value of X fixed at X=10.
7. Suppose that a researcher estimates a consumptions function and obtains the following
results:
C  15  0.81Yd n  19
(3.1) (18.7) R 2  0.99
where C=Consumption, Yd=disposable income, and numbers in the parenthesis are the ‘t-ratios’
a. Test the significant of Yd statistically using t-ratios
b. calculate the standard errors of the parameter estimates
8. State and prove Guass-Markov theorem
9. Given the model:
Yi   0  1 X i  U i with usual OLS assumptions. Derive the expression for the error
variance.

52