Chapter III: The Simple Regression Model

Regression analysis refers to estimating functions

showing the r/ship b/n two or more variables and
corresponding tests(test for significance parameters).
Types of regression:
1. If it estimate functions showing the r/ship b/n only two
variables regression is simple linear regression
2. If it estimate functions showing the r/ship b/n more than
two variables regression is multiple linear regression
Now, we will restrict our discussion in this part only to
two variables and deal with more variables in the next
section.
 Ordinary Least Square Method (OLS) and
Classical Assumptions
 There are two major ways of estimating regression
functions. These are:
1. OLS method and
2. maximum likelihood (MLH) method.
3. Moment method (read this method)
 Both the methods are basically similar to their
application in estimations

 OLS method is the easiest and the most commonly

used method as opposed to the maximum likelihood
(MLH) method which is limited by its assumptions.
 For instance, the MLH method is valid only for
large sample as opposed to the OLS method which can
be applied to smaller samples.
 Owing to this merit, our discussion mainly focuses
on the ordinary least square (OLS).
 Ordinary Least Square Method (OLS)
 OLS method of estimating parameters or regression
function is about finding or estimating values of the
parameters ( of the simple linear regression function given
below for which the errors or residuals are minimized.
 Thus, it is about minimizing the residuals or the errors.

Yi    X i  U i
 The above identity represents population regression
function (to be estimated from total enumeration of data
from the entire population).
 Ordinary Least Square Method (OLS)
But, most of the time it is difficult to generate population
data owing to several reasons; and most of the time we
use sample data to estimate sample regression function.
Thus, we use the following sample regression function
for the derivation of the parameters and related analysis.

Yˆi  ˆ  ˆX i
 Before discussing the details of the OLS estimation techniques,
let‟s see the classical assumptions that are necessary for the
validity of the analysis, interpretations and conclusions of the
regression function.
 Classical Assumptions(CA)
For the validity of a regression function our regression
function should fulfill the following conditions known as
classical assumptions.
1. The error terms ‘Ui’ are the disturbance terms are not
correlated.
 relation among the value of the error terms (Ui and Uj);
Where i = 1, 2, 3, …….., j = 1, 2, 3, ……. and .
 Cov (Ui , Uj) = 0 for population data
 Cov (ei, ej) = 0 for sample data
Otherwise, the error terms do not serve an adjustment
purpose rather it causes an autocorrelation problem.
CA---Cont’d
2. The disturbance terms „Ui‟ have zero mean.
This implies the sum of the individual disturbance terms
is zero.
The disturbance terms are negative; some are zero and
some are positive and the sum or the average is zero.
This is given by the following identities.
E(Ui) =  U i  0 . for population
The same argument is true for sample regression
function and so for residual terms given as follows:
for sample
 Otherwise the estimated models will be biased and
cause the regression function to shift.
 For instance: if (or positive) it is going to shift the
estimation upward from the true representative model.
 This is demonstrated by the following fig. Error not zero
CA---Cont’d
3. The disturbance terms have constant variance in each
period.
This is given as follows:
Var U i   E ((U i  E (U i )) 
2
 2
 = u .
2

This assumption is known as the assumption of

homoscedasticity.
If this condition is not fulfilled or if the variance of the
error terms varies as sample size changes or as the value
of explanatory variables changes, then this leads to
Heteroscedasticity problem.
CA---Cont’d
4. Explanatory variables „Xi‟ and disturbance terms „Ui‟ are
uncorrelated or independent.
 All the co-variances of the successive values of the error
term are equal to zero. This condition is given by .

Cov U i , X i   0 .

 It is followed from this that the following identity holds

true; .
 ei X i  0 .
CA---Cont’d
5. The explanatory variable Xi is fixed in repeated samples.
Each value of Xi does not vary for instance owing to
change in sample size.
This means the explanatory variables are non-random and
hence distributional free variable.
6. Linearity of the model in parameters.
 The SLR requires linearity in parameters; but not necessarily
linearity in variables.
 The same technique can be applied to estimate regression functions
of the following forms: Y = f (X ); Y = f (X 2); Y = f (X 3); Y = f
(X – kX ); and so on. What is important is transforming the data as
required.
CA---Cont’d
7. Normality assumption-
 The disturbance term Ui is assumed to have a normal distribution
with zero mean and a constant variance(homoscedasticity).
 This assumption is given as follows:

Ui ˜ 
N 0,  u
2
.
 This assumption or combination of assumptions is used in testing
hypotheses about significance of parameters.
 It is also useful in both estimating parameters and testing their
significance in maximum likelihood method.
CA---Cont’d
8. Explanatory variables should not be perfectly, linearly and/or
highly correlated.
 Using explanatory variables which are highly or perfectly
correlated in a regression function causes a biased function or
model. It also results in multicollinearity problem.
9. The relationship between variables (or the model) is correctly
specified.
 All the necessary variables are included in the model.
 For instance, “Y = f (X 2)” may better reflect the relationship
between Y and X than “Y = f (X )”.
10.The explanatory variables do not have identical value.
 This assumption is very important for improving the precision of
estimators.
 OLS method of Estimation

 Suppose we want to estimate the following equation

• The procedure is given as follows.
Yi   0  1 X i  U i

• it is difficult to get population data the corresponding

sample regression function is given as follows.
Yˆi  ˆ0  ˆ1 X i

• From this identity, we solve for the residual term , ' e ' ,
i

square both sides and then take sum of both sides. These
three steps are given (respectively as follows.
 ei  Yi  Yˆi  Yi  ˆ0  ˆ1 X i

 ei   Yi  ˆ0  ˆ1 X i
2
 
2

Where,  ei  RSS= Residual Sum of Squares.

2

 To minimize the residual sum of squares we take the first order

partial derivatives of Equation above and equate them to zero.
That is, the partial derivative with respect to ˆ 0 :

  ei
 
2
 2  Yi  ˆ0  ˆX i (1)  0 2.9
ˆ 0

 Y i  ˆ0  ˆ1 X i  0  2.10

  Yi  nˆ0  ˆ1  X i =0 2.11

 Y i  nˆ0  ˆ1  X i 2.12

Where n is the sample size.

 Partial derivative With respect to ˆ1
  ei
 
2
 2  Yi  ˆ0  ˆ1 X i ( X i )  0 2.
ˆ 1

  Y X
i i  ˆ
 X
0 i  ˆ
1 i 0
X 2
 2

  X i Yi  ˆ0  X i  ˆ1  X i
2
2

^ ^
Note that the equation (Yi   0  1 X i ) 2 is a composite function and we should apply a
chain rule in finding the partial derivatives with respect to the parameter estimates.
Equations from above are together called the system of normal equations. Solving the system
of normal equations simultaneously we obtain:
 n XY  ( X )( Y )
ˆ1 
Or
n X 2  ( X ) 2

_ _
  XY  n Y X and we have ˆ 0  Y  ˆ1 X
1  _
from above
 X i2  n X 2
Example 2.4: Given the following sample data of three pairs of „Y‟ (dependent variable) and
„X‟ (independent variable), find a simple linear regression function; Y = f(X).

Yi Xi
10 30
20 50
30 60

a) find a simple linear regression function; Y = f(X)

b) Interpret your result.
c) Predict the value of Y when X is 45.
Solution
To fit the regression equation we do the following computations.
Yi Xi Yi Xi Xi2
10 30 300 900
20 50 1000 2500
30 60 1800 3600
Sum 60 140 3100 7000
Mean Y = 20 140
X=
3

n XY  ( X )( Y )
3(3100)  (140)(60)
ˆ1    0.64
3(7000)  (140) 2
n X 2  ( X ) 2

ˆ0  Y  ˆ1 X  20  0.64(140 / 3)  10

Thus the fitted regression function is given by: Yˆi   10  0.64 X i

 b) Interpretation, the value of the intercept term,-10, implies that the value of the
dependent variable „Y‟ is – 10 when the value of the explanatory variable is zero. The
value of the slope coefficient ( ˆ  0.64 ) is a measure of the marginal change in the
dependent variable „Y‟ when the value of the explanatory variable increases by one.
For instance, in this model, the value of „Y‟ increases on average by 0.64 units when
„X‟ increases by one.
c) Predication Yˆi   10  0.64 X i =-10+(0.64)(45)=18.8
That means when X assumes a value of 45, the value of Y on average is expected to be 18.8.
The regression coefficients can also be obtained by simple formulae by taking the deviations
between the original values and their means.
 Mean and Variance of Parameter Estimates

 Formula for mean and variance of the respective

parameter estimates and the error term are:
 
1. The mean of 1  E (1 )  1
     U2
2. The variance of 1  Var ( 1 )  E ((1  E ( )) 2 
 xi2
  
3. The mean of  0  E (  0 )   0
    U2  X i2
4. The variance of  0  E (( 0  E ( 0 )) 2 
n  xi2

 ei2
5. The estimated value of the variance of the error term  U 
2

n2
 Hypothesis Testing of OLS Estimates

 A model must be tested for its significance before

it can be used for any other purpose.
 The available test criteria are divided in to three
groups:
I. Theoretical or a priori criteria,
II. Statistical criteria and
III. Econometric criteria.
I. Priori criteria set by economic theories are in
line with the consistency of coefficients of
econometric model to the economic theory.
II. Statistical criteria(first order tests) are set by
statistical theory and refer to evaluate the statistical
reliability of the model.
III.Econometric criteria refer to whether the
assumptions of an econometric model employed in
estimating the parameters are fulfilled or not.

 Two most commonly used tests in econometrics.

These are:
1. Square of correlation coefficient
2. Standard error test
1. The square of correlation coefficient (r2)
 Used for judging the explanatory power of X‟s
 r2 = R2 (in simple regression model)
 R2 measures the goodness of fit of the model.
2. Standard error test
For judging the statistical reliability of the
estimates(Significance test)
Measures the degree of confidence that we may
attribute to the estimates.
 The Coefficient of determination (R2)
The R2 is the measure of the amount of the total
variation of the dependent variable that is explained
by the model(presented explanatory variable in the
model).
• The total variation of the dependent variable is
measured from its arithmetic mean: TSS=ESS + RSS
_
Total var iationin Yi  (Yi  Y ) 2
 _
Total exp lained var iation  (Yi  Y ) 2

Total un exp lained var iation   ei2

The coefficient of determination is given by the formula:

 (Yi  Yˆi ) 2  ei
2

Or R2  1
Unexplained Variation in Y
 1  1
Total Variation inY
 i
(Y  Y ) 2
 yi 2

2 
Since  y i   1  xi y i

 1  xi y i
R 
2

 y i2
 The higher R2 is the better the fit(good model)

 The smaller R2 is the poorer the fit(bad model)

 Why the R2 is used to compare two or more

models. B/c:
 1-R2 is the coefficient of non-determination, and
 1-R2 is proportion of unexplained by the model.
2. Testing the significance of a given regression coefficient

 There are different tests that are available to

test the statistical reliability(significance) of
the parameter estimates.

 Three common of testing of significance :

A. The standard error test
B. The standard normal test
C. The students t-test
A. The Standard Error Test
• This test first establishes the two hypotheses that
are going to be tested.
1. The null hypothesis (H0) addresses that the sample
is coming from the population whose parameter is
not significantly different from zero
2. The alternative hypothesis (H1) addresses that the
sample is coming from the population whose
parameter is significantly different from zero.
 The two hypotheses are given as follows
symbolically:
 H0: βi=0
 H1: βi≠0
 The standard error test is outlined as follows:
1. Compute the standard deviations of parameters
which is:   U2
se( 1 ) 
 xi2
  U2  X i2
se(  0 ) 
n  xi2

2. Compare the standard errors of the estimates with

the numerical values of the estimates and make
decision.
 1 
1. If se(  i )  2 (  i ), ,reject the null hypothesis and we can
conclude that the estimate is statistically significant.
 outlined cont‟d---:
 1 
2. If se(  i )  ( i ) ,
, accept the null hypothesis and
2
conclude that the estimate is not statistically
significant.
B.The Standard Normal Test
This test is based on the normal distribution. The test is
applicable if:
 The standard deviation of the population is known
irrespective of the sample size
 The standard deviation of the population is unknown
provided that the sample size is sufficiently large (n>30).
The standard normal test or Z-test is outline as follows;
1.Specify the (H0) and the (H1).
2.Specify the significance level, 
3.Identify the sampling distribution (if it is Z or
t) of the estimator.
4.Identify the critical region( Ztab or ttab)
5.Calculate a statistic. ( Zcal or tcal)
6.Making decision(reject or accept H0)
7.Summarization of the result.(conclusion)
 Accepting the null hypothesis(H0) means that
rejecting the alternative hypothesis (H1) and vice
versa.
 Hypothesized value of is denoted by , then
one can formulate two sided (1) and one sided (2
and 3) hypothesis as follows:

1. H 0 :   0 vs H1 :   0
2. H 0 :   0 vs H1 :   0
3. H 0 :   0 vs H1 :   0
 Calculate a statistic. ( Zcal or tcal)
Case 1: When sampling is from a normal distribution
with known standard deviation
 The relevant test statistic is:

X  0
Z cal 
 n
 Decision
H0 Reject H0 if Accept H0 if Inconclusive if
  0 Z cal  Z 2 Z cal  Z 2 Z cal  Z 2 or Z cal  Z 2
  0 Z cal  Z Z cal  Z Z cal  Z
  0 Z cal  Z Z cal  Z Z cal  Z
 Calculate a statistic. ( Zcal or tcal)
Case 2: When sampling is from a normal distribution with
unknown standard deviation and small sample size
 The relevant test statistic is
X  0
t cal  ~ t with n  1 deg rees of freedom.
S n

 Decision
H0 Reject H0 if Accept H0 if Inconclusive if
  0 tcal  t 2 tcal  t 2 tcal  t 2 or tcal  t 2
  0 tcal  t tcal  t tcal  t
  0 tcal  t tcal  t tcal  t
 Calculate a statistic. ( Zcal or tcal)
Case 3: When a population whose functional form is
unknown.
 The relevant test statistic is

X  0
Z cal  , if  2 is known.
 n
X  0
 , if  2 is unknown.
S n

 Decision: The decision rule is the same as case I.

  
Example: If the regression has a value of 1 =29.48 and the standard error of 1 is 36. Test
the hypothesis that the value of 1  25 at 5% level of significance using standard normal
test.
Solution: We have to follow the procedures of the test.
H 0 : 1  25
H1 : 1  25
After setting up the hypotheses to be tested, the next step is to determine the level of
significance in which the test is carried out. In the above example the significance level is
given as 5%.
The third step is to find the theoretical value of Z at specified level of significance. From the
standard normal table we can get that Z 0.025  1.96 .
The fourth step in hypothesis testing is computing the observed or calculated value of the
standard normal distribution using the following formula.
ˆ1  1 29.48  25
Z cal    0.12 . Since the calculated value of the test statistic is less than
ˆ
se( 1 ) 36
the tabulated value, the decision is to accept the null hypothesis and conclude that the value of
the parameter is 25.
 C.The Student t-Test
In conditions where Z-test is not applied (in small samples), t-test can be used to test the
statistical reliability of the parameter estimates. The test depends on the degrees of freedom
that the sample has. The test procedures of t-test are similar with that of the z-test. The
procedures are outlined as follows;
1. Set up the hypothesis. The hypotheses for testing a given regression coefficient is given
by:
H 0 : i  0
H1 :  i  0
2. Determine the level of significance for carrying out the test. We usually use a 5% level
significance in applied econometric research.
3.Determine the tabulated value of t from the table with n-k degrees of freedom, where k is
the number of parameters estimated.
4.Determine the calculated value of t. The test statistic (using the t- test) is given by:
ˆi
tcal 
se( ˆi )
The test rule or decision is given as follows:
Reject H0 if | t cal | t / 2,nk
Example: The following table gives the quantity supplied
(Y in tons) and its price (X pound per ton) for a commodity
over a period of twelve years.
Y 69 76 52 56 57 77 58 55 67 53 72 64
X 9 12 6 10 9 10 7 8 12 6 11 8

Use Tables to answer the following questions

1. Estimate the Coefficient of determination (R2)
2. Run significance test of regression coefficients using the
following test methods
A)The standard error test
B)The students t-test
3. Fit the linear regression equation
 Properties of OLS Estimators
• Optimum properties that the OLS estimates possess may be
summarized by well known theorem known as the Gauss-
Markov Theorem.
 The OLS estimators are BLUE.
• The theorem referred as the BLUE theorem i.e. Best, Linear,
Unbiased Estimator. An estimator is called BLUE if:
a. Linear: a linear function of the random variable, such as, the
dependent variable Y.
b. Unbiased: its average or expected value is equal to the true
population parameter.
c. Minimum variance: It has a minimum variance in the class of
linear and unbiased estimators. An unbiased estimator with the
least variance is known as an efficient estimator.