LECTURE 2: SIMPLE LINEAR REGRESSION

MODEL

1
 REGRESSION
 Regression is probably the single most important tool at the econometrician’s
disposal.
 But what is regression analysis?
 It is concerned with describing and evaluating the relationship between a given
variable (usually called the dependent variable) and one or more other variables
(usually known as the independent variable(s)).
 Some Notation
 Denote the dependent variable by y and the independent variable(s) by x1, x2, ... , xk
where there are k independent variables.
 Some alternative names for the y and x variables:

y x
dependent variable independent variables
regressand regressors
effect variable causal variables
explained variable explanatory variable
2
 Note that there can be many x variables but we will limit ourselves to the case where
there is only one x variable to start with.
REGRESSION IS DIFFERENT FROM CORRELATION
 If we say y and x are correlated, it means that we are treating y
and x in a completely symmetrical way.
 In regression, we treat the dependent variable (y) and the
independent variable(s) (x’s) very differently.
 The y variable is assumed to be random or “stochastic” in
some way, i.e. to have a probability distribution.
 The x variables are, however, assumed to have fixed (“non-
stochastic”) values in repeated samples.

3
 SIMPLE REGRESSION
 For simplicity, say k=1. This is the situation where y depends on only one x variable.
 Examples of the kind of relationship that may be of interest include:
 How asset returns vary with their level of market risk
 Measuring the long-term relationship between stock prices and
dividends.
 Constructing an optimal hedge ratio
 Example: suppose that we have the following data on the excess returns on
a fund manager’s portfolio (“fund XXX”) together with the excess returns
on a market index:
Year, t Excess return Excess return on market index
= rXXX,t – rft = rmt - rft
1 17.8 13.7
2 39.0 23.2
3 12.8 6.9
4 24.2 16.8
5 17.2 12.3

 We have some intuition that the beta on this fund is positive, and we
therefore want to find whether there appears to be a relationship between x4
and y given the data that we have. The first stage would be to form a scatter
plot of the two variables.
GRAPH (SCATTER DIAGRAM)

45
40
Excess return on fund XXX

35
30
25
20
15
10
5
0
0 5 10 15 20 25
Excess return on market portfolio

5
 FINDING A LINE OF BEST FIT
 We can use the general equation for a straight line,
y=a+bx to get the line that best “fits” the data.
 However, this equation (y=a+bx) is completely deterministic.

 Is this realistic?

 No. So what we do is to add a random disturbance term, u into

the equation.
yt =  + xt + ut where t = 1,2,3,4,5
 Why do we include a Disturbance term?

 The disturbance term can capture a number of features:

 We always leave out some determinants of yt

 There may be errors in the measurement of yt that cannot be

modelled. 6

 Random outside influences on yt which we cannot model

DETERMINING THE REGRESSION COEFFICIENTS
 So how do we determine what  and  are?
 Choose  and  so that the (vertical) distances from the data
points to the fitted lines are minimised (so that the line fits the
data as closely as possible):

x 7
 ORDINARY LEAST SQUARES
 The most common method used to fit a line to the data is known as OLS (ordinary
least squares).
 What we actually do is take each distance and square it (i.e. take the area of each of
the squares in the diagram) and minimise the total sum of the squares (hence least
squares).
 Tightening up the notation, let
yt denote the actual data point t
ŷt denote the fitted value from the regression line
ût denote the residual, yt - ŷt
y

yi

û i

ŷ i

xi x
8

Actual vs fitted
HOW OLS WORKS
5
So min. uˆ  uˆ  uˆ  uˆ  uˆ , or minimise  t
2 2 2 2 2
 1 2 3 4 5 ˆ
u 2
. This is
known as the residual sum of squares. t 1

 But what was uˆ t ? It was the difference between the actual

point and the line, yt - ŷt .

So minimising   yt  yˆ t  is equivalent to minimising

2


with respect to ̂ and .

9
DERIVING THE OLS ESTIMATOR
But yˆ t  ˆ  ˆx t , so let L   ( y t  yˆ t )   ( y t  ˆ  ˆx t )
2 2

t i

 Want to minimise L with respect to (w.r.t.)  and  , so

differentiate L w.r.t.  and 
L (1)
  2  ( yt  ˆ  xt )  0
ˆ
 ̂
ˆ t

L (2)
ˆ
 2  xt ( yt  ˆ  xt )  0
ˆ  t

 From (1),  (y t  ˆ  ˆ x t )  0   y t  T ˆ  ˆ  x t  0
t

 But  y t  Ty and  x t  Tx .
10
DERIVING THE OLS ESTIMATOR (CONT’D)
 So we can write Ty  Tˆ  Tˆx  0 or y  ˆ  ˆx  0 (3)
 From (2),  xt ( y t  ˆ  ˆxt )  0 (4)
t

 From (3), ˆ  y  ˆ x (5)

 Substitute into (4) for  from (5),

 xt ( yt  y  ˆx  ˆxt )  0
t

 t t  t
x y  y x  ˆ
x  t
x  ˆ
  t 0
x
2

 t t
x y  T y x  ˆ
T x 2
 ˆ
  t 0
x
2

11
DERIVING THE OLS ESTIMATOR (CONT’D)
 Rearranging for  ,

ˆ (Tx 2   xt2 )  Tyx   xt yt

 So overall we have:

ˆ
   xt y t  T x y
and ˆ  y  ˆx
 xt2  T x 2

 This method of finding the optimum is known as ordinary

least squares.
12
WHAT DO WE USE  AND  FOR?
 In the CAPM example used above, plugging the 5
observations in to make up the formulae given above would
lead to the estimates
 = -1.74 and  = 1.64. We would write the fitted line
as:
yˆ i   1 . 74  1 . 64  20  31 . 06

 Question: If an analyst tells you that she expects the market

to yield a return 20% higher than the risk-free rate next year,
what would you expect the return on fund XXX to be?
 Solution: We can say that the expected value of y = “-1.74 +
1.64 * value of x”, so plug x = 20 into the equation to get the
expected value for y:
13
ACCURACY OF INTERCEPT ESTIMATE
 Care needs to be exercised when considering the intercept
estimate, particularly if there are no or few observations close
to the y-axis:

0 x 14
THE POPULATION AND THE SAMPLE
 The population is the total collection of all objects or people to be studied,
for example,

 Interested in Population of interest

predicting outcome the entire electorate
of an election
 A sample is a selection of just some items from the population.

 A random sample is a sample in which each individual item in the

population is equally likely to be drawn.

15
THE DATA GENERATING PROCESS (DGP) AND THE PRF
 The population regression function (PRF) is a description of the model
that is thought to be generating the actual data and the true relationship
between the variables (i.e. the true values of  and ).

 The PRF is yt    xt  ut

 The SRF is yˆ t  ˆ  ˆx t

and we also know that .
uˆ t  y t  yˆ t
 We use the SRF to infer likely values of the PRF.

 We also want to know how “good” our estimates of  and  are.

16
LINEARITY
 In order to use OLS, we need a model which is linear in the parameters
( and  ). It does not necessarily have to be linear in the variables (y
and x).

 Linear in the parameters means that the parameters are not multiplied
together, divided, squared or cubed etc.

 Some models can be transformed to linear ones by a suitable substitution

or manipulation, e.g. the exponential regression model

Y t  e  X t e u t  ln Y t     ln X t  u t

 Then let yt=ln Yt and xt=ln Xt

yt    xt  ut
17
LINEAR AND NON-LINEAR MODELS

 This is known as the exponential regression model. Here, the coefficients

can be interpreted as elasticities.

 Similarly, if theory suggests that y and x should be inversely related:


yt     ut
xt
then the regression can be estimated using OLS by substituting
1
zt 
xt
But some models are intrinsically non-linear, e.g. 

yt    xt  ut

 Estimator or Estimate?
 Estimators are the formulae used to calculate the coefficients
 Estimates are the actual numerical values for the coefficients.
18
 THE ASSUMPTIONS UNDERLYING THE
CLASSICAL LINEAR REGRESSION MODEL (CLRM)
 The model which we have used is known as the classical linear
regression model.
 We observe data for xt, but since yt also depends on ut, we must be
specific about how the ut are generated.
 We usually make the following set of assumptions about the ut’s (the
unobservable error terms):
 Assumptions Interpretation
1. E(ut) = 0 The errors have zero mean
2. Var (ut) = 2 The variance of the errors is constant and finite
over all values of xt (homoskedasticity)
3. Cov (ui, uj)=0 The errors are statistically independent of
one another
4. Cov (ut, xt)=0 No relationship between the error and
corresponding x variate
19
 THE ASSUMPTIONS UNDERLYING THE CLRM AGAIN

 An alternative assumption to 4., which is slightly stronger, is that the xt’s

are non-stochastic or fixed in repeated samples.

 A fifth assumption is required if we want to make inferences about the

population parameters (the actual  and ) from the sample parameters
( and ).

 Additional Assumption
5. ut is normally distributed

20
 PROPERTIES OF THE OLS ESTIMATOR

 If assumptions 1. through 4. hold, then the estimators  and determined by

OLS are known as Best Linear Unbiased Estimators (BLUE).
 What does the acronym stand for?
 “Estimator” -  is an estimator of the true value of .

 “Linear” -  is a linear estimator

 “Unbiased”- on average, the actual value of the  and  ’s will be equal
to the true values.
 “Best”- means that the OLS estimator  has minimum variance among
the class of linear unbiased estimators.
 The Gauss-Markov theorem proves that the OLS estimator is best.

21
 CONSISTENCY/UNBIASEDNESS/EFFICIENCY
 Consistent
 The least squares estimators  and are consistent.
 i.e., the estimates will converge to their true values as the sample size
increases to infinity.
 Unbiased
 The least squares estimates  and  are unbiased.
E( ) = and E(  ) =
 Unbiasedness is a stronger condition than consistency.
 Efficiency
 An estimator  of parameter  is said to be efficient if it is unbiased and
no other unbiased estimator has a smaller variance.
 If the estimator is efficient, we are minimising the probability that it is a
long way off from the true value of .

22
PRECISION AND STANDARD ERRORS
 Recall that the estimators of  and  from the sample parameters (  and
) are given by:
ˆ   t 2 t
x y  Tx y
andˆ  y  ˆx
 xt  T x 2

 Beside the coefficients what we need is some measure of the reliability or

precision of the estimators ( and  ).
 The precision of the estimate is given by its standard error.
 Given assumptions 1 - 4 above, then the standard errors are given by:

x  xt
2 2

SE (ˆ )  s t
s ,
T  (x  x)
t
2
T x T x
2
t
2 2

1 1
SE ( ˆ )  s s
 ( xt  x ) 2
 t
x 2
 T x 2

where s is the estimated standard deviation of the residuals. 23

ESTIMATING THE VARIANCE OF THE DISTURBANCE TERM

 The variance of the random variable ut is given by:

Var(ut) = E[(ut)-E(ut)]2
which reduces to
Var(ut) = E(ut2)

 We could estimate this using the average of ut2 :

1
s2 
T
 ut2
 Unfortunately, this is not workable since ut is not observable.
 We can use the sample counterpart to ut, which is û :
t
1
s   uˆt2
2

T
 But this estimator is a biased estimator of s2.
An unbiased estimator of  is given by

s
 uˆ 2
t
24

T 2
EXAMPLE: HOW TO CALCULATE THE PARAMETERS AND STANDARD ERRORS

 Assume we have the following data calculated from a regression of y on

a single variable x and a constant over 22 observations.
 Data:
 xt yt  830102, T  22, x  416.5, y  86.65,
 t  3919654, RSS  130.6
x 2

 Calculations:
 830102  (22 * 416.5 * 86.65)
  2  0.35
3919654  22 *(416.5)

 We write   86.65  0.35 * 416.5  59.12

yˆ t  ˆ  ˆx t
ŷ t  59.12  0.35x t 25
EXAMPLE (CONT’D)

 SE(regression),

s
 uˆ t2

130.6
 2.55
T 2 20

3919654
SE ( )  2.55 *  3.35
22  3919654  22  416.5 
2

1
SE (  )  2.55 *  0.0079
3919654  22  416.5 
2

 We now write the results as

yˆ t   59.12  0.35 xt
(3.35) (0.0079)
26
 STATISTICAL INFERENCE
 We want to make inferences about the likely population values from the
regression parameters.

 Example: Suppose we have the following regression results:

yˆ t  20.3  0.5091xt
(14.38) (0.2561)

 ˆ  0.5091 is a single (point) estimate of the unknown population

parameter, .
 How “reliable” is this estimate?
 The reliability of the point estimate is measured by the coefficient’s
standard error.

27
 HYPOTHESIS TESTING
 We can use the information in the sample to make inferences about the
population.
 We will always have two hypotheses that go together, the null hypothesis
(denoted by H0) and the alternative hypothesis (denoted by H1).
 The null hypothesis is the statement or the statistical hypothesis that is
actually being tested.
 The alternative hypothesis represents the remaining outcomes of interest.
 For example, suppose given the regression results above, we are interested
in the hypothesis that the true value of  is in fact 0.5.
 We would use the notation:
H0 :  = 0.5
H1 :   0.5
 This would be known as a two sided test.

28
 ONE-SIDED HYPOTHESIS TESTS
 Sometimes we may have some prior information that, for example, we
would expect  > 0.5 rather than  < 0.5.
 In this case, we would do a one-sided test:
H0 :  = 0.5
H1 :  > 0.5
or we could have had
H0 :  = 0.5
H1 :  < 0.5
 There are two ways to conduct a hypothesis test:
 via the test of significance approach or
 via the confidence interval approach.

29
DISTRIBUTION OF THE LEAST SQUARES ESTIMATORS

 We assume that ut  N(0,2)

 Since the least squares estimators are linear combinations of the random
variables, i.e.   w y
 t t
 The weighted sum of normal random variables is also normally
distributed, so:
  N(, Var())
  N(, Var())

 What if the errors are not normally distributed?

 Will the parameter estimates still be normally distributed?
 Yes, if the other assumptions of the CLRM hold, and the sample size is sufficiently
large.

30
DISTRIBUTION OF THE LEAST SQUARES ESTIMATORS (CONT’D)

 Standard normal variates can be constructed from and  :

ˆ  
~ N 0,1 ˆ  
var  and ~ N 0,1
var 

 But var() and var() are unknown, so

ˆ   and ˆ  
~ tT  2 ~ tT  2
SE (ˆ ) ˆ
SE (  )

31
TESTING HYPOTHESES: SIGNIFICANCE APPROACH

 Assume the regression equation is given by ,

y t     x t  u t for t=1,2,...,T

 The steps involved in doing a test of significance are:

1. Estimate  ,  and SE( ) , SE(  ) in the usual way

2. Calculate the test statistic. This is given by the formula

   *
test statistic 
SE (  )
where  * is the value of  under the null hypothesis.

32
THE TEST OF SIGNIFICANCE APPROACH (CONT’D)
3. We need some tabulated distribution with which to compare the
estimated test statistics.
 Test statistics derived in this way can be shown to follow a t-distribution with T-2
degrees of freedom.
 As the number of degrees of freedom increases, we need to be less cautious in our
approach since we can be more sure that our results are robust.

4. We need to choose a “significance level”, often denoted .

 This is also sometimes called the size of the test and it determines the region
where we will reject or not reject the null hypothesis that we are testing.
 It is conventional to use a significance level of 5%.
 Intuitive explanation is that we would only expect a result as extreme
as this or more extreme 5% of the time as a consequence of chance
alone.
 Conventional to use a 5% size of test, but 10% and 1% are also
commonly used.
33
DETERMINING THE REJECTION REGION FOR A TEST OF SIGNIFICANCE

5. Given a significance level, we can determine a rejection region and non-

rejection region. For a 2-sided test:

f(x)

2.5% 95% non-rejection 2.5%

rejection region region rejection region

34
THE REJECTION REGION FOR A 1-SIDED TEST
f(x )

Upper tail
9 5 % n on -re je c tio n
5 % re jec tio n re gio n

f(x)

5% rejection region
95% non-rejection region Lower tail
35
TEST OF SIGNIFICANCE APPROACH: DRAWING CONCLUSIONS

6. Use the t-tables to obtain a critical value or values with which to

compare the test statistic.

7. Finally perform the test.

 If the test statistic lies in the rejection region then reject the null
hypothesis (H0), else do not reject H0.

36
THE CONFIDENCE INTERVAL APPROACH TO HYPOTHESIS TESTING

 An example of its usage:

 We estimate a parameter, say to be 0.93, and a “95% confidence
interval” to be (0.77, 1.09).
 This means that we are 95% confident that the interval containing the
true (but unknown) value of .
 Confidence intervals are almost invariably two-sided, although in theory
a one-sided interval can be constructed.

37
HOW TO CARRY OUT A HYPOTHESIS TEST USING CONFIDENCE INTERVALS

1. Calculate  ,  and SE( ) , SE(  ) as before.

2. Choose a significance level, , (again the convention is 5%).

 This is equivalent to choosing a (1-)100% confidence interval, i.e. 5%
significance level = 95% confidence interval

3. Use the t-tables to find the appropriate critical value, which will again
have T-2 degrees of freedom.

4. The confidence interval is given by:

( ˆ  t crit  SE ( ˆ ), ˆ  t crit  SE ( ˆ ))
5. Perform the test: If the hypothesised value of  (*) lies outside the
confidence interval, then reject the null hypothesis that  = *, otherwise
do not reject the null.
38
CONFIDENCE INTERVALS VERSUS TESTS OF SIGNIFICANCE
 Note that the Test of Significance and Confidence Interval approaches
always give the same answer.

 Under the test of significance approach, we would not reject H0 that  = *

if the test statistic lies within the non-rejection region, i.e. if

   *
t crit   tcrit
SE (  )
SE(
 Rearranging, we would not reject if

 t crit  SE ( ˆ )  ˆ   *  t crit  SE ( ˆ )

 But this is just the rule under the confidence interval approach.
ˆ  t crit  SE ( ˆ )   *  ˆ  t crit  SE ( ˆ ) 39
EXAMPLE
 Using the regression results above,
yˆ t  20.3  0.5091xt
(14.38) (0.2561) , T=22

 Using both the test of significance and confidence interval approaches, test
the hypothesis that  =1 against a two-sided alternative.

 The first step is to obtain the critical value.

 We want tcrit = t20;5%
f(x )

2 .5 % re je c tio n re g io n 2 .5 % re je c tio n re g io n

40
-2 .0 8 6 + 2 .0 8 6
PERFORMING THE TEST

 The hypotheses are:

H0 :  = 1
H1 :   1

Test of significance approach Confidence interval approach

   * ˆ  t crit  SE ( ˆ )
test stat 
SE (  )
05091
. 1  0.5091  2.086  0.2561
  1917
.
0.2561  (0.0251,1.0433)
Do not reject H0 since Since 1 lies within the
test stat lies within confidence interval,
non-rejection region do not reject H0

41
CHANGING THE SIZE OF THE TEST
 But note that we looked at only a 5% size of test. In marginal cases
(e.g. H0 :  = 1), we may get a completely different answer if we use a
different size of test. This is where the test of significance approach is
better than a confidence interval.

 For example, say we wanted to use a 10% size of test. Using the test of
significance approach,    *
test stat 
SE (  )
05091
. 1
  1917
.
0.2561
as above. The only thing that changes is the critical t-value.
f(x )

5 % re je c tio n r e g io n 5 % re je c tio n r e g io n
42

- 1 .7 2 5 + 1 .7 2 5
CHANGING THE SIZE OF THE TEST: THE CONCLUSION
 t20;10% = 1.725. So now, as the test statistic lies in the rejection region, we
would reject H0.

 Caution should therefore be used when placing emphasis on or making

decisions in marginal cases (i.e. in cases where we only just reject or not
reject).
 Note
 If we reject the null hypothesis at the 5% level, we say that the result of the test is
statistically significant.
 Note that a statistically significant result may be of no practical
significance.
 E.g. if a shipment of cans of beans is expected to weigh 450g per tin,
but the actual mean weight of some tins is 449g, the result may be
highly statistically significant but presumably nobody would care
about 1g of beans.
43
THE ERRORS THAT WE CAN MAKE USING HYPOTHESIS TESTS

 We usually reject H0 if the test statistic is statistically significant at a chosen

significance level.
 There are two possible errors we could make:
1. Rejecting H0 when it was really true. This is called a type I error.
2. Not rejecting H0 when it was in fact false. This is called a type II error.

Reality
H0 is true H0 is false
Significant Type I error 
Result of (reject H0) =
Test Insignificant Type II error
( do not  =
reject H0)
44
THE TRADE-OFF BETWEEN TYPE I AND TYPE II ERRORS

 The probability of a type I error is just , the significance level or size of

test we chose.
 To see this, recall what we said significance at the 5% level meant: it is
only 5% likely that a result as or more extreme as this could have occurred
purely by chance.
 Note that there is no chance for a free lunch here! What happens if we
reduce the size of the test (e.g. from a 5% test to a 1% test)? We reduce the
chances of making a type I error ... but we also reduce the probability that
we will reject the null hypothesis at all, so we increase the probability of a
type II error: less likely
to falsely reject
Reduce size  more strict  reject null
of test criterion for hypothesis more likely to
rejection less often incorrectly not
reject

 So there is always a trade off between type I and type II errors when45
choosing a significance level. The only way we can reduce the chances of
both is to increase the sample size.
A SPECIAL TYPE OF HYPOTHESIS TEST: THE T-RATIO
 Recall that the formula for a test of significance approach to hypothesis
testing using a t-test was:
i   i*
test statistic 
SE  i 
 If the test is H 0 : i = 0
H1 : i  0
i.e. a test that the population coefficient is zero against a two-sided
alternative, this is known as a t-ratio test:
i
test stat 
 Since  i* = 0, SE ( i )

 The ratio of the coefficient to its SE is known as the t-ratio or t-statistic.

46
EXAMPLE OF T-RATIO

 Suppose that we have the following parameter estimates, standard errors

and t-ratios for an intercept and slope respectively.

Coefficient 1.10 -4.40

SE 1.35 0.96
t-ratio 0.81 -4.63

 Compare this with a tcrit with 15-3 = 12 d.f.

(2.5% in each tail for a 5% test) = 2.179 5%
= 3.055 1%
 Do we reject H 0: 1 = 0? (No)
H0 : 2 = 0? (Yes)

47
WHAT DOES THE T-RATIO TELL US?

 If we reject H0, we say that the result is significant.

 If the coefficient is not “significant” (e.g. the intercept coefficient in the
last regression above), then it means that the variable is not helping to
explain variations in y.
 Variables that are not significant are usually removed from the regression
model.
 In practice there are good statistical reasons for always having a constant
even if it is not significant. Look at what happens if no intercept is
included:
yt

xt
48
See lab exercise 1 for practical lesson!

49