Week 3

The simple linear regression model

(SLRM)
Part 2

1
 An introduction to statistical inference

• The population values that describe the true relationship between the variables
would be of more interest than the sample, but are never available.
• We want to make inferences about the likely population values from the
regression parameters.
Example: Suppose we have the following regression results:
yˆ t  20.3  0.5091xt
(14.38) (0.2561)
• $  0.5091 is a single (point) estimate of the unknown population parameter, .
How “reliable” is this estimate?
• The reliability of the point estimate is measured by the coefficient’s standard
error.
• The information from one or more of the sample coefficients and their standard
errors can be used to make inferences about the population parameters.
 Hypothesis testing: Some concepts

• We will always have two hypotheses that go together, the null hypothesis
(denoted H0) and the alternative hypothesis (denoted H1).
• The null hypothesis is the statement or the statistical hypothesis that is actually
being tested. The alternative hypothesis represents the remaining outcomes of
interest.
• For example, suppose given the regression results above, we are interested in
the hypothesis that the true value of  is in fact 0.5. We would use the notation
H0 :  = 0.5
H1 :   0.5
– This states that the hypothesis that the true but unknown value of β could be 0.5 is
being tested against an alternative hypothesis where β is not 0.5.
– This would be known as a two sided test, since the outcomes of both β < 0.5 and β >
0.5 are subsumed under the alternative hypothesis.
 One-sided hypothesis tests

• Sometimes we may have some prior information that, for example, we

would expect  > 0.5 rather than  < 0.5. In this case, we would do a one-
sided test:
H0 :  = 0.5
H1 :  > 0.5
or we could have had
H0 :  = 0.5
H1 :  < 0.5
• This prior information should come from the financial theory of the
problem under consideration, and not from an examination of the
estimated value of the coefficient.
• There are two ways to conduct a hypothesis test: via the test of
significance approach or via the confidence interval approach.
 One-sided hypothesis tests

• In very general terms, if the estimated value is a long way away from the
hypothesised value, the null hypothesis is likely to be rejected.

• If the value under the null hypothesis and the estimated value are close to
one another, the null hypothesis is less likely to be rejected.

• What is required now is a statistical decision rule that will permit the
formal testing of such hypotheses.
 The probability distribution of the least squares estimators

• We assume that ut ~ N(0, σ2)

– Since yt depends partially on ut, it can stated that if ut is normally distributed, yt will also be
normally distributed.

Probability Distribution of yt
• Mean
E(yt) = E( + xt + ut)
= E( + xt) + E(ut)
=  + xt [⸪ E(ut) = 0 and the concept of expectation does not relate to α, β, and xt).
• Variance
Var(yt) = E[yt – E(yt)]2
= E(𝑢𝑡2 ) [⸪ yt = α + βxt + ut)
= σ2
⸫ Probability distribution of yt:
yt ~ N(α + xt, σ2)
 The probability distribution of the least squares estimators

1 𝑋ത 2 መ 𝜎 2
𝛼~𝑁
ො 𝛼, + σ 𝑋𝑡2
𝛽~𝑁 𝛽, σ 𝑋 2
𝑛 𝑡
• simplify, it becomes:

𝛼~𝑁
ො 𝛼, 𝑉𝑎𝑟(α) መ
𝛽~𝑁 𝛽, 𝑉𝑎𝑟(β)

• However, these results are useful when the variance of the disturbance term (𝜎 2 ) is
known, and needs to be estimated as:

ˆ 2   t
ˆ
u 2

T 2
• What if the errors are not normally distributed? Will the parameter estimates still be
normally distributed?

• Yes, if the other assumptions of the CLRM hold, and the sample size is sufficiently
large.
 The probability distribution of the least squares estimators

• Standard normal variates can be constructed from$ and $ by

subtracting the mean and dividing by the square root of the variance:

ˆ   ˆ  
~ N 0,1 and ~ N 0,1
var   var  

• The square roots of the coefficient variances ( var( ) , var(  ) ) are

the standard errors.
• But var() and var() are unknown, so

ˆ   ˆ  
~ tT  2 and ~ tT 2
SE (ˆ ) ˆ
SE (  )
• The standardised statistics follow a t-distribution with T − 2 degrees
of freedom rather than a normal distribution.
 A note on the t and the normal distribution

• You should all be familiar with the normal distribution and its characteristic
“bell” shape, and and its symmetry around the mean (of zero for a standard
normal distribution).

• We can scale a normal variate to have zero mean and unit variance by
subtracting its mean and dividing by its standard deviation.

• There is, however, a specific relationship between the t- and the standard
normal distribution. Both are symmetrical and centred on zero. The t-
distribution has another parameter, its degrees of freedom. We will always
know this (for the time being from the number of observations -2).
 What does the t-distribution look like?

• t-distribution: looks similar to a normal distribution, but with fatter tails, and a smaller peak at the
mean.
• As the number of degrees of freedom for the t-distribution increases from 4 to 40, the critical
values fall substantially.
• This is represented by a gradual increase in the height of the distribution at the centre and a
reduction in the fatness of the tails as the number of degrees of freedom increases.

normal distribution

t-distribution
 Comparing the t and the normal distribution

• In the limit, a t-distribution with an infinite number of degrees of freedom is

a standard normal, i.e., t (¥)  N (01
, ), so the normal distribution can be viewed
as a special case of the t.
• Examples from statistical tables:
Significance level N(0,1) t(40) t(4)
50% 0 0 0
5% 1.64 1.68 2.13
2.5% 1.96 2.02 2.78
0.5% 2.57 2.70 4.60
• The critical values for the t-distribution are larger in absolute value than those
from the standard normal. This arises from the increased uncertainty
associated with the situation where the error variance must be estimated.
– In other words, the reason for using the t-distribution rather than the standard normal is that
we had to estimate  2, the variance of the disturbances.
 Testing hypotheses: The test of significance approach

• Assume the regression equation is given by ,

yt    xt  ut for t = 1,2,...,T
• The steps involved in doing a test of significance are:
1. Estimate $ , $ and SE($ ) , SE( $ ) in the usual way

2. Calculate the test statistic. This is given by the formula

$   *
test statistic 
SE ( $ )
where  * is the value of  under the null hypothesis.
– The estimated value of $ is compared with the value that is subject to test under the
null hypothesis, but this difference is ‘normalised’ or scaled by the standard error
of the coefficient estimate.
– If a standard error is small, the value of the test statistic will be large.
 The test of significance approach (cont’d)

3. We need some tabulated distribution with which to compare the estimated

test statistics. Test statistics derived in this way can be shown to follow a t-
distribution with T-2 degrees of freedom.
As the number of degrees of freedom increases, we need to be less cautious
in our approach since we can be more sure that our results are robust.

4. We need to choose a “significance level”, often denoted . This is also

sometimes called the size of the test and it determines the region where we
will reject or not reject the null hypothesis that we are testing. It is
conventional to use a significance level of 5%.
Intuitive explanation is that we would only expect a result as extreme as this
or more extreme 5% of the time as a consequence of chance alone.
Conventional to use a 5% size of test, but 10% and 1% are also commonly
used.
 Determining the rejection region for a test of significance

5. Given a significance level, we can determine a rejection region and non-

rejection region. If a 5% significance level is employed, this means that 5% of
the total distribution (5% of the area under the curve) will be in the rejection
region. For a 2-sided test:
f(x)

2.5% 95% non-rejection 2.5%

rejection region region rejection region
 The rejection region for a 1-sided test (upper tail)

H0 : β = β∗
H1 : β > β∗
f(x)

95% non-rejection
region 5% rejection region
 The rejection region for a 1-sided test (lower tail)

H0 : β = β∗
H1 : β < β∗ f(x)

95% non-rejection region

5% rejection region
The test of significance approach: Drawing conclusions

6. Use the t-tables to obtain a critical value or values with which to

compare the test statistic.

7. Finally perform the test. If the test statistic lies in the rejection
region then reject the null hypothesis (H0), else do not reject H0.
 The test of significance approach: Drawing conclusions

• One potential problem with the use of a fixed (e.g., 5%) size of test is that if
the sample size is sufficiently large, any null hypothesis can be rejected.
– What happens is that the standard errors reduce as the sample size increases,
thus leading to an increase in the value of all t-test statistics.
– Some econometricians have suggested that a lower size of test (e.g. 1%) should
be used for large samples.

• It is incorrect to state that if the null hypothesis is not rejected, it is

‘accepted’, and it is never said that the alternative hypothesis is accepted or
rejected.
– One reason why it is not sensible to say that the null hypothesis is ‘accepted’ is
that it is impossible to know whether the null is actually true or not!
 The confidence interval approach to hypothesis testing

• An example of its usage: We estimate a parameter, say to be 0.93, and

a “95% confidence interval” to be (0.77,1.09). This means that we are
95% confident that the interval containing the true (but unknown)
value of .

• Confidence intervals are almost invariably two-sided, although in

theory a one-sided interval can be constructed.
How to carry out a hypothesis test using confidence intervals

1. Calculate $ , $ and SE($ ) , SE( $ ) as before.

2. Choose a significance level, , (again the convention is 5%). This is equivalent to

choosing a (1-)100% confidence interval, i.e., 5% significance level = 95%
confidence interval.

3. Use the t-tables to find the appropriate critical value, which will again have T-2
degrees of freedom.

4. The confidence interval is given by ( ˆ  t crit  SE ( ˆ ), ˆ  t crit  SE ( ˆ ))

5. Perform the test: If the hypothesised value of  (i.e., *) lies outside the
confidence interval, then reject the null hypothesis that  = *, otherwise do not
reject the null.
 Confidence intervals vs tests of significance

• Note that the Test of Significance and Confidence Interval approaches

always give the same answer.

• Under the test of significance approach, we would not reject H0 that  =

* if the test statistic lies within the non-rejection region, i.e., if
$   *
tcrit £ $ £ tcrit
SE (  )

• Rearranging, we would not reject if

 t crit  SE ( ˆ ) £ ˆ   * £ t crit  SE ( ˆ )
ˆ  t crit  SE ( ˆ ) £  * £ ˆ  t crit  SE ( ˆ )
• But this is just the rule under the confidence interval approach.
Constructing tests of significance and confidence intervals: An
example

• Using the regression results above,

yˆ t  20.3  0.5091xt , T=22

(14.38) (0.2561)
• Using both the test of significance and confidence interval approaches,
test the hypothesis that  =1 against a two-sided alternative.

• The first step is to obtain the critical value. We want tcrit = t20;5%
 Determining the Rejection Region

f(x)

2.5% rejection region 2.5% rejection region

-2.086 +2.086
 Performing the test

• The hypotheses are:

H0 :  = 1
H1 :   1

Test of significance Confidence interval

approach approach
test stat 
$   * ˆ  t crit  SE ( ˆ )
SE ( $ )
05091
. 1  0.5091  2.086  0.2561
  1917
.
0.2561  (0.0251,1.0433 )
Do not reject H0 since Since 1 lies within the
test stat lies within confidence interval,
non-rejection region do not reject H0
 Testing other hypotheses

• What if we wanted to test H0 :  = 0 or H0 :  = 2?

• Note that we can test these with the confidence interval approach.
For interest (!), test
H0 :  = 0
vs. H1 :   0

H0 :  = 2
vs. H1 :   2

• A confidence interval approach better than a significance approach.

 Changing the size of the test

• But note that we looked at only a 5% size of test. In marginal cases

(e.g., H0 :  = 1), we may get a completely different answer if we use
a different size of test. This is where the test of significance approach
is better than a confidence interval (the interval itself would have to
have been re-estimated since the critical value is embedded in the
calculation of the confidence interval).

• For example, say we wanted to use a 10% size of test. Using the test of
significance approach, $   *
test stat 
SE ( $ )
05091
. 1
  1917
.
0.2561
as above. The only thing that changes is the critical t-value.
Changing the size of the test: The new rejection regions

f(x)

5% rejection region 5% rejection region

-1.725 +1.725
 Changing the size of the test: The conclusion

 t20;10% = 1.725. So now, as the test statistic lies in the rejection region, we would
reject H0.
 Testing of a number of different hypotheses - easier under the confidence interval
approach.
 A consideration of the effect of the size of the test on the conclusion - easier under the
test of significance approach.

 Caution should therefore be used when placing emphasis on or making decisions in

marginal cases (i.e., in cases where we only just reject or not reject).
 The appropriate conclusion to draw is that the results are marginal and that no strong
inference can be made one way or the other.
 Should conduct a sensitivity analysis on the results to determine whether using a different
size of test alters the conclusions.
 If the conclusion (i.e., ‘reject’ or ‘do not reject’) is robust to changes in the size of the test,
then one can be more confident that the conclusions are appropriate.
 If the outcome of the test is qualitatively altered when the size of the test is modified, the
conclusion must be that there is no conclusion one way or the other!
 Some more terminology

• If we reject the null hypothesis at the 5% level, we say that the result of the
test is statistically significant.
• If the null hypothesis is not rejected, the result of the test is ‘not significant’,
or that it is ‘insignificant’.
• If the null hypothesis is rejected at the 1% level, the result is termed ‘highly
statistically significant’.
• Note that a statistically significant result may be of no practical significance.
– E.g., if a shipment of cans of beans is expected to weigh 450g per tin, but the
actual mean weight of some tins is 449g, the result may be highly statistically
significant but presumably nobody would care about 1g of beans.
– E.g., if the estimated beta for a stock under a CAPM regression is 1.05, and a null
hypothesis that β = 1 is rejected, the result will be statistically significant. But it
may be the case that a slightly higher beta will make no difference to an investor’s
choice as to whether to buy the stock or not. In that case, the result of the test was
statistically significant, but financially or practically insignificant.
 The errors that we can make using hypothesis tests

• We usually reject H0 if the test statistic is statistically significant at a

chosen significance level.

• There are two possible errors we could make:

1. Rejecting H0 when it was really true. This is called a type I error.
2. Not rejecting H0 when it was in fact false. This is called a type II error.
Reality
H0 is true H0 is false
Significant Type I error 
Result of (reject H0) =
Test Insignificant Type II error
( do not  =
reject H0)
 The trade-off between type I and type II errors

• The probability of a type I error is just , the significance level or size of test we chose. To
see this, recall what we said significance at the 5% level meant: it is only 5% likely that a
result as or more extreme as this could have occurred purely by chance.
• What happens if we reduce the size of the test (e.g., from a 5% test to a 1% test)? We
reduce the chances of making a type I error ... but we also reduce the probability that we
will reject the null hypothesis at all, so we increase the probability of a type II error:

less likely
→ Lower chance of
to falsely reject
type 1 error.
Reduce size  more strict  reject null
of test criterion for hypothesis more likely to → Higher chance of
rejection less often incorrectly not type II error
reject
• So there is always a trade-off between type I and type II errors when choosing a
significance level. The only way we can reduce the chances of both is to increase the
sample size.
• In practice, type I errors are usually considered more serious and hence a small size of
test is usually chosen (5% or 1% are the most common).
 A special type of hypothesis test: The t-ratio

• Recall that the formula for a test of significance approach to hypothesis

testing using a t-test was
$i   i*
test statistic 
SE $i 
• If the test is H 0 : i = 0
H 1 : i  0
i.e., a test that the population coefficient is zero against a two-sided
alternative, this is known as a t-ratio test:

$i
Since  i* = 0, test stat 
SE ( $i )

• The ratio of the coefficient to its SE is known as the t-ratio or t-statistic.

 The t-ratio: An example

• Suppose that we have the following parameter estimates, standard errors

and t-ratios for an intercept and slope respectively.

Coefficient 1.10 -4.40

SE 1.35 0.96
t-ratio 0.81 -4.63

Compare this with a tcrit with 15-3 = 12 d.f.

(2½% in each tail for a 5% test) = 2.179 5%
= 3.055 1%
• Do we reject H0: 1 = 0? (No)
H 0: 2 = 0? (Yes)
 The t-ratio: An Example

• If the variable is not ‘significant’, it means that while the estimated value of the
coefficient is not exactly zero (e.g. 1.10 in the example), the coefficient is
indistinguishable statistically from zero.
• If a zero were placed in the fitted equation instead of the estimated value, this
would mean that whatever happened to the value of that explanatory variable, the
dependent variable would be unaffected.
– The variable is not helping to explain variations in y, and that it could therefore be removed from the
regression equation.
• It is worth noting that, for degrees of freedom greater than around 25, the 5%
two-sided critical value is approximately ±2.
• So, as a rule of thumb (i.e., a rough guide), the null hypothesis would be rejected if
the t-statistic exceeds 2 in absolute value.
 What does the t-ratio tell us?

• If we reject H0, we say that the result is significant. If the coefficient is not
“significant” (e.g., the intercept coefficient in the last regression), then it
means that the variable is not helping to explain variations in y. Variables that
are not significant are usually removed from the regression model.
• In practice there are good statistical reasons for always having a constant
even if it is not significant. Look at what happens if no intercept is included:
yt

xt
 The exact significance Level or p-value

• This is equivalent to choosing an infinite number of critical t-values from

tables. It gives us the marginal significance level where we would be
indifferent between rejecting and not rejecting the null hypothesis.

• If the test statistic is large in absolute value, the p-value will be small, and
vice versa. The p-value gives the plausibility of the null hypothesis.

e.g. a test statistic is distributed as a t62 = 1.47.

The p-value = 0.12.

• Do we reject at the 5% level?...........................No

• Do we reject at the 10% level?.........................No
• Do we reject at the 20% level?.........................Yes
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
Estimation of SLRM using Eviews
 Estimation of SLRM using Eviews

T-values for H0: β = 0

• 0.00 ≤ p-value 0.01 ≤ → statistically
significant at 1% level.
• 0.01 ≤ p-value 0.05 ≤ → statistically
significant at 5% level.
• 0.05 ≤ p-value 0.10 ≤ → statistically
significant at 10% level.
• p-value > 0.10 → statistically
insignificant.

When p-value > 0.05, the

computed-F is
statistically insignificant.
Not in this case.
Estimation of SLRM using Eviews

Reject H0
Estimation of SLRM using Eviews

Cannot reject H0