Lecture 4.

Multiple
Regression Analysis:
Inference
Prepared by Quanquan Liu
Fall 2024
Sampling Distributions of the OLS
Estimators
 Assumption MLR.6. Normality
 The population error u is independent of the explanatory variables and is normally
distributed with zero mean and variance .
 u is independent of the ;
 ;
 .
 For cross-sectional regression applications, Assumptions MLR.1-6 are called the classical
linear model (CLM) assumptions.
 Under the CLM assumptions, the OLS estimators are the minimum variance unbiased
estimators.
Sampling Distributions of the OLS
Estimators
 Under the CLM assumptions
.
 Theorem 1. Normal Sampling Distributions
 Under the CLM Assumptions MLR.1 through MLR.6, conditional on the sample values of
the independent variables,
,
where . Therefore,
.
 Going further: any linear combination of the is also normally distributed, and any subset
of the has a joint normal distribution.
Testing Single Population Parameter:
The t Test
 Theorem 2. t Distribution for the Standardized Estimators
 Under the CLM Assumptions MLR.1 through MLR.6,
,
where is the number of unknown parameters in the population model ( slope
parameters and the intercept ) and is the degrees of freedom (df).
 Assume our primary interest lies in testing the null hypothesis
,
where j corresponds to any of the k independent variables.
 Null hypothesis: In classical hypothesis testing, we take this hypothesis as true and require the
data to provide substantial evidence against it.
The t Test

 The statistic we use to test (the t statistic or the t ratio of ) is defined as

 The t-statistic will be used to test the above null hypothesis. The farther the estimated coefficient
is away from zero, the less likely it is that the null hypothesis holds true. But what does “far”
away from zero mean?
 Distribution of the t-statistic if the null hypothesis is true

 Goal: Define a rejection rule so that, if it is true, H0 is rejected only with a small probability
(= significance level, e.g. 5%)
The t Test

 Alternative hypothesis: The hypothesis against which the null hypothesis is tested.
 Consider a one-sided alternative of the form
.
We are testing against .
 Significance level: The probability of a Type I error (rejecting the null hypothesis when it's actually
true) in hypothesis testing.
 A 5% significance level means that we are willing to mistakenly reject H 0 when it is true 5% of the time.

 We are looking for a “sufficiently large” positive value of in order to reject in favor of .
 The definition of “sufficiently large,” with a 5% significance level, is the 95 th percentile in a t
distribution with n − k − 1 degrees of freedom; denote this by c.
 The rejection rule is that: is rejected in favor of at the 5% significance level if .
The t Test

 The rejection rule is that is rejected in

favor of at the 5% significance level if
.
 For example, for a 5% level test and with
n − k − 1 = 28 degrees of freedom, the
critical value is c = 1.701. If , then we fail
to reject in favor of at the 5% level.
 Excel:
 Stata: display invttail(28, 0.05)
 As the degrees of freedom in the t
distribution get large, the t distribution
approaches the standard normal
distribution.
The t Test

 Example. Hourly Wage Equation

 Test: whether the return to exper, controlling for educ and tenure, is zero in the
population, against the alternative that it is positive.
 use WAGE1, clear
 reg lwage educ exper
tenure

Test against .
 t statistic

 Conclusion: The effect of experience on hourly wage is statistically greater than zero at the
5% (and even at the 1%) significance level.
The t Test

 The one-sided alternative that the parameter is

less than zero is
.
 Think of the rejection rule as
,
where c is the critical value for the alternative .
 For example, if the significance level is 5% and
the degrees of freedom is 18, then c = 1.734,
and so is rejected in favor of at the 5% level if
1.734.
 or display invttail(18, 0.05)
 The t Test

 Example. Student Performance and School Size

 Test: whether smaller school size leads to better student performance.
 use MEAP93, clear
 reg math10 totcomp staff
enroll

Test against .

 t statistic

 Conclusion: One cannot reject the hypothesis that there is no effect of school size on
student performance (not even for a lax significance level of 10%).
The t Test

 A more general case is to test the null

hypothesis against a two-sided alternative:

 When the alternative is two-sided, we are

interested in the absolute value of the t
statistic. The rejection rule for against is

 To find c, we again specify a significance level,

say 5%. For a two-tailed test, c is chosen to
make the area in each tail of the t distribution
an equal 2.5%. In other words, c is the 97.5th
percentile in the t distribution with n − k − 1
degrees of freedom. Assume df is 25:
 or display invttail(25, 0.025)
The t Test

 Example. Determinants of College GPA

 use GPA1, clear

 reg colGPA hsGPA ACT skipped
The t Test

 Generally, if the null is stated as

,
where is our hypothesized value of , then the appropriate t statistic is
The t Test

 Suppose that we wish to test the null hypothesis that a parameter is zero against a two-
sided alternative, and with 40 df we get a t statistic equal to 1.85. While and :
.
 Question: Given the observed value of the t statistic, what is the smallest significance level
at which the null hypothesis would be rejected?
 p-value: The smallest significance level at which the null hypothesis can be rejected.
Equivalently, the largest significance level at which the null hypothesis cannot be rejected.
 The t Test

 We obtain the actual p-value by computing the

probability that a t random variable, with 40 df,
is larger than 1.85 in absolute value.
 In general, the p-value for testing the null
hypothesis against the two-sided alternative is

where T denote a t distributed random variable

with n − k − 1 degrees of freedom and t denote
the numerical value of the test statistic.
 p-value
 or display ttail(40,1.85)

If the null hypothesis is true, we would observe an absolute value of

the t statistic as large as 1.85 about 7.2 percent of the time!
The t Test

 A small p-value is evidence against the null hypothesis because one would reject the null
hypothesis even at small significance levels.
 A large p-value is evidence in favor of the null hypothesis.
 Rejection rule: If denotes the significance level of the test, then is rejected if p-value .
 It is simple to obtain the one-sided p-value: just divide the two-sided p-value by 2.
The t Test

 The statistical significance of a variable is determined entirely by the size of , whereas the
economic significance or practical significance of a variable is related to the size (and sign)
of .
 Check for statistical significance. If the variable is statistically significant, discuss the
magnitude of the coefficient to get an idea of its practical or economic importance.
 If a variable is statistically and economically important but has the “wrong” sign, the
regression model might be misspecified.
 If a variable is not statistically significant at the usual levels (10%, 5%, or 1%), you might
still ask if the variable has the expected effect on y and whether that effect is practically
large. If it is large, you should compute a p-value for the t statistic. For small sample sizes,
you can sometimes make a case for p-values a little bit larger than usual levels. With large
p-values, the practically large estimates may be due to sampling error.
Confidence Intervals

 Confidence interval (CI): A rule used to construct a random interval so that a certain
percentage of all data sets, determined by the confidence level, yields an interval that
contains the population value.
 Using the fact that , simple manipulation leads to a CI for the unknown : e.g., a 95%
confidence interval is given by

where the constant c is the 97.5th percentile in a distribution.

 If random samples were obtained over and over again, with computed each time, the
(unknown) population value would lie in the interval for 95% of the samples.
 If the null hypothesis is , then is rejected against at (say) the 5% significance level if, and
only if, is not in the 95% confidence interval.
Confidence Intervals

 Example. Model of R&D Expenditures

 use RDCHEM, clear

 reg lrd lsales profmarg
.0128
Testing Single Linear Restriction

 Consider a simple model comparing the returns to education at junior colleges and four-
year colleges (“universities”):

 Hypothesis of interest: whether one year at a junior college is worth one year at a
university:
against
 Rewrite the null and alternative as and
Testing Single Linear Restriction

 : The standard error of the difference in parameters is difficult to obtain with standard
regression output

 An alternative method is to make a substitution in variables.

Testing Multiple Linear Restrictions:
The F Test
 Consider the following model that explains major league baseball players’ salaries:

 Test: once years in the league and games per year have been controlled for, the statistics
measuring performance - bavg, hrunsyr, and rbisyr - have no effect on salary.
against is not true.
 Exclusion restrictions: Restrictions which state that certain variables are excluded from the
model (or have zero population coefficients).
The F Test

 Example. Major League Baseball Players’ salaries (continued)

 use MLB1, clear

 reg lsalary years gamesyr bavg hrunsyr rbisyr
The F Test

 The model without the three variables in question is

.
 In the context of hypothesis testing, model above is the restricted model for testing ; the
original model is called the unrestricted model.
 Restricted model: In hypothesis testing, the model obtained after imposing all of the restrictions
required under the null.
 When we estimate the restricted model using the same data, we obtain

 reg lsalary years gamesyr

The F Test

 In general, write the unrestricted model with k independent variables as

;
 Suppose that we have q exclusion restrictions to test, the null hypothesis is then stated as
.
 When we impose the restrictions under , we are left with the restricted model:

 The F statistic is defined by

 The F Test

 Critical value c depends on our chosen

significance level, q and n – k – 1.
 Once c has been obtained, we reject H0
in favor of H1 at the chosen significance
level if

 If H0 is rejected, then we say that are

jointly statistically significant at the
appropriate significance level.

 Assume and , with a 5% significance level, the critical value or display

invFtail(3,60,0.05)
The F Test

 Example. Major League Baseball Players’ salaries (continued)

.
;
 or display invFtail(3,347,0.05)

: the null hypothesis is overwhelmingly rejected (even at very small significance levels).
 A more convenient way to conduct F test in STATA is using the command test:
 test bavg hrunsyr rbisyr
 Discussion:
 The three variables are “jointly significant”.
 They were not significant when tested individually.
 The likely reason is multicollinearity between them.
The F Test

 Relationship between F and t Statistics

 What happens if we apply the F statistic to the case of testing significance of a single
independent variable?
 Take the null to be and to test the single exclusion restriction that has no significant
partial effect on .
 It can be shown that the F statistic for testing exclusion of a single variable is equal to the
square of the corresponding t statistic.

 Because has an distribution, the two approaches lead to exactly the same outcome,
provided that the alternative is two-sided.
The F Test

 The R-Squared Form of the F Statistic

 Using the fact that and , we can substitute and obtain

 As a general rule, the SSR form of the F statistic should be used if a different dependent
variable is needed in running the restricted regression.
The F Test

 Computing p-Values for F Tests

 In the F testing context, the p-value is defined as
p-value ,
where we let ℱ denote an F random variable with degrees of freedom, and F is the
actual value of the test statistic.
 p-value is the probability of observing a value of F at least as large as we did, given that
the null hypothesis is true.
 A small p-value is evidence against H0.
Summary

 Multiple Regression Analysis: Inference

 Sampling Distributions of the OLS Estimators
 Testing Hypotheses about a Single Population Parameter: The t Test
 Testing against One-Sided Alternatives
 Two-Sided Alternatives
 Computing p-Values for t Tests
 Confidence Intervals
 Testing Hypotheses about a Single Linear Combination of the Parameters
 Testing Multiple Linear Restrictions: The F Test
 Testing Exclusion Restrictions
 Relationship between F and t Statistics