Chapter 3

Multiple Regression Model

(con’t)

1
Typical pitfalls in multiple regression

• To get motivated for the necessity of MLR

assumptions, let’s meet some “diseaseas” in OLS:
• Perfect collinearity
• Omitted variable
• Heteroskedasticity

2
Perfect collinearity - I

Definition:
F In the sample, one of the regressors is an exact linear function of the other
regressors.
F Namely,the case that λ1 X1i + λ2 X2i + · · · + λk Xki = 0 can happen for
some constant λj s, j = 1..k, and all i in the sample.

3
Perfect collinearity -II
• Some examples :
1) You include some regressors twice.
2) Second example: regress Wage on a constant, D, and
B, where: Di = 1 if Edu ≤ 12, = 0 otherwise; Bi = 1 if
Edu >12, = 0 otherwise, so Bi = 1 – Di and there is
perfect multicollinearity
3) Infamous “dummy variable trap”…
• 2) is actually a case of 3) and both can be reduced
to 1)

4
Perfect collinearity -III
• Suppose you have a set of multiple binary (dummy) variables, which
are mutually exclusive and exhaustive – that is, there are multiple
categories and every observation falls in one and only one category
(male or female). If you include all these dummy variables and a
constant, you will have perfect multicollinearity – this is sometimes
called the dummy variable trap.
• Why is there perfect multicollinearity here? Intuitively, you can not
form certeris paribus analysis. Why?
• Solutions to the dummy variable trap:
• Omit one of the groups (e.g. Senior), or
• Omit the intercept
• The interpretations for the above solutions are different. We will
cover this in the future.

5
Perfect collinearity -IV
• Key implication: each regressor we add to a
multiple regression must contain new information
at the margin.
• Scope:
• It is an symptom of the sample data instead of
population.
• This only applies to linear relations among the
explanatory variables: including x and x2 is not an issue
of perfect collinearity.

6
Omitted variable - I
Assume that the population model is

y = β0 + β1 x1 + β2 x2 + u,

but we specify the relationship without x2 :

y = β0 + β1 x1 + u.

What will happen to the OLS estimator of β̂1 in this case?

7
Omitted variable - II
F It can be shown:
Pn
(xi1 − x̄1 )xi2
Eβ̂1 = β1 + β2 Pi=1
n 2
i=1 (xi1 − x̄1 )

F The higher magnitude of β2 , the more severe biasedness of β̂1 will be.
F The more correlation between regressors, the more severe biasedness of β̂1
will be. Why?
F How to further interpret the sign of biasedness in practice?

8
Omitted variable - III
F The fraction in the equation of last slide is just a simple regression coeffi-
cient if we regress x2 on x1 .
F If we denote this coefficient as d1 , we have:

Eβ̂1 − β1 = β2d1

F The left measures the biasedness and depends on both the relation between
y and x2 and interrelation among the regressors.
F If the left is
I positive: upward bias
I negative: downward bias
I zero: unbiased (very rare! When will it occur?)
F If Eβ̂1 is closer to zero than β1 , it is biased toward zero or attenuated.

9
Omitted variable - IV
• Examples of omitted variable from the wage
regression on education:
• English ability: the worker with English as a second
language plausibly is less competitive than native
speakers in US job market.
• Origin of living: immigrant communities tend to be less
affluent and thus have smaller school budgets.
• What is the direction of this bias?
• What does common sense suggest?
• If common sense fails you, there is a formula…
10
Omitted variable - V
• The case of omitted variable actually belongs to the
broader concept: misspecification:
• Overspecification: including irrelevant explanatory
variables. By definition, their population
coefficient are zero. Therefore, not much damage is
created except the increase in degree of freedom
and hence reduction in the precision of estimators
(this is covered soon).
• Underspecification: omitted variable. This is far
more hazardous as we have seen.
11
Heteroskedasticity
• The error u has variance which is not identical for
all combinations of the explanatory variables.
• The OLS estimates are still unbiased in this case,
but our estimates of their variances are not.

12
Pitfalls and MLR assumptions
• MLR assumptions needed for the unbiasedness of OLS
estimator are linearity of the population model, random
sample, zero conditional mean of the error, and the
absence of perfect collinearity.
• Extra assumption of homoscedasticity is required for the
unbiasedness in the estimation of sampling variance of
OLS estimates.
• Most of the MLR assumptions are similar to those of SLR.
Which are new? Why are they added in terms of the
pitfalls discussed? Can you find the correspondence
between each pitfall and MLR assumptions?
13
Variance of OLS estimators - I
σ2
Given homoscedasticity, we can derive: Varβ̂j = SSTj (1−R2 ) , j = 1, .., k,
j
where SSTj is the total variation in xj about its mean, and Rj is the R2 from
2

an auxiliary regression from xj on all the rest of x variables plus intercept term.
Therefore, the precision of estimation will be higher if
F the larger variation in the associated x variable
F the smaller amount of variable xj can be explained by other variables
through regression:
I Rj2 = 1, very bad (which pitfall?)
I Rj2 → 0, xj does have large marginal contribution to regression of depen-
dent variable besides those from other independent variables.
I Rj2 → 1, bad since it is hard to ”partial out” the effect of xj on y because it
does not look quite different from other explanatory variables from sample.
Which pitfall is related with the discussion of Rj2 above? How? How will
overspecification cause the change of sampling variances though they are benign
for point esstimates (how will Rj2 usually change here)?

14
 Variance of OLS estimators - II
F Like the case in SLR, we usually need a consistent estimator for σ2 :
Pn 2
û
i=1 i
s2 = ,
n−k−1
where the denominator is again a correction of degree of freedome(df )
(why? Wiki has a good interpretation about df... )
F This also suggests we need a sample size strictly larger than the number
of explanatory variables plus one.

15
Gauss-Markov theorem
• Among the class of linear, unbiased estimators of the
population regression function, OLS provides the best
estimators, in terms of minimum sampling variance: OLS
estimators are best linear unbiased estimators (BLUE).
• Linearity means that the estimator–the rule for computing
the estimates–can be written as a linear function of the data
y.
• We do not need stronger assumption about normal
distribution in the error term for this theorem; thereofore,
OLS is best in a very strong sense.

16