MFIN 514:
Multiple Regression,
Violations of OLS
Assumptions
Dr. Ryan Ratcliff
SCHOOL OF BUSINESS
Multiple Regression Model
Consider the case of two (or more) regressors:
Yi = β0 + β1X1i + β2X2i + ei, i = 1,…,n
Y is the dependent variable
X1, X2 are the two independent variables (regressors)
(Yi, X1i, X2i) denote the ith observation on Y, X1, and X2.
β0 = intercept
β1 = effect on Y of a change in X1, holding X2 constant
β2 = effect on Y of a change in X2, holding X1 constant
ei = the regression error (omitted factors)
With a few exceptions, most of what you know about simple
regression will generalize to this case with multiple regressors
SCHOOL OF BUSINESS
Omitted Variable Bias
• In our test scores example, we
found that test scores were
negatively correlated with
higher student teacher ratio
(STR).
• Everything that might affect
test scores that’s not STR is in
the error term.
• OLS assumes that the error is
uncorrelated with STR. If
there is something in the error
term that’s correlated with
STR, our estimate of β will be
biased.
SCHOOL OF BUSINESS
Omitted Variable Bias
• Districts with lower % Eng. Learners (PCT_EL) have higher test
scores AND lower STR (smaller classes)
• Do we find negative correlation between STR and Tests Scores
b/c STR is just a proxy for PCT_EL?
SCHOOL OF BUSINESS
Omitted Variable Bias: Math
TESTSCRi = a + b(STRi)+ d(EL_PCTi)+ei
This is the “correct” regression that accounts for both variables, and the b,d
coefficient have the usual “holding all else constant” interpretation.
EL_PCTi = c + g(STRi)+ui (STR, EL_PCT are correlated)
TESTSCRi = a + b(STRi)+ d[c + g(STRi)+ui ]+ei
TESTSCRi = (a+dc)+ (b+dg)STRi + (dui +ei)
If we just have STR, the estimated coefficient is actually (b+dg): a mix of
the coefficient we’re trying to estimate (b) and the indirect effect that
EL_PCT matters for TEST_SCR (d) and STR is correlated with this
omitted variable (g). The dg term is called “omitted variable bias.”
SCHOOL OF BUSINESS
Omitted Variable Bias: Intuition
TESTSCRi = a + b(STRi)+ d(EL_PCTi)+ei
EL_PCTi = c + g(STRi)+ui (STR, EL_PCT are correlated)
TESTSCRi = (a+dc)+ (b+dg)STRi + (dui +ei)
Our regression shows a negative coefficient on STR: bigger classes
predict lower test scores. However, if
1) big classes tend to have high shares of non-native speakers (g>0) AND
2) a high el_pct also predicts lower test scores (d<0), then
dg<0 will make our estimated coefficient in the STR regression look more
negative than the base effect, b – our estimate is biased. In some sense,
STR “gets credit” for some of EL_PCT’s negative effect on test scores.
SCHOOL OF BUSINESS
Cures for Omitted Variable Bias
Three ways to overcome omitted variable bias
1. Run a controlled experiment in which treatment (STR) is randomly
assigned: then PctEL is still a determinant of TestScore, but PctEL
is uncorrelated with STR (g=0). (This solution to OV bias is often
infeasible.)
2. Adopt the “cross tabulation” approach, with finer gradations of STR
and PctEL – within each group, all classes have the same PctEL,
so we “control for PctEL” (Common in Finance)
3. Use a regression in which the omitted variable (PctEL) is no longer
omitted: include PctEL as an additional regressor in a multiple
regression.
SCHOOL OF BUSINESS
CA Test Scores: Mult. Regression
-------------------------------------------------------------------------
| Robust
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
--------+----------------------------------------------------------------
str | -2.279808 .5194892 -4.39 0.000 -3.300945 -1.258671
_cons | 698.933 10.36436 67.44 0.000 678.5602 719.3057
-------------------------------------------------------------------------
------------------------------------------------------------------------------
| Robust
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
str | -1.101296 .4328472 -2.54 0.011 -1.95213 -.2504616
pctel | -.6497768 .0310318 -20.94 0.000 -.710775 -.5887786
_cons | 686.0322 8.728224 78.60 0.000 668.8754 703.189
------------------------------------------------------------------------------
• The t-test for the significance of an individual coefficient is the
same as before…
• Compare these two regressions, and relate these results to our
previous discussion of Omitted Variable Bias.
SCHOOL OF BUSINESS
Coef. Tests,Predictions Same as Before
� 𝐻𝐻𝐻𝐻
𝛽𝛽−𝛽𝛽 �
𝛽𝛽
t-statistic: 𝑡𝑡 = =
𝑆𝑆𝑆𝑆(𝛽𝛽) 𝑆𝑆𝑆𝑆(𝛽𝛽)
Conf Interval: {𝛽𝛽̂ ± 5% Crit Value×SE(𝛽𝛽)}
̂
Predicted Values:
------------------------------------------------------------------------------
| Robust
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
str | -1.101296 .4328472 -2.54 0.011 -1.95213 -.2504616
pctel | -.6497768 .0310318 -20.94 0.000 -.710775 -.5887786
_cons | 686.0322 8.728224 78.60 0.000 668.8754 703.189
------------------------------------------------------------------------------
TESTSCR Prediction = 686.03 – 1.10*STR – 0.6498 * PCTEL
SCHOOL OF BUSINESS
N – k : Degrees of Freedom
reg testscr str pctel, robust;
Regression with robust standard errors Number of obs = 420
F( 2, 417) = 223.82
Prob > F = 0.0000
R-squared = 0.4264
Root MSE = 14.464
------------------------------------------------------------------------------
| Robust
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
str | -1.101296 .4328472 -2.54 0.011 -1.95213 -.2504616
pctel | -.6497768 .0310318 -20.94 0.000 -.710775 -.5887786
_cons | 686.0322 8.728224 78.60 0.000 668.8754 703.189
------------------------------------------------------------------------------
• A number of tests (esp. ANOVA) will require you to know Sample Size
(N), and # of regressors beyond constant (K)
• Here, N = 420, and K = 2. Several formulae (eg Fstat) will contain a
“degrees of freedom” correction N – K – 1 (-1 is for the constant).
SCHOOL OF BUSINESS
N –k: SER and RMSE
As in regression with a single regressor, the Std.
Error of the Regression and the Root Mean-Sq.
Error are measures of the spread of the Ys around
the regression line (Std. Dev. of Errors):
1 n
SER = ∑
n − k − 1 i =1
ˆ
ui
2 Degrees of Freedom
Correction
1 n 2
RMSE = ∑
n i =1
uˆi No Correction
SCHOOL OF BUSINESS
ANOVA: Same, except N - k
Source of Sum of
Df Mean Square
Variation Squares
Regression
k RSS MSR = RSS/k
(explained)
Error
n–k–1 SSE MSE=SSE/(n-k-1)
(unexplained)
Total n–1 SST
𝑅𝑅𝑅𝑅𝑅𝑅�
2 explained variation RSS 𝑘𝑘 𝑀𝑀𝑀𝑀𝑀𝑀
R = = 𝐹𝐹 =
𝑆𝑆𝑆𝑆𝑆𝑆
=
𝑀𝑀𝑀𝑀𝑀𝑀
total variation SST 𝑛𝑛 − 𝑘𝑘 − 1
SCHOOL OF BUSINESS
R2 vs. Adjusted R2
Recall R2 = RSS / SST = 1 – SSE / SST.
This formulation has the annoying feature that R2 always
increases when we add variables to the regression, even if
they are insignificant.
Adj. R2 uses the N – k logic to apply a penalty to including
another variable.
𝑛𝑛−1 𝑆𝑆𝑆𝑆𝑆𝑆 𝑛𝑛−1
Adj R2= 1 − =1− (1 − 𝑅𝑅2 )
𝑛𝑛−𝑘𝑘−1 𝑆𝑆𝑆𝑆𝑆𝑆 𝑛𝑛−𝑘𝑘−1
If SSE doesn’t go down enough, the benefit of the new
variables does not exceed the cost, so Adj. R2 won’t increase.
SCHOOL OF BUSINESS
CFA 2 Questions on Mult. Regression
Standard Error of the
Variable Coefficient t-statistic p-value
Coefficient
Intercept 0.043 0.01159 3.71 < 0.001
Ln(No. of Analysts) −0.027 0.00466 −5.80 < 0.001
Ln(Market Value) 0.006 0.00271 2.21 0.028
Degrees of Freedom Sum of Squares Mean Square
Regression 2 0.103 0.051
Residual 194 0.559 0.003
Total 196 0.662
Dave Turner is a security analyst who is using regression analysis to determine how well two
factors explain returns for common stocks. The independent variables are the natural logarithm
of the number of analysts following the companies, Ln(no. of analysts), and the natural
logarithm of the market value of the companies, Ln(market value). The regression output
generated from a statistical program is given in the following tables. Each p-value corresponds
to a two-tail test.
Turner plans to use the result in the analysis of two investments. WLK Corp. has twelve analysts
following it and a market capitalization of $2.33 billion. NGR Corp. has two analysts following it
and a market capitalization of $47 million.
SCHOOL OF BUSINESS
CFA 2 Questions on Mult. Regression
Standard Error of the
Variable Coefficient t-statistic p-value
Coefficient
Intercept 0.043 0.01159 3.71 < 0.001
Ln(No. of Analysts) −0.027 0.00466 −5.80 < 0.001
Ln(Market Value) 0.006 0.00271 2.21 0.028
Degrees of Freedom Sum of Squares Mean Square
Regression 2 0.103 0.051
Residual 194 0.559 0.003
Total 196 0.662
The 95% confidence interval (use a t-stat of 1.96 for this question only) of the
estimated coefficient for the independent variable Ln(Market Value) is closest to:
A) 0.011 to 0.001
B) 0.014 to -0.009
C) -0.018 to -0.036
SCHOOL OF BUSINESS
CFA 2 Questions on Mult. Regression
Standard Error of the
Variable Coefficient t-statistic p-value
Coefficient
Intercept 0.043 0.01159 3.71 < 0.001
Ln(No. of Analysts) −0.027 0.00466 −5.80 < 0.001
Ln(Market Value) 0.006 0.00271 2.21 0.028
Degrees of Freedom Sum of Squares Mean Square
Regression 2 0.103 0.051
Residual 194 0.559 0.003
Total 196 0.662
NGR Corp. has two analysts following it and a market capitalization of $47
million. If the number of analysts on NGR Corp. were to double to 4, the change
in the forecast of NGR would be closest to?
A) −0.019.
B) −0.035.
C) −0.055.
SCHOOL OF BUSINESS
CFA 2 Questions on Mult. Regression
Standard Error of the
Variable Coefficient t-statistic p-value
Coefficient
Intercept 0.043 0.01159 3.71 < 0.001
Ln(No. of Analysts) −0.027 0.00466 −5.80 < 0.001
Ln(Market Value) 0.006 0.00271 2.21 0.028
Degrees of Freedom Sum of Squares Mean Square
Regression 2 0.103 0.051
Residual 194 0.559 0.003
Total 196 0.662
Based on a R2 calculated from the information in Table 2, the analyst should
conclude that the number of analysts and ln(market value) of the firm explain:
A) 84.4% of the variation in returns.
B) 18.4% of the variation in returns.
C) 15.6% of the variation in returns.
SCHOOL OF BUSINESS
Model Interpretation
Big picture, across model comments
Across all variations, STR has a statistically
significant, negative relationship with test
scores.
Which model seems best?
Both STR and PCT_EL change magnitudes
depending on the presence of other
controls. Model 3 has highest Adj. R2 /
lowest SER and seems to best address
omitted variable bias.
Econ. interp. / sig of coeffs
In Model 3, a 1 student increase in average
class size predicts a 1 point drop in test
scores. Since average class sizes vary by
more than 20 students across the sample,
these differences matter on 800 point test
SCHOOL OF BUSINESS
Specification Tricks: Dummy Vars.
A dummy variable is a 0 or 1 variable that groups the data
into categories:
• ACTION = 1 if this was an action movie
• SEQUEL = 1 if this movie is a sequel
SCHOOL OF BUSINESS
Specification Tricks: Dummy Vars.
To interpret the dummy, write out the pred. eqtn by category.
Overall, it’s
BOX = $5,672,516 + 236,527* BUDGET – 2,807,283*ACTION + …
For a new comedy (ACTION, SEQUEL, HORROR = 0), our prediction
is BOX = $5,672,516 + 236,527* BUDGET – 2,807,283*(0)
For a new action movie: BOX = $5,672,516 + 236,527* BUDGET –
2,807,283*(1) = $2,865,233 + 236,527* BUDGET
SCHOOL OF BUSINESS
Specification Tricks: Dummy Vars.
For a new action movie: BOX = $5,672,516 + 236,527* BUDGET –
2,807,283*(1) = $2,865,233 + 236,527* BUDGET
When the dummy appears alone, it is an intercept shifter: the
prediction is box office will be lower for an action movie with the
same budget as a non-action movie (?!)
However, significance matters here: the weak t-stat (p-value 53%)
says that we can’t reject the null that box office for action movies is
no different than other movies.
SCHOOL OF BUSINESS
Specification Tricks: Dummy Vars.
What if we had the idea that action movies generate more box office
per dollar of budget – a different slope.
By multiplying the variable by the appropriate dummy, we can model
this difference in slope:
BOX = a + c*ACTION + b*BUDGET + d*(BUDGET*ACTION) +…
Not Action (ACTION=0)
BOX = a + c*0 + b*BUDGET + d*(BUDGET*0) +.. = a + b*BUDGET…
Action Movie (ACTION=1)
BOX = a + c*1 + b*BUDGET + d*(BUDGET*1) = (a+c) + (b+d)*BUDGET …
d is difference in slope for action movies; c is the different intercept.
SCHOOL OF BUSINESS
Specification Tricks: Dummy Vars.
How do you interpret these results?
SCHOOL OF BUSINESS
Specification Tricks: Logs
Lots of regression specifications use logs:
I. linear-log Yi = β0 + β1ln(Xi) + ui
II. log-linear ln(Yi) = β0 + β1Xi + ui
III. log-log ln(Yi) = β0 + β1ln(Xi) + ui
There are two main reasons to use logs:
1) It’s an easy cure for skewed/heteroskedastic data
2) Coefficients on logs give percentage changes
(elasticities)
SCHOOL OF BUSINESS
Logs: Skew / Heteroskedasticity
Lots of size type variables in finance are very skewed, which will distort OLS.
Taking the log of data like this gives a more normal distribution.
Current Assets ln(Current Assets)
4.0e-04
.3
3.0e-04 4.0e-04
.3
2.0e-04 3.0e-04
.2 .2
Density
Density
Density
Density
1.0e-04 2.0e-04
.1 .1
1.0e-04 0
0 20000 40000 60000 2 4 6 8 10
Current Assets - Total l_act
0
0 20000 40000 60000 2 4 6 8 10
Current Assets - Total l_act
SCHOOL OF BUSINESS
Specification Tricks: Logs
I. linear-log Yi = β0 + β1ln(Xi) + ui
1% change in X β1 unit change in Y
II. log-linear ln(Yi) = β0 + β1Xi + ui
1 unit change in X β1% change in Y
III. log-log ln(Yi) = β0 + β1ln(Xi) + ui
1% change in X β1% change in Y
Important: You can’t compare SER, R2, etc.
across a model of Y vs. ln(Y) – different units.
SCHOOL OF BUSINESS
Violations of Regression Assumptions
Regression Assumption Condition if Violated
Error term has constant Heteroskedasticity
variance.
Error terms are not Serial correlation
correlated with each other. (autocorrelation)
No exact linear relationship Multicollinearity
among “X” variables.
Define it, Explain its effect on OLS
Detect it, Correct for it
SCHOOL OF BUSINESS
Heteroskedasticity
Type 1: Unconditional heteroskedasticity – doesn’t matter
Type 2: Conditional heteroskedasticity
• Related to independent variables (next slide)
• This IS a problem
• Impact: t-stats are usually artificially high
Not affected
Too small in OLS
• Standard error too low = t-stat too high; Type I errors
SCHOOL OF BUSINESS
Conditional Heteroskedasticity
Y Low residual
variance
High residual
variance
0 X
Detection: Scatter diagrams can show when error
variance changes systematically with an X variable.
SCHOOL OF BUSINESS
Conditional Heteroskedasticity
Breusch-Pagan test: Regress squared
residuals on “X” variables.
• Point: Test significance of resulting R2 (do the independent
variables explain a significant part of the variation in
squared residuals?)
• H0: No heteroskedasticity
• Chi-square test: BP = Rresid2 × n (with k df)
Name Drop: B-P Test detects heteroskedasticity
SCHOOL OF BUSINESS
Correcting for Heteroskedasticity
First Method: Use STATA “robust” standard
errors (Huber-White standard errors).
Result: Relative to OLS, standard errors
higher, t-stats lower, and conclusions more
accurate
Second Method: Use generalized least
squares, modifying original equation to
eliminate heteroscedasticity (not on CFA 2).
SCHOOL OF BUSINESS
Serial Correlation: Definition
Positive autocorrelation: Each error term is
positively correlated w/ previous error.
• Common in financial time series data; not as common for
cross-sectional data.
Same problems as heteroskedasticity
• OLS t-stats are too high (Type I errors)
Again: False
significance
Not affected
Too small in OLS
SCHOOL OF BUSINESS
Serial Correlation: Detection
Residual Plots – clusters of +/- errors
Durbin-Watson statistic
DW ≅ 2(1 – r)
Three cases: No correlation, positive correlation, and
negative correlation
• No autocorrelation (ρ = 0)
• DW ≅ 2(1 – 0) = 2
• Positive serial correlation (ρ = 1)
• DW ≅ 2(1 – 1) = 0
• Negative serial correlation (ρ = –1)
• DW ≅ 2(1 – (– 1) ) = 4
SCHOOL OF BUSINESS
Serial Correlation: Correction
Preferred method: Use HAC Std. Errors
• Hansen or Newey West Heteroskedastcity and
Autocorrelation Consistent Std. Errors are bigger than OLS
errors, which offsets OLS tendency to over-reject H0.
• Some gymnastics required to implement in STATA
Alternative: Quasi-Differencing
• Old school: transform data with an estimate of the
correlation between errors so that the new data is not
serially correlated.
SCHOOL OF BUSINESS
Multicollinearity
Define: Two or more “X” variables are strongly
correlated with each other
Intuition: X1 and X2 strongly correlated: hard
to estimate effect of changing X1 when X2 is
held constant.
Effects:
• Inflates OLS SEs; reduces OLS t-stats; increases chance of
Type II “should reject but don’t” errors
• Point: t-stats artificially small so variables falsely look
unimportant
SCHOOL OF BUSINESS
Multicollinearity: Detection & Correction
Observation 1: Significant F-stat (and high R2),
but all t-stats insignificant
Observation 2: High correlation between “X”
variables (more complicated for k>2)
Correction
• Omit one or more of the correlated “X” variables
SCHOOL OF BUSINESS
Perfect Multicollinearity: Dummy Trap
Suppose your data can be perfectly sorted into 2 (or more)
categories: e.g. USD Alums and others
If you include a USD Alum and Not USD Alum dummy together,
then summing the Alum + Not variables will equal 1 for every
observation, which is identical to the constant – “perfect
multicollinearity”
Correction
• Estimate the constant and leave out one dummy: constant is
the intercept for the omitted dummy, other dummies are
deviations from that intercept.
• No constant and all the dummies: each dummy is an intercept.
SCHOOL OF BUSINESS
Summarizing Problems & Fixes
Conditional
Violation Serial Correlation Multicollinearity
Heteroskedasticity
Residual variance related Residuals are Two or more X’s are
What is it?
to level of X’s correlated correlated
Effect? Type I errors (overreject) Type I errors Type II errors
Breusch-Pagan Conflicting t and F
Detection? Durbin-Watson test
chi-square test statistics
Hansen / Newey
Use White-corrected Drop one of the correlated
Correction? West standard errors
standard errors variables
SCHOOL OF BUSINESS
Key Concepts for CFA 2
• Regression: Output, Hypo Test, Conf. Int.
• ANOVA table
• OLS Problems: Detect, Effects, Cures
SCHOOL OF BUSINESS
