Variable Selection: Prof. Sharyn O'Halloran Sustainable Development U9611 Econometrics II

Lecture 6: Variable Selection
Prof. Sharyn OHalloran Sustainable Development U9611 Econometrics II
Regression Diagnostics: Review

After estimating a model, we want to check the entire regression for:

Normality
of the residuals Omitted and unnecessary variables Heteroskedasticity

We also want to test individual variables for:

Outliers Leverage Influence Collinearity Functional
form
Spring 2005 2
U9611
Look at Residuals: rvfplot

First, examine the ^ residuals ei vs. Y. Any pattern in the residuals indicates a problem. Here, there is an obvious U-shape & heteroskedasticity.
3
reg price weight mpg forXmpg foreign rvfplot, plotr(lcol(black)) yline(0)

U9611 Spring 2005
Check Residuals for Normality

Histo gram B ox Plot
Density Residuals 0 5000 10000
-5000
Distance above median 2000 4000 6000 8000
Symmetry Plo t
Qu antile-Normal Plo t
1000
2000
3000
Residuals -4000
-2000
2000
4000
Residual plots also indicate non-normality

U9611 Spring 2005 4
Stata Commands: imtest

. imtest Cameron & Trivedi's decomposition of IM-test --------------------------------------------------Source | chi2 df p ---------------------+----------------------------Heteroskedasticity | 18.86 10 0.0420 Skewness | 11.69 4 0.0198 Kurtosis | 2.33 1 0.1273 ---------------------+----------------------------Total | 32.87 15 0.0049 ---------------------------------------------------
The command imtest stands for information matrix. Here, we see heteroskedasticity and skewness.
U9611 Spring 2005 5
Omitted Variables
Omitted variables are variables that significantly influence Y and so should be in the model, but are excluded. Questions: Why are omitted variables a problem? How can we test for them? What are the possible fixes?
Lets
U9611
check the Venn diagram

Spring 2005 6
Omitted Variables
Y Y is determined by X and W, but we omit W from the regression. Here, the impact of X on Y is area 1, and the impact of W is area 3. W
Spring 2005 7
1 2
X
U9611
Omitted Variables
Y By omitting W, we now estimate the impact of X on Y by areas 1 and 2, rather than just area 1.
1 2
X
U9611 Spring 2005 8
Omitted Variables
Y
2
1 2
X
U9611 Spring 2005
This means that: 1. The estimate of 1 is biased (since area 2 actually belongs to W as well as X). 2. The variance of 1 is reduced (since its estimated by areas 1 and 2). 3. The unexplained variance for Y (2) increases.
9
Omitted Variables
This is only a representation of the variance in Y, not the variance itself. It is but a pale imitation of the actual variance; dont be confused!!
1 2
X
U9611 Spring 2005
This means that: 1. The estimate of 1 is biased (since area 2 actually belongs to W as well as X). 2. The variance of 1 is reduced (since its estimated by areas 1 and 2). 3. The unexplained variance for Y (2) increases.
10
Omitted Variables
This is only a representation of the variance in Y, not the variance itself. It is but a pale imitation of the actual variance; dont be confused!! Dont take this too literally! It would be a grievous error! 1 2
X
U9611 Spring 2005
This means that: 1. The estimate of X is biased (since area 2 actually belongs to W as well as X). 2. The variance of X is reduced (since its estimated by areas 1 and 2). 3. The unexplained variance for Y (2) increases.
11
Stata Command: ovtest

. ovtest Ramsey RESET test using powers of the fitted values of price Ho: model has no omitted variables F(3, 66) = 7.77 Prob > F = 0.0002 . hettest Breusch-Pagan / Cook-Weisberg test for heteroskedasticity Ho: Constant variance Variables: fitted values of price chi2(1) Prob > chi2
U9611
= =
6.50 0.0108
Spring 2005 12

U9611
Evidence of omitted vars
= =
6.50 0.0108
Spring 2005 13

U9611
Evidence of omitted vars and non-constant variance, as before
= =
6.50 0.0108
Spring 2005 14
Including Unnecessary Variables

Y Here, variable W adds little on its own to explaining variation in Y (area 3).
3
1 2
X
U9611
W
Spring 2005
Any explanatory power is due to its correlation with X (area 2).

15

Y
2
1 2 3
X
U9611
W
Spring 2005
This means that: 1. The estimate of X is unbiased (since area 2 actually belongs only to X). 2. The variance of X is increased (since area 2 is removed). 3. The unexplained variance for Y (2) is essentially the same (since area 3 is so small).
16

Y
Solution: omit W from the regression. This is why we remove insignificant variables from regression equations.
3
2
1 2
X
U9611
W
Spring 2005
Note: This is similar to multicollinearity: the more variables added to the model, the more uncertainty there is in estimating X.
17
Checking Individual Variables

If the diagnostics on the regression as a whole show potential problems, move to

Checking observations for:

Leverage Outliers Influence
Analyzing the contributions of individual variables to the regression:

Avplots Cprplots
U9611
Spring 2005
18
Diagnostic Plots: lvr2plot
reg price weight mpg forXmpg foreign lvr2plot, mlab(make) mlabp(0) m(none) mlabsize(small)
U9611 Spring 2005 19

Worrisome Points
U9611 Spring 2005 20

Problem: Only diesel in sample Fix: Could omit
U9611 Spring 2005 21

Problem: Data entered incorrectly Fix: Recode!
U9611 Spring 2005 22

Still no explanation
U9611 Spring 2005 23

Still no explanation Lets look at these points variable-by variable.
U9611 Spring 2005 24
Stata Commands: avplot

Say the original model is: Y = 0 + 1x1 + 2x2 + 3x3 We would like to graph the relation between Y and a single regressor x1.
Cant do this directly, as we can with only one independent variable (too many dimensions). There is a 1-to-1 correspondence btwn. Yi & x1i. A regression of Y on x1 has the same slope and standard error as in the multiple regression. The outlierliness of each observation is preserved.
Spring 2005 25
Added variable plots have the property that:

1. 2. 3.
U9611
Stata Commands: avplot

To obtain the avplot for x1:

1. 2. 3.
Regress Y on x2 and x3 and calculate the residual; call this e(Y|x2,x3) Regress x1 on x2 and x3 and calculate the residual; call this e(x1|x2,x3) The avplot is then e(Y|x2,x3) vs. e(x1|x2,x3)

U9611
The avplot thus provides a view of the relationship between Y and x1 with the effects of x2 and x3 taken out of both. The slope coefficient in the avplot is the same as in the multiple regression. Why?
Spring 2005 26
Example: Two Variables

Y Regress Y on just W first and take the residual. This takes out areas 2 and 3
1 2
X
U9611 Spring 2005
W
27

Y Regress Y on just W first and take the residual. This takes out areas 2 and 3 Note: estimate of W will be biased. W
Spring 2005 28
1 2
X
U9611

Y Now regress X on W and take the residual.
1 2
X
U9611 Spring 2005
W
29

Y Now regress X on W and take the residual. This takes out area 4 as well.
1 2
X
U9611 Spring 2005
W
30

Y In the resulting figure, the overlap of Y and X is area 1, just as in the original multivariate regression! Thats why we get the same coefficient W
Spring 2005 31
1 2
X
U9611
- Is the state with largest expenditure influential? - Is there an association of expend and SAT, after accounting for takers?
Added variable plots: example
U9611
Spring 2005
32
Alaska is unusual in its expenditure, and is apparently quite influential
U9611
Spring 2005
33
X-axis: residuals after regression expendi = b0 + b1*takersi Y-axis: residuals after regression SAT b + b1*takersi i = Spring 20050 U9611
34
After accounting for % of students who take SAT, there is a positive association between expenditure and mean SAT scores. 35 Spring 2005 U9611
Component Plus Residual Plots

Wed like to plot y versus x2 but with the effect of x1 subtracted out; i.e. plot
y 0 1 x1
versus x2
To calculate this, get the partial residual for x2: a. Estimate
0 , 1 , and 2
in
y = 0 + 1 x1 + 2 x2 +
y 0 1 x1 = 2 x2 + i
b. Use these results to calculate c. Plot this quantity vs. x2.

Whereas the avplots are better for detecting outliers, cprplots are better for determining functional form.
Spring 2005 36
U9611
Graph cprplot x1
ei + b1 x1i
Graphs each observation's residual plus its component predicted from x1 against values of x1i
U9611
Spring 2005
37
Graph cprplot x1
ei + b1 x1i
Here, the relationship looks fairly linear, although Alaska still has lots of influence.
U9611
Spring 2005
38
Regression Fixes
If you detect possible problems with your initial regression, you can:
1. 2. 3. 4. 5.
Check for mis-coded data Divide your sample or eliminate some observations (like diesel cars) Try adding more covariates if the ovtest turns out positive Change the functional form on Y or one of the regressors Use robust regression
Spring 2005 39
U9611
Robust Regression
This is a variant on linear regression that downplays the influence of outliers

1. 2. 3. 4.
First performs the original OLS regression Drops observations with Cooks distance > 1 Calculates weights for each observation based on their residuals Performs weighted least squares regression using these weights

U9611
Stata command: rreg instead of reg

Spring 2005 40
Robust Regression: Example

. reg mpg weight foreign Source | SS df MS -------------+-----------------------------Model | 1619.2877 2 809.643849 Residual | 824.171761 71 11.608053 -------------+-----------------------------Total | 2443.45946 73 33.4720474 Number of obs F( 2, 71) Prob > F R-squared Adj R-squared Root MSE = = = = = = 74 69.75 0.0000 0.6627 0.6532 3.4071
-----------------------------------------------------------------------------mpg | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------weight | -.0065879 .0006371 -10.34 0.000 -.0078583 -.0053175 foreign | -1.650029 1.075994 -1.53 0.130 -3.7955 .4954422 _cons | 41.6797 2.165547 19.25 0.000 37.36172 45.99768 ------------------------------------------------------------------------------

U9611
This is the original regression Run the usual diagnostics

Spring 2005 41

Residual vs. Fitted Plot
15 -5 10 0 Residuals 5 10
15
20 Fitted valu es
25
30
U9611
Spring 2005
42

. reg mpg weight foreign Source | SS df MS -------------+-----------------------------Model | 1619.2877 2 809.643849 Residual | 824.171761 71 11.608053 -------------+-----------------------------Total | 2443.45946 73 33.4720474 Number of obs F( 2, 71) Prob > F R-squared Adj R-squared Root MSE = = = = = = 74 69.75 0.0000 0.6627 0.6532 3.4071
-----------------------------------------------------------------------------mpg | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------weight | -.0065879 .0006371 -10.34 0.000 -.0078583 -.0053175 foreign | -1.650029 1.075994 -1.53 0.130 -3.7955 .4954422 _cons | 41.6797 2.165547 19.25 0.000 37.36172 45.99768 ------------------------------------------------------------------------------

U9611
rvfplot shows heterskedasticity Also, fails a hettest

Spring 2005 43

. rreg mpg weight foreign, genwt(w) Huber Huber Huber Huber Biweight Biweight Biweight Biweight iteration iteration iteration iteration iteration iteration iteration iteration 1: 2: 3: 4: 5: 6: 7: 8: maximum maximum maximum maximum maximum maximum maximum maximum difference difference difference difference difference difference difference difference in in in in in in in in weights weights weights weights weights weights weights weights = = = = = = = = .80280176 .2915438 .08911171 .02697328 .29186818 .11988101 .03315872 .00721325 Number of obs = F( 2, 71) = Prob > F = 74 168.32 0.0000
Robust regression estimates
-----------------------------------------------------------------------------mpg | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------weight | -.0063976 .0003718 -17.21 0.000 -.007139 -.0056562 foreign | -3.182639 .627964 -5.07 0.000 -4.434763 -1.930514 _cons | 40.64022 1.263841 32.16 0.000 38.1202 43.16025 ------------------------------------------------------------------------------
U9611
Spring 2005
44


U9611
Note: Coefficient on foreign changes from -1.65 to -3.18

Spring 2005 45


U9611
This command saves the weights generated by rreg

Spring 2005 46

. sort w . list make mpg weight w if w<.467, sep(0) +-------------------------------------------+ | make mpg weight w | |-------------------------------------------| | Subaru 35 2,050 0 | | VW Diesel 41 2,040 0 | | Datsun 210 35 2,020 0 | | Plym. Arrow 28 3,260 .04429567 | | Cad. Seville 21 4,290 .08241943 | | Toyota Corolla 31 2,200 .10443129 | | Olds 98 21 4,060 .28141296 | +-------------------------------------------+
1. 2. 3. 4. 5. 6. 7.
This shows that three observations were dropped by the rreg, including the VW Diesel
Spring 2005 47
U9611
Theories, Tests and Models

Question: Which variables should we include on the RHS of our estimation? This is a fundamental question of research design and testing.
NOT merely a mechanical process.
So lets first review some basic concepts of theory-driven research.

Theory-driven = based on a model of the phenomenon of interest, formal or otherwise. We always do this, somehow, in our research
Spring 2005 48
U9611

IV1 IV2 IV3 Dep. Var.
Say we have a theory predicting IV1, IV2, and IV3 all affect dependent variable Y.
Spring 2005 49
U9611

IV1 IV2 IV3 Model
Dep. Var.
Spring 2005 50
U9611

IV1 IV2 IV3 Model

Data-Generating Process (DGP)
Dep. Var.
Spring 2005 51
U9611

IV1 IV2 IV3
Dep. Var.
For instance:
U9611
Spring 2005
52

C IV2 IV3
Dep. Var.
For instance:
1.
Congressional Committee ideal points,
U9611
Spring 2005
53

C P IV3
Dep. Var.
For instance:
1. 2.
Congressional Committee ideal points, The Presidents ideal point, and
U9611
Spring 2005
54

C P U
Dep. Var.
For instance:
1. 2. 3.
Congressional Committee ideal points, The Presidents ideal point, and Uncertainty in the policy area
Spring 2005 55
U9611

C P U
Delegation
For instance:
Congressional Committee ideal points, 2. The Presidents ideal point, and 3. Uncertainty in the policy area Affect Delegation to the executive.
1.
U9611
Spring 2005
56

C P U
Delegation
Question: What if another variable is suggested that might also impact Y?
U9611
Spring 2005
57

C P U I?

U9611
Delegation
Question: What if another variable is suggested that might also impact Y? For instance, the Interest group environment surrounding the issue.
Spring 2005 58

Remember: our original regression variables came from a particular model of our subject. There are, generally, three options when new variables are suggested:
1. 2. 3.
Re-solve the model with the new variable(s) included as well; Assume the model is a complete Data Generating Process (DGP) and ignore other potential factors; Treat the model as a partial DGP.

U9611
We will consider each in turn.

Spring 2005 59
Add New Variables to the Model

We could expand our model to include the new variable(s).

Formal models: re-solve the equilibrium Qualitative models: re-evaluate the predicted effects There will always be more factors that might affect the phenomenon of interest Dont want to take the position that you cant have a theory of anything without having a theory of everything
Spring 2005 60
In the long run, though, this is not feasible

U9611
Treat the Model as a Complete DGP

This means to just look at the impact of the variables suggested by your theory on the dependent variable This is unrealistic, but an important first cut at the problem You can do this with

Naturally occuring data Experimental data

U9611
Either way, this is a direct test of the theory

Spring 2005 61
Treat the Model as a Partial DGP

This is the modal response just add the new variables to the estimation model But it has some shortcomings:

You lose the advantages of modelling Youre not directly testing your theory any more, but rather your theory plus some conjectures. Have to be careful about how the error term enters the equation where does come from?

U9611
If you do this, best to use variables already known to affect the dependent variable.
Spring 2005 62
How To Handle Many Regressors

With this in mind, how can we handle a situation where we have many potential indep. variables? Two good reasons for seeking a subset of these:

General principle: smaller is better (Occams razor) Unnecessary terms add imprecision to inferences Fit of all possible models (include or exclude each X) Compare with these statistics:
Computer assisted tools

Cp, AIC, or BIC
Stepwise regression (search along favorable directions)
But dont expect a BEST or a TRUE model or a law of nature

Spring 2005 63
U9611
Objectives when there are many Xs

(12.2.1)
Assessment of one X, after accounting for

many others

Ex: Do males receive higher salaries than females, after accounting for legitimate determinants of salary? Strategy: first find a good set of Xs to explain salary; then see if the sex indicator is significant when added
Fishing for association; i.e. what are the

important Xs?
The trouble with this: we can find several subsets of Xs that explain Y; but that doesnt imply importance or causation Best attitude: use this for hypothesis generation This is a straightforward objective Find a useful set of Xs; no interpretation required
Spring 2005 64
Prediction

U9611
Loss of precision due to multicollinearity

Review: variance of L.S. estimator of slope in simple reg. =
Variance about the regression
2 2 x
(n 1) s
Sample variance of X
Fact: variance of L.S. estimator of coef. of Xj in mult. reg. =
Sample variance of Xj
(n 1) s (1 R )
2 j 2 j
Spring 2005
R2 in the regression of Xj on the other Xs in model
U9611
65
Implications of Multicollinearity
So variance of an estimated coef. will tend to be larger if there are other Xs in the model that can predict Xj. The S.E. of prediction will also tend to be larger if there are unnecessary or redundant Xs in the model. The tradeoff for adding more regressors is:

You
explain more of the variance in Y, but You estimate the impact of all other variables on Y using less information
U9611 Spring 2005 66
Implications of Multicollinearity Multicollinearity:

2 The situation in which s 2 is small for one ( 1 R j j ) or more js (usually characterized by highly correlated Xs).
Strategy:
There isnt a real need to decide whether multicollinearity is or isnt present, as long as one tries to find a subset of xs that adequately explains
(Y), without redundancies.
Good subsets of xs:

U9611
2 (a) lead to a small
(b) with as few xs as possible (Criteria Cp, AIC, and BIC formalize this)
Spring 2005 67
Strategy for dealing with many Xs

1. Identify objectives; identify relevant set of Xs 2. Exploration:
3. Resolve transformation and influence before variable selection 4. Computer-assisted variable selection:
matrix of scatterplots; correlation matrix; Residual plots after fitting tentative models
Best:
Next best:
Compare all possible subset models using either Cp, AIC, or BIC; find some model with a fairly small value Use sequential variable selection, like stepwise regression (this doesnt look at all possible subset models, but may be more convenient with some statistical programs)
Spring 2005 68
U9611
Sequential Variable Selection

1. Forward selection
a. Start with no Xs in the model St 412/512 page 98. b. Find the most significant additional X (with an F-test). c. If its p-value is less than some cutoff (like .05) add it to the model (and re-fit the model with the new set of Xs). d. Repeat (b) and (c) until no further Xs can be added.
2. Backward elimination
a. Start with all Xs in the mode.l b. Find the least significant of the Xs currently in the model. c. If its p-value is greater than some cutoff (like .05) drop it from the model (and re-fit with the remaining xs). d. Repeat until no further Xs can be dropped.
U9611 Spring 2005 69
Sequential Variable Selection (cont.)

3. Stepwise regression
a) b) c) d) Start with no Xs in St 412/512 page 99 Do one step of forward selection Do one step of backward elimination Repeat (b) and (c) until no further Xs can be added or dropped
4. Notes
a) b) c) Add and drop factor indicator variables as a group Dont take p-values and CIs for selected variables seriouslybecause of serious data snooping (not a problem for objectives 1 and 3) A drawback: the product is a single model. This is deceptive.

U9611
Think not: here is the best model. Think instead: here is one, possibly useful model.
Spring 2005 70
Criteria for Comparing Models
U9611
Spring 2005
71
Comparing Models (continued)

1. The proposed criteria: Mallows Cp Statistic, Schwarzs Bayesian Information Criterion (BIC or SBC), and Akaikes Information Criterion (AIC) 2. The idea behind these is the same, but the theory for 2 and small p differs arriving at the trade-off between small 3. My opinion: theres no way to truly say that one of these Criteria is better than the others 4. Computer programs: Fit all possible models; report the best 10 or so according to the selected criteria 5. Note: one other criteria: 2 As good). An equivalent criterion is - R Adjusted (see Sect.10.4.1)
U9611 Spring 2005 72
2 + 0 (sometimes used; but isnt
Cross Validation (12.6.4)

If tests, CIs, or prediction intervals are needed after Variable selection and if n is large, maybe try: Cross validation
Randomly
divide the data into 75% for model construction and 25% for inference Perform variable selection with the 75% Refit the same model (dont drop or add anything) on the remaining 25% and proceed with inference using that fit.
U9611 Spring 2005 73
Example: Sex Discrimination
U9611
Spring 2005
74
U9611
Spring 2005
75
U9611
Spring 2005
76
U9611
Spring 2005
77
U9611
Spring 2005
78

All possible regressions in Stata using commands window
U9611
Spring 2005
79

Variable Selection: Prof. Sharyn O'Halloran Sustainable Development U9611 Econometrics II

Uploaded by

Copyright:

Available Formats

Variable Selection: Prof. Sharyn O'Halloran Sustainable Development U9611 Econometrics II

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Variable Selection: Prof. Sharyn O'Halloran Sustainable Development U9611 Econometrics II

Uploaded by

Copyright:

Available Formats

Lecture 6: Variable Selection

Prof. Sharyn OHalloran Sustainable Development U9611 Econometrics II

Regression Diagnostics: Review

After estimating a model, we want to check the entire regression for:

of the residuals Omitted and unnecessary variables Heteroskedasticity

We also want to test individual variables for:

Look at Residuals: rvfplot

reg price weight mpg forXmpg foreign rvfplot, plotr(lcol(black)) yline(0)

Check Residuals for Normality

Distance above median 2000 4000 6000 8000

Residual plots also indicate non-normality

Stata Commands: imtest

check the Venn diagram

Stata Command: ovtest

Stata Command: ovtest

Evidence of omitted vars

Stata Command: ovtest

Evidence of omitted vars and non-constant variance, as before

Including Unnecessary Variables

Any explanatory power is due to its correlation with X (area 2).

Including Unnecessary Variables

Including Unnecessary Variables

Checking Individual Variables

If the diagnostics on the regression as a whole show potential problems, move to

Checking observations for:

Analyzing the contributions of individual variables to the regression:

Diagnostic Plots: lvr2plot

Diagnostic Plots: lvr2plot

Diagnostic Plots: lvr2plot

Diagnostic Plots: lvr2plot

Diagnostic Plots: lvr2plot

Diagnostic Plots: lvr2plot

Stata Commands: avplot

Added variable plots have the property that:

Stata Commands: avplot

To obtain the avplot for x1:

Example: Two Variables

Example: Two Variables

Example: Two Variables

Example: Two Variables

Example: Two Variables

Added variable plots: example

Alaska is unusual in its expenditure, and is apparently quite influential

Component Plus Residual Plots

To calculate this, get the partial residual for x2: a. Estimate

b. Use these results to calculate c. Plot this quantity vs. x2.

This is a variant on linear regression that downplays the influence of outliers

Stata command: rreg instead of reg

Robust Regression: Example

This is the original regression Run the usual diagnostics

Robust Regression: Example

Robust Regression: Example

rvfplot shows heterskedasticity Also, fails a hettest

Robust Regression: Example

Robust regression estimates

Robust Regression: Example

Robust regression estimates

Note: Coefficient on foreign changes from -1.65 to -3.18