Scribd
Upload
67 views10 pages

Multiple Linear Regression-Part V Exhaustive Search: Dr. Gaurav Dixit

This document discusses methods for reducing the number of predictors in multiple linear regression models. It describes exhaustive search and partial-iterative search approaches. Exhaustive search evaluates all possible predictor combinations using criteria like adjusted R-squared and Mallow's Cp. Partial-iterative search uses forward, backward, or stepwise selection in a computationally cheaper way, but may miss the best model. Exhaustive search is preferred for a moderate number of predictors, while partial-iterative search works better for many predictors.

Uploaded by

Aniket Sujay
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
67 views10 pages

Multiple Linear Regression-Part V Exhaustive Search: Dr. Gaurav Dixit

This document discusses methods for reducing the number of predictors in multiple linear regression models. It describes exhaustive search and partial-iterative search approaches. Exhaustive search evaluates all possible predictor combinations using criteria like adjusted R-squared and Mallow's Cp. Partial-iterative search uses forward, backward, or stepwise selection in a computationally cheaper way, but may miss the best model. Exhaustive search is preferred for a moderate number of predictors, while partial-iterative search works better for many predictors.

Uploaded by

Aniket Sujay
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 10

MULTIPLE LINEAR REGRESSION-PART V

EXHAUSTIVE SEARCH
LECTURE 26

DR. GAURAV DIXIT


DEPARTMENT OF MANAGEMENT STUDIES

1
MULTIPLE LINEAR REGRESSION

• Steps to reduce the no. of predictors


– Summary statistics and graphs
– Statistical methods using computational power
• Exhaustive search: all possible combinations
• Partial-iterative search: algorithm based

• Exhaustive Search
– Large no. of subsets
– Criteria to compare models
• Adjusted R2

2
MULTIPLE LINEAR REGRESSION

• Adjusted R2
𝑛−1
R2adj = 1 − (1 − 𝑅2)
𝑛−𝑝−1
Where R2 is proportion of explained variability in the model
𝑛
𝑆𝑆𝐸 𝑖=1(𝑦𝑖−𝑦𝑖 )
R2 =1− =1− 𝑛 (𝑦𝑖−𝑦)
𝑆𝑆𝑇 𝑖=1
1 𝑛
𝑦= 𝑖=1 𝑦𝑖
𝑛
• R2 is called coefficient of determination

3
MULTIPLE LINEAR REGRESSION

• R2 would be equal to squared correlation in a single predictor


model, that is how R2 gets its name

• Adjusted R2 introduces a penalty on the no. of predictors to


trade-off between artificial increase vs. amount of
information

• High adjusted R2 values -> low σ2


4
MULTIPLE LINEAR REGRESSION

• Exhaustive Search
– Criteria to compare models
• Mallow’s Cp

• Mallow’s Cp
SSR
Cp = +2 p+1 −n
σf 2

Where σf 2 is estimated value of σ2 in the full model


and SSR = 𝑛𝑖=1(𝑦𝑖 − 𝑦)

5
MULTIPLE LINEAR REGRESSION

• Mallow’s Cp
– Assumption: full model with all predictors is unbiased
• Predictors elimination would reduce the variability
– Best subset model would have Cp ~ p+1 and p would be a small value
– Requires high n value for the training partition relative to p
• Open RStudio

6
MULTIPLE LINEAR REGRESSION

• Partial-iterative search
– Computationally cheaper
– Best subset is not guaranteed
• Potential of missing “good” sets of predictors
– Produce close-to-best subsets
– Preferred approach for large no. of predictors
– For moderate no. of predictors, exhaustive search is better
• Trade-off between computation cost vs. potential of finding
best subset
7
MULTIPLE LINEAR REGRESSION

• Partial-iterative search algorithms


– Forward selection
• Add predictors one by one
• Strength as a single predictor is used
– Backward elimination
• Drop predictors one by one
– Stepwise regression
• Add predictors one by one and consider dropping insignificant ones

• Open RStudio

8
Key References

• Data Science and Big Data Analytics: Discovering, Analyzing,


Visualizing and Presenting Data by EMC Education Services
(2015)
• Data Mining for Business Intelligence: Concepts, Techniques,
and Applications in Microsoft Office Excel with XLMiner by
Shmueli, G., Patel, N. R., & Bruce, P. C. (2010)

9
Thanks…

10

You might also like