Scribd
Upload
1K views2 pages

Shrinkage Method

Shrinkage method in Machine learning

Uploaded by

Anamika Sharma
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
1K views2 pages

Shrinkage Method

Shrinkage method in Machine learning

Uploaded by

Anamika Sharma
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

SHRINKAGE METHOD IN ML

• Shrinkage Methods in Machine Learning are techniques that regularize models to prevent
overfitting by penalizing the complexity of the model. They "shrink" the estimated coefficients
toward zero to reduce variance and improve the model's generalizability. These methods are
especially useful when dealing with high-dimensional datasets where the number of features
(predictors) is large relative to the number of observations.

Common Shrinkage Methods :

1. Ridge Regression (L2 Regularization) :

○ Objective: Minimize the sum of squared errors while also penalizing large coefficient values.

○ Penalty: The L2 penalty adds a term to the cost function proportional to the square of the
magnitude of the coefficients.

○ Effect: It forces the coefficients to be smaller (shrinks them), but it doesn't shrink them to
exactly zero. This is particularly useful when all features are relevant, but their coefficients
need to be reduced to avoid overfitting.

○ Cost Function: J(θ)=

, where lambda controls the degree of shrinkage.

2. Lasso Regression (L1 Regularization) :

○ Objective: Similar to ridge regression but with a different penalty.

○ Penalty: The L1 penalty adds a term to the cost function proportional to the absolute values
of the coefficients.

○ Effect: Lasso can shrink some coefficients to zero, effectively performing feature selection by
eliminating irrelevant features.

○ Cost Function: J(θ)=

○ It is useful in situations where you expect that only a subset of the features is truly important
for the prediction.

3. Elastic Net :
3. Elastic Net :

○ Objective: Combines both L1 and L2 penalties, blending the benefits of ridge and lasso.

○ Penalty: The cost function includes both the L1 and L2 terms.

○ Effect: It can handle situations where groups of correlated features exist, and some feature
selection is desired, but pure lasso would eliminate too many features.

○ Cost Function: J(θ)=

○ The relative importance of L1 and L2 is controlled by the parameters λ1\lambda_1λ1 and λ2


\lambda_2λ2.

Why Shrinkage is Important :

• Prevents Overfitting : Shrinkage reduces model complexity, ensuring that the model captures the
underlying pattern rather than the noise.

• Improves Prediction : By reducing variance, shrinkage methods can lead to better generalization on
unseen data.

• Handles Multicollinearity : In high-dimensional data, where features may be correlated, shrinkage


methods like ridge regression help by keeping all predictors but shrinking their influence.

Summary :

• Shrinkage methods are valuable when you have a large number of predictors or when you're
concerned about overfitting. Ridge, Lasso, and Elastic Net are popular techniques, each suited to
different types of data and problems in machine learning.

You might also like