![Backward Elimination
• A method for identifying and removing independent
variables that do not contribute enough to the model.
• Steps:
• Fit (Train) the model with all the independent variables.
• Calculate the P-value of each independent variable.
• Eliminate independent variable with highest P-value above
threshold (e.g., 0.05 [5 percent]).
• Repeat (re-fit) until there are no independent variables with
P-value above threshold.
All Variables Train
Is Variable with highest
P-value > Threshold
DONE
Eliminate the variable](https://image.slidesharecdn.com/mlmultiplelinearregression-170919114353/85/ML-Multiple-Linear-Regression-5-320.jpg)
![Backward Elimination in Python
import statsmodels.formula.api as sm
X = np.append( arr = np.ones( (nrows,1 )).astype(int), values = X, axis = 1 )
• Prepare for Backward Elimination.
• The statsmodel does not take into account the constant b0.
• Need to fake it by adding a x0 = 1 independent variable for b0.
Function to create
column of ones
Append column
of ones to X
Create array of one
column of nrows
Append ones
to this array
X_opt = X[:, [0, 1, 2, 3, 4] ]
• Create array of optional independent variables (features) from which we
will eliminate independent variables.
All rows Start with all columns (i.e., 0, 1, 2 .. N)](https://image.slidesharecdn.com/mlmultiplelinearregression-170919114353/85/ML-Multiple-Linear-Regression-7-320.jpg)
![Backward Elimination in Python (2)
ols = sm.OLS( endog = y, exog = X_opt).fit() # Create OLS object and fit the model
ols.summary() # Display Statistical Metrics including P-values
• Use the class Ordinary Linear Square (OLS) from stats model to train (fit)
the model and get P-values.
Independent
Variables
Dependent
Variable (Label)
• Example elimination of an independent variable (x2).
X_opt = X[:, [0,1,3,4,5]]
ols = sm.OLS( endog = y, exog = X_opt).fit()
ols.summary()
Eliminate x2 (2), where 0 is x0 (constant)
• Repeat elimination until independent variable with highest P-value is not
greater than the threshold (e.g., 0.05).](https://image.slidesharecdn.com/mlmultiplelinearregression-170919114353/85/ML-Multiple-Linear-Regression-8-320.jpg)
ML - Multiple Linear Regression
- 1. Regression Methods in Machine Learning Multiple Linear Regression Portland Data Science Group Created by Andrew Ferlitsch Community Outreach Officer July, 2017
- 2. Multiple Linear Regression X1 (Independent Variable) Y (Dependent Variable) Hyperplane • Used to Predict a correlation between more than one independent variables and a dependent variable. e.g., Income and Age is correlated with Spending • When the data is plotted on a graph, there appears to be a hyperplane relationship. X2 (Independent Variable)
- 3. Simple vs. Multiple Linear Regression • Simple Linear Regression – one independent variable. y = b0 + b1x1 • Multiple Linear Regression – multiple independent variables. y = b0 + b1x1 + b2x2 … + bnxn 2nd independent variable and weight (coefficient) nth independent variable and weight (coefficient)
- 4. Feature Elimination ID Income Age Height Spending 37 18000 18 5’8” 38 75000 40 5’9” 39 27000 28 6’1” 40 24000 26 5’6” 41 45000 34 6’2” • In a dataset, we may not want to keep all the independent variables (features) in our model: • More Features = More Complex Model • If feature does not contribute to prediction, adds noise to the model. ID fields are like random numbers – do not contribute to prediction. Height not likely or very little to influence spending
- 5. Backward Elimination • A method for identifying and removing independent variables that do not contribute enough to the model. • Steps: • Fit (Train) the model with all the independent variables. • Calculate the P-value of each independent variable. • Eliminate independent variable with highest P-value above threshold (e.g., 0.05 [5 percent]). • Repeat (re-fit) until there are no independent variables with P-value above threshold. All Variables Train Is Variable with highest P-value > Threshold DONE Eliminate the variable
- 6. Multiple Linear Regression in Python from sklearn.linear_model import LinearRegression # sci-kit learn library for linear regression regressor = LinearRegression() # instantiate linear regression object regressor.fit(X_train, y_train) # train (fit) the model • Perform Linear Regression with all independent variables. y_pred = regressor.predict( X_test ) # y_pred is the list of predicted results • Run (Predict) the model on the test data. • Analyze (View) the predicted values (y_pred) to the actual values (y_test)
- 7. Backward Elimination in Python import statsmodels.formula.api as sm X = np.append( arr = np.ones( (nrows,1 )).astype(int), values = X, axis = 1 ) • Prepare for Backward Elimination. • The statsmodel does not take into account the constant b0. • Need to fake it by adding a x0 = 1 independent variable for b0. Function to create column of ones Append column of ones to X Create array of one column of nrows Append ones to this array X_opt = X[:, [0, 1, 2, 3, 4] ] • Create array of optional independent variables (features) from which we will eliminate independent variables. All rows Start with all columns (i.e., 0, 1, 2 .. N)
- 8. Backward Elimination in Python (2) ols = sm.OLS( endog = y, exog = X_opt).fit() # Create OLS object and fit the model ols.summary() # Display Statistical Metrics including P-values • Use the class Ordinary Linear Square (OLS) from stats model to train (fit) the model and get P-values. Independent Variables Dependent Variable (Label) • Example elimination of an independent variable (x2). X_opt = X[:, [0,1,3,4,5]] ols = sm.OLS( endog = y, exog = X_opt).fit() ols.summary() Eliminate x2 (2), where 0 is x0 (constant) • Repeat elimination until independent variable with highest P-value is not greater than the threshold (e.g., 0.05).