Random

Forest

1
Contents
• What is Random Forest?

• Ensemble Methods - Bagging

• How does Random Forest work?

• Hyper-Parameters in Random Forest

• Parameter Tuning - Cross-Validation & GridSearchCV

• Building RF in Scikit-learn

• Pros and Cons

2
What is Random Forest?
• Random Forest is a
supervised learning
algorithm and capable of
performing both regression
and classification tasks.

• As the name suggests,

Random Forest algorithm
creates a forest with a
number of decision trees.

3
Ensemble method
• Use multiple learning algorithms to obtain
better predictions.
• Train various different models, aggregate
their predictions to improvise stability and
predictive power.

• As we see, we need numbers of

models(learners), whose predictive power
is just slightly better than random chance.
Such learners are called as weak learners.
• We take such weak learners to make one
combined strong learner.

4
Bagging
• The idea behind bagging is combining the results of multiple models (for
instance, all decision trees) to get a generalized result.
• Bagging uses a sampling technique called – Bootstrapping.
• Bootstrapping is a sampling technique in which we create subsets of
observations from the original dataset, with replacement.
• Bagging (or Bootstrap Aggregating) technique uses these subsets (bags) to
get a fair idea of the distribution (complete set).
• The size of subsets created for bagging may be same or less than the original
set.

5
Bagging
• Multiple subsets are created from the original dataset, selecting
observations with replacement.

6
Bagging
• A base model (weak model) is created on
each of these subsets.
• The models run in parallel and are
independent of each other.
• The final predictions are determined by
combining the predictions from all the
models.

7
How does Random Forest work?
• RF consists multiple decision trees which act as base learners.

• Each decision tree is given a subset of random samples from the data set
(hence the name random).

• RF algorithm uses an Ensemble method – Bagging (Bootstrap Aggregating)

• Then, Random Forest train each base learner (i.e Decision Tree) on a different
sample of data and the sampling of data points happens with replacement.

8
Example
• Consider a training dataset : [X1, X2, X3, … X10, Y].

• Random forest will create decision trees taking the input from subset using
bagging as shown below:

9
Hyper-Parameters Random Forest
• Optimization of RF depends on few inbuilt parameters.

• n_estimators* - number of decision trees that the algorithm creates. As the

number tree increases, the performance increases and the predictions are
more stable but it slows down the computation.

• max_features* - maximum number of features that are considered for

splitting a node.

• n_jobs - number of jobs to run in parallel. If n_jobs=1, it uses one processor.

If n_jobs=-1, then the number of jobs is set to the number of cores available.

10
Parameters Random Forest
• max_depth is the maximum depth of the tree. The deeper the tree, the more
splits it has and it captures more information about the data.

• criterion is the function to measure the quality of a split. Supported criteria

are “gini” for the Gini impurity and “entropy” for the information gain.

11
Cross-Validation (CV)
• Cross-validation is a statistical method used to estimate the performance of
machine learning models.

• It is a resampling procedure used to evaluate machine learning models on a

limited data sample.

• The most common method is K-Fold CV.

• Normally, we split the data into – train & test data sets.

• In K-fold CV, training data is further split into K number of subsets, called
folds.

12
Cross-Validation (CV)
• Then iteratively fit the model K times, each time training the data on K-1 of
the folds and evaluating on the Kth fold (called the validation data).

• For example, consider the train data is splitted into 5 folds (K = 5).

• 1st Iteration - train on the first four folds and evaluate on the fifth.

• 2nd Iteration - train on the first, second, third, and fifth fold and evaluate on the fourth.

• And repeat the same procedure.

• At end of training, we average the performance on each of the folds to come

up with final validation metrics for the model.

13
Cross-Validation (CV)
• 5 Fold Cross Validation –

• For hyperparameter tuning, we perform many iterations of the entire K-Fold CV

process, each time using different model settings.

• If we have 10 sets of hyperparameters and are using 5-Fold CV, that represents 50
training loops.

14
GridSearchCV
• Grid-search is used to find the optimal hyperparameters of a model which
results in the most ‘accurate’ predictions.

• To implement the Grid Search algorithm we need to import GridSearchCV

class from the sklearn.model_selection library.

• The first step you need to perform is to create a dictionary of all the
parameters and their corresponding set of values that you want to test for
best performance.

15
Pros & Cons
Pros:
• Random Forest algorithm avoids overfitting.
• For both classification and regression task, the same random forest algorithm can be used.
• The Random Forest algorithm can be used for identifying the most important features from
the training dataset. It helps in feature engineering.

Cons:
• Random Forest is difficult to interpret. Because of averaging the results of many trees
becomes hard for us to figure out why a random forest is making predictions the way it is.
• Random Forest takes a longer time to create. It is computationally expensive compared to a
Decision Tree.

16