Scribd
Upload
348 views11 pages

@vtucode - in Module 4 AI 2021 Scheme 5th Sem

The document discusses decision tree learning and algorithms. It introduces decision trees, describing their structure with root, branch and leaf nodes. It then covers two major procedures of decision trees: building the tree and performing classification. Several algorithms for building decision trees are described, including ID3, C4.5 and CART. The document also discusses validating and pruning decision trees to prevent overfitting.

Uploaded by

Ratan Shet
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
348 views11 pages

@vtucode - in Module 4 AI 2021 Scheme 5th Sem

The document discusses decision tree learning and algorithms. It introduces decision trees, describing their structure with root, branch and leaf nodes. It then covers two major procedures of decision trees: building the tree and performing classification. Several algorithms for building decision trees are described, including ID3, C4.5 and CART. The document also discusses validating and pruning decision trees to prevent overfitting.

Uploaded by

Ratan Shet
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

MODULE 4
CHAPTER 6
DECISION TREE LEARNING
6.1 Introduction

 Why called as decision tree ?


 As starts from root node and finds number of solutions .
 The benefits of having a decision tree are as follows :
 It does not require any domain knowledge.
 It is easy to comprehend.
 The learning and classification steps of a decision tree are simple and fast.
 Example : Toll free number

6.1.1 Structure of a Decision Tree A decision tree is a structure that includes a root
node, branches, and leaf nodes. Each internal node denotes a test on an attribute, each
branch denotes the outcome of a test, and each leaf node holds a class label. The topmost
node in the tree is the root node.

Applies to classification and regression model.

vtucode.in
1
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

The decision tree consists of 2 major procedures:

1) Building a tree and

2) Knowledge inference or classification.

Building the Tree

Knowledge Inference or Classification

Advantages of Decision Trees

vtucode.in
2
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

Disadvantages of Decision Trees

6.1.2 Fundamentals of Entropy

 How to draw a decision tree ?


Entropy
Information gain

vtucode.in
3
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

Algorithm 6.1: General Algorithm for Decision Trees

6.2 DECISION TREE INDUCTION ALGORITHMS

6.2.1 ID3 Tree Construction(ID3 stands for Iterative Dichotomiser 3 )


A decision tree is one of the most powerful tools of supervised learning algorithms
used for both classification and regression tasks.
It builds a flowchart-like tree structure where each internal node denotes a test on an
attribute, each branch represents an outcome of the test, and each leaf node (terminal

vtucode.in
4
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

node) holds a class label. It is constructed by recursively splitting the training data
into subsets based on the values of the attributes until a stopping criterion is met, such
as the maximum depth of the tree or the minimum number of samples required to split
a node.

vtucode.in
5
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

6.2.2 C4.5 Construction


C4.5 is a widely used algorithm for constructing decision trees from a dataset.
Disadvantages of ID3 are: Attributes must be nominal values, dataset must not include
missing data, and finally the algorithm tend to fall into overfitting.
To overcome this disadvantage Ross Quinlan, inventor of ID3, made some
improvements for these bottlenecks and created a new algorithm named C4.5. Now, the
algorithm can create a more generalized models including continuous data and could
handle missing data. And also works with discrete data, supports post-prunning.

vtucode.in
6
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

Dealing with Continuous Attributes in C4.5

vtucode.in
7
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

6.2.3 Classification and Regression Trees Construction


Classification and Regression Trees (CART) is a widely used algorithm for
constructing decision trees that can be applied to both classification and regression
tasks. CART is similar to C4.5 but has some differences in its construction and splitting
criteria.
The classification method CART is required to construct a decision tree based on Gini's
impurity index. It serves as an example of how the values of other variables can be used
to predict the values of a target variable. It functions as a fundamental machine-learning
method and provides a wide range of use cases

vtucode.in
8
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

6.2.4 Regression Trees

vtucode.in
9
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

6.3 VALIDATING AND PRUNING OF DECISION TREES

Validating and pruning decision trees is a crucial part of building accurate and robust
machine learning models. Decision trees are prone to overfitting, which means they can
learn to capture noise and details in the training data that do not generalize well to new,
unseen data.

Validation and pruning are techniques used to mitigate this issue and improve the
performance of decision tree models.

The pre-pruning technique of Decision Trees is tuning the hyperparameters prior to


the training pipeline. It involves the heuristic known as ‘early stopping’ which stops the
growth of the decision tree - preventing it from reaching its full depth. It stops the tree-
building process to avoid producing leaves with small samples. During each stage of
the splitting of the tree, the cross-validation error will be monitored. If the value of the
error does not decrease anymore - then we stop the growth of the decision tree.

The hyperparameters that can be tuned for early stopping and preventing overfitting

are: max_depth, min_samples_leaf, and min_samples_split

These same parameters can also be used to tune to get a robust model

Post-pruning does the opposite of pre-pruning and allows the Decision Tree model to

grow to its full depth. Once the model grows to its full depth, tree branches are removed

vtucode.in
10
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING(21CS54)

to prevent the model from overfitting. The algorithm will continue to partition data into

smaller subsets until the final subsets produced are similar in terms of the outcome

variable. The final subset of the tree will consist of only a few data points allowing the

tree to have learned the data to the T. However, when a new data point is introduced

that differs from the learned data - it may not get predicted well.

The hyperparameter that can be tuned for post-pruning and preventing overfitting

is: ccp_alpha

ccp stands for Cost Complexity Pruning and can be used as another option to control

the size of a tree. A higher value of ccp_alpha will lead to an increase in the number of

nodes pruned.

vtucode.in
11

You might also like