2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Decision Tree in Machine Learning

Read Courses Practice Video Jobs

A decision tree in machine learning is a versatile, interpretable algorithm used for predictive
modelling. It structures decisions based on input data, making it suitable for both
classification and regression tasks. This article delves into the components, terminologies,
construction, and advantages of decision trees, exploring their applications and learning
algorithms.

Decision Tree in Machine Learning

A decision tree is a type of supervised learning algorithm that is commonly used in machine
learning to model and predict outcomes based on input data. It is a tree-like structure where
each internal node tests on attribute, each branch corresponds to attribute value and each leaf
node represents the final decision or prediction. The decision tree algorithm falls under the
category of supervised learning. They can be used to solve both regression and classificat ion
problems.

Decision Tree Terminologies

There are specialized terms associated with decision trees that denote various components
and facets of the tree structure and decision-making procedure. :

Root Node: A decision tree’s root node, which represents the original choice or feature
from which the tree branches, is the highest node.
Internal Nodes (Decision Nodes): Nodes in the tree whose choices are determined by the
values of particular attributes. There are branches on these nodes that go to other nodes.
Leaf Nodes ( Terminal Nodes): The branches’ termini, when choices or forecasts are
decided upon. There are no more branches on leaf nodes.
Branches (Edges): Links between nodes that show how decisions are made in response to
particular circumstances.
Splitt ing: The process of dividing a node into two or more sub-nodes based on a decision
criterion. It involves selecting a feature and a threshold to create subsets of data.

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 1/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Parent Node: A node that is split into child nodes. The original node from which a split
originates.
90%Child
RefundNode: Nodes
@Courses created
Machine as a Tutorial
Learning result of Data
a split fromTutorial
Analysis a parentPython
node.- Data visualization tutorial Nu
Decision Criterion: The rule or condition used to determine how the data should be split at
a decision node. It involves comparing feature values against a threshold.
Pruning: The process of removing branches or nodes from a decision tree to improve its
generalization and prevent overfitting.

Understanding these terminologies is crucial for interpreting and working with decision trees
in machine learning applications.

How Decision Tree is formed?

The process of forming a decision tree involves recursively partitioning the data based on the
values of different attributes. The algorithm selects the best attribute to split the data at each
internal node, based on certain criteria such as information gain or Gini impurity. This splitting
process continues until a stopping criterion is met, such as reaching a maximum depth or
having a minimum number of instances in a leaf node.

Why Decision Tree?

Decision trees are widely used in machine learning for a number of reasons:

Decision trees are so versatile in simulating intricate decision-making processes, because

of their interpretability and versatility.
Their portrayal of complex choice scenarios that take into account a variety of causes and
outcomes is made possible by their hierarchical structure.

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 2/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

They provide comprehensible insights into the decision logic, decision trees are especially
helpful for tasks involving categorization and regression.
They are proficient with both numerical and categorical data, and they can easily adapt to a
variety of datasets thanks to their autonomous feature selection capability.
Decision trees also provide simple visualization, which helps to comprehend and elucidate
the underlying decision processes in a model.

Decision Tree Approach

Decision tree uses the tree representation to solve the problem in which each leaf node
corresponds to a class label and attributes are represented on the internal node of the tree.
We can represent any boolean function on discrete attributes using the decision tree.

Below are some assumptions that we made while using the decision tree:
At the beginning, we consider the whole training set as the root.

Feature values are preferred to be categorical. If the values are continuous then they are
discretized prior to building the model.
On the basis of attribute values, records are distributed recursively.
We use statistical methods for ordering attributes as root or the internal node.

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 3/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

As you can see from the above image the Decision Tree works on the Sum of Product form
which is also known as Disjunctive Normal Form. In the above image, we are predicting the
use of computer in the daily life of people. In the Decision Tree, the major challenge is the
identification of the attribute for the root node at each level. This process is known as
attribute selection. We have two popular attribute selection measures:

1. Information Gain
2. Gini Index

1. Informat ion Gain:

When we use a node in a decision tree to partition the training instances into smaller subsets
the entropy changes. Information gain is a measure of this change in entropy.

Suppose S is a set of instances,

A is an attribute
S v is the subset of S
v represents an individual value that the attribute A can take and Values (A) is the set of all
possible values of A, then

Entropy: is the measure of uncertainty of a random variable, it characterizes the impurity of an

arbitrary collection of examples. The higher the entropy more the information content.

Suppose S is a set of instances, A is an attribute, S v is the subset of S with A = v, and Values

(A) is the set of all possible values of A, then

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 4/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Example:

For the set X = {a,a,a,b,b,b,b,b}

Total instances: 8
Instances of b: 5
Instances of a: 3

Building Decision Tree using Informat ion Gain T he essent ials:

Start with all training instances associated with the root node
Use info gain to choose which attribute to label each node with
Note: No root-to-leaf path should contain the same discrete attribute twice
Recursively construct each subtree on the subset of training instances that would be
classified down that path in the tree.
If all positive or all negative training instances remain, the label that node “yes” or “no”
accordingly
If no attributes remain, label with a majority vote of training instances left at that node
If no instances remain, label with a majority vote of the parent’s training instances.

Example: Now, let us draw a Decision Tree for the following data using Information gain.
Training set: 3 features and 2 classes

X Y Z C

1 1 1 I

1 1 0 I

0 0 1 II

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 5/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

X Y Z C

1 0 0 II

Here, we have 3 features and 2 output classes. To build a decision tree using Information gain.
We will take each of the features and calculate the information for each feature.

Split on feature X

Split on feature Y

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 6/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Split on feature Z

From the above images, we can see that the information gain is maximum when we make a
split on feature Y. So, for the root node best-suited feature is feature Y. Now we can see that
while splitting the dataset by feature Y, the child contains a pure subset of the target variable.
So we don’t need to further split the dataset. The final tree for the above dataset would look
like this:

2. Gini Index
Gini Index is a metric to measure how often a randomly chosen element would be
incorrectly identified.
It means an attribute with a lower Gini index should be preferred.
Sklearn supports “Gini” criteria for Gini Index and by default, it takes “gini” value.
The Formula for the calculation of the Gini Index is given below.

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 7/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

The Gini Index is a measure of the inequality or impurity of a distribution, commonly used in
decision trees and other machine learning algorithms. It ranges from 0 to 1, where 0
represents perfect equality (all values are the same) and 1 represents perfect inequality (all
values are different ).

Some additional features and characteristics of the Gini Index are:

It is calculated by summing the squared probabilities of each outcome in a distribution and
subtracting the result from 1.
A lower Gini Index indicates a more homogeneous or pure distribution, while a higher Gini
Index indicates a more heterogeneous or impure distribution.
In decision trees, the Gini Index is used to evaluate the quality of a split by measuring the
difference between the impurity of the parent node and the weighted impurity of the child
nodes.
Compared to other impurity measures like entropy, the Gini Index is faster to compute and
more sensitive to changes in class probabilities.
One disadvantage of the Gini Index is that it tends to favor splits that create equally sized
child nodes, even if they are not optimal for classification accuracy.
In practice, the choice between using the Gini Index or other impurity measures depends
on the specific problem and dataset, and often requires experimentation and tuning.

Example of a Decision Tree Algorithm

Forecast ing Act ivit ies Using Weat her Informat ion

Root node: Whole dataset

Attribute : “Outlook” (sunny, cloudy, rainy).
Subsets: Overcast, Rainy, and Sunny.
Recursive Splitt ing: Divide the sunny subset even more according to humidity, for
example.
Leaf Nodes: Activities include “swimming,” “hiking,” and “staying inside.”

Beginning wit h t he ent ire dataset as t he root node of t he decision tree:

Determine the best attribute to split the dataset based on information gain, which is
calculated by the formula: Information gain = Entropy(parent ) – [Weighted average] *
Entropy(children), where entropy is a measure of impurity or disorder of a set of examples,
and the weighted average is based on the number of examples in each child node.
Create a new internal node that corresponds to the best attribute and connects it to the
root node. For example, if the best attribute is “outlook” (which can have values “sunny”,

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 8/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

“overcast”, or “rainy”), we create a new node labeled “outlook” and connect it to the root
node.
Partition the dataset into subsets based on the values of the best attribute. For example,
we create three subsets: one for instances where the outlook is “sunny”, one for instances
where the outlook is “overcast”, and one for instances where the outlook is “rainy”.
Recursively repeat steps 1-4 for each subset until all instances in a given subset belong to
the same class or no further splitting is possible. For example, if the subset of instances
where the outlook is “overcast” contains only instances where the activity is “hiking”, we
assign a leaf node labeled “hiking” to this subset. If the subset of instances where the
outlook is “sunny” is further split based on the humidity attribute, we repeat steps 2-4 for
this subset.
Assign a leaf node to each subset that contains instances that belong to the same class.
For example, if the subset of instances where the outlook is “rainy” contains only instances
where the activity is “stay inside”, we assign a leaf node labeled “stay inside” to this subset.
Make predictions based on the decision tree by traversing it from the root node to a leaf
node that corresponds to the instance being classified. For example, if the outlook is
“sunny” and the humidity is “high”, we traverse the decision tree by following the “sunny”
branch and then the “high humidity” branch, and we end up at a leaf node labeled
“swimming”, which is our predicted activity.

Advantages of Decision Tree

Easy to understand and interpret, making them accessible to non-experts.

Handle both numerical and categorical data without requiring extensive preprocessing.
Provides insights into feature importance for decision-making.
Handle missing values and outliers without significant impact.
Applicable to both classification and regression tasks.

Disadvantages of Decision Tree

Disadvantages include the potential for overfitting

Sensitivity to small changes in data, limited generalization if training data is not
representative
Potential bias in the presence of imbalanced data.

Conclusion

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 9/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Decision trees, a key tool in machine learning, model and predict outcomes based on input
data through a tree-like structure. They offer interpretability, versatility, and simple
visualization, making them valuable for both categorization and regression tasks. While
decision trees have advantages like ease of understanding, they may face challenges such as
overfitting. Understanding their terminologies and formation process is essential for effective
application in diverse scenarios.

Frequently Asked Questions (FAQs)

1. What are the major issues in decision tree learning?

Major issues in decision tree learning include overfitting, sensitivity to small data
changes, and limited generalization. Ensuring proper pruning, tuning, and handling
imbalanced data can help mitigate these challenges for more robust decision tree
models.

2. How does decision tree help in decision mak ing?

Decision trees aid decision-making by representing complex choices in a hierarchical

structure. Each node tests specific attributes, guiding decisions based on data values.
Leaf nodes provide final outcomes, offering a clear and interpretable path for decision
analysis in machine learning.

3. What is the maximum depth of a decision tree?

The maximum depth of a decision tree is a hyperparameter that determines the

maximum number of levels or nodes from the root to any leaf. It controls the
complexity of the tree and helps prevent overfitting.

4. What is the concept of decision tree?

A decision tree is a supervised learning algorithm that models decisions based on

input features. It forms a tree-like structure where each internal node represents a
https://www.geeksforgeeks.org/decision-tree-introduction-example/ 10/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

decision based on an attribute, leading to leaf nodes representing outcomes.

5. What is entropy in decision tree?

In decision trees, entropy is a measure of impurity or disorder within a dataset. It

quantifies the uncertainty associated with classifying instances, guiding the algorithm
to make informative splits for effective decision-making.

6. What are the Hyperparameters of decision tree?

1. Max Depth: Maximum depth of the tree.

2. Min Samples Split: Minimum samples required to split an internal node.
3. Min Samples Leaf: Minimum samples required in a leaf node.
4. Criterion: The function used to measure the quality of a split

Don't miss your chance to ride the wave of the data revolution! Every industry is scaling new
heights by tapping into the power of data. Sharpen your skills and become a part of the
hottest trend in the 21st century.

Dive into the future of technology - explore the Complete Machine Learning and Data Science
Program by GeeksforGeeks and stay ahead of the curve.

Three 90 Challenge ending on 29th Feb! Last chance to get 90% refund by completing 90%
course in 90 days. Explore offer now.

Last Updated : 04 Dec, 2023 88

Previous Next

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 11/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Share your thoughts in the comments Add Your Comment

Similar Reads
How to Extract the Decision Rules from scikit- Decision Threshold In Machine Learning
learn Decision-tree?

Support vector machine in Machine Learning Azure Virtual Machine for Machine Learning

Machine Learning Model with Teachable Difference Between Artificial Intelligence vs

Machine Machine Learning vs Deep Learning

Need of Data Structures and Algorithms for Machine Learning - Learning VS Designing
Deep Learning and Machine Learning

Passive and Active learning in Machine Automated Machine Learning for Supervised
Learning Learning using R

Complete Tutorials
Python Crash Course Python API Tutorial: Getting Started with APIs

Advanced Python Tutorials Python Automation Tutorial

OpenAI Python API - Complete Guide

A Abhishek …

Article Tags : Machine Learning , Python

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 12/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Practice Tags : Machine Learning, python

Additional Information

A-143, 9th Floor, Sovereign Corporate

Tower, Sector-136, Noida, Uttar Pradesh -
201305

Company Explore
About Us Job-A-Thon Hiring Challenge
Legal Hack-A-Thon
Careers GfG Weekly Contest
In Media Offline Classes (Delhi/NCR)
Contact Us DSA in JAVA/C++
Advertise with us Master System Design
GFG Corporate Solution Master CP
Placement Training Program GeeksforGeeks Videos
Apply for Mentor Geeks Community

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 13/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Languages DSA
Python Data Structures
Java Algorithms
C++ DSA for Beginners
PHP Basic DSA Problems
GoLang DSA Roadmap
SQL Top 100 DSA Interview Problems
R Language DSA Roadmap by Sandeep Jain
Android Tutorial All Cheat Sheets
Tutorials Archive

Data Science & ML HTML & CSS

Data Science With Python HTML
Data Science For Beginner CSS
Machine Learning Tutorial Web Templates
ML Maths CSS Frameworks
Data Visualisation Tutorial Bootstrap
Pandas Tutorial Tailwind CSS
NumPy Tutorial SASS
NLP Tutorial LESS
Deep Learning Tutorial Web Design

Python Computer Science

Python Programming Examples GATE CS Notes
Django Tutorial Operating Systems
Python Projects Computer Network
Python Tkinter Database Management System
Web Scraping Software Engineering
OpenCV Python Tutorial Digital Logic Design
Python Interview Question Engineering Maths

DevOps Competitive Programming

Git Top DS or Algo for CP
AWS Top 50 Tree

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 14/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

Docker Top 50 Graph

Kubernetes Top 50 Array
Azure Top 50 String
GCP Top 50 DP
DevOps Roadmap Top 15 Websites for CP

System Design JavaScript

High Level Design JavaScript Examples
Low Level Design TypeScript
UML Diagrams ReactJS
Interview Guide NextJS
Design Patterns AngularJS
OOAD NodeJS
System Design Bootcamp Lodash
Interview Questions Web Browser

NCERT Solutions School Subjects

Class 12 Mathematics
Class 11 Physics
Class 10 Chemistry
Class 9 Biology
Class 8 Social Science
Complete Study Material English Grammar

Commerce UPSC Study Material

Accountancy Polity Notes
Business Studies Geography Notes
Economics History Notes
Management Science and Technology Notes
HR Management Economy Notes
Finance Ethics Notes
Income Tax Previous Year Papers

SSC/ BANKING Colleges

SSC CGL Syllabus Indian Colleges Admission & Campus Experiences

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 15/16
2/10/24, 1:27 AM Decision Tree in Machine Learning - GeeksforGeeks

SBI PO Syllabus List of Central Universities - In India

SBI Clerk Syllabus Colleges in Delhi University
IBPS PO Syllabus IIT Colleges
IBPS Clerk Syllabus NIT Colleges
SSC CGL Practice Papers IIIT Colleges

Companies Preparation Corner

META Owned Companies Company-Wise Recruitment Process
Alphabhet Owned Companies Resume Templates
TATA Group Owned Companies Aptitude Preparation
Reliance Owned Companies Puzzles
Fintech Companies Company-Wise Preparation
EdTech Companies

Exams More Tutorials

JEE Mains Software Development
JEE Advanced Software Testing
GATE CS Product Management
NEET SAP
UGC NET SEO - Search Engine Optimization
Linux
Excel

Free Online Tools Write & Earn

Typing Test Write an Article
Image Editor Improve an Article
Code Formatters Pick Topics to Write
Code Converters Share your Experiences
Currency Converter Internships
Random Number Generator
Random Password Generator

@GeeksforGeeks, Sanchhaya Education Private Limited, All rights reserved

https://www.geeksforgeeks.org/decision-tree-introduction-example/ 16/16