International Journal of Computer Applications (0975 – 8887)

Volume 163 – No 8, April 2017

Analysis of Various Decision Tree Algorithms for

Classification in Data Mining
Bhumika Gupta, PhD Aditya Rawat Akshay Jain
Assistant Professor, B.Tech IV Year B.Tech IV Year
C.S.E.D G.B.P.E.C, Pauri G.B.P.E.C, Pauri
G.B.P.E.C, Pauri Uttarakhand, India Uttarakhand, India
Uttarakhand, India
Arpit Arora Naresh Dhami
B.Tech IV Year B.Tech IV Year
G.B.P.E.C, Pauri G.B.P.E.C, Pauri
Uttarakhand, India Uttarakhand, India

ABSTRACT of a target variable by giving several inputs. A decision tree is

Today the computer technology and computer network a decision support tool that uses a tree-like graph or model of
technology has developed so much and is still developing decisions and their possible outcomes. A decision tree is a
with pace.Thus, the amount of data in the information classifier in the form of a tree structure where each node is
industry is getting higher day by day. This large amount of either:
data can be helpful for analyzing and extracting useful  Leaf node- Leaf node is an indicator of the value of
knowledge from it. The hidden patterns of data are analyzed target attribute(class) of examples, or
and then categorized into useful knowledge. This process is  A decision node- A decision node specifies all possible
known as Data Mining. [4].Among the various data mining tests on a single attribute-value, with one branch and
techniques, Decision Tree is also the popular one. Decision sub-tree for each possible outcome of the test [6].
tree uses divide and conquer technique for the basic learning
strategy. A decision tree is a flow chart-like structure in which 2. CONCEPT
each internal node represents a “test” on an attribute where 2.1 Theory
each branch represents the outcome of the test and each leaf Decision Tree learning is one of the most widely used and
node represents a class label. This paper discusses various practical methods for inductive inference over supervised
algorithms of the decision tree (ID3, C4.5, CART), their data. Based on various attributes a decision tree represents a
features, advantages, and disadvantages. procedure that classifies the categorical data. Besides this
decision tree are also for processing a large amount of data
Keywords and thus find use in data mining applications. For constructing
Decision Tree, ID3, C4.5, Entropy, Information Gain. decision trees no domain knowledge or parameter setting is
required. Thus decision trees are sufficient as well as
1. INTRODUCTION appropriate for exploratory knowledge discovery and their
Data mining comprises extracting information from a data set representation of acquired knowledge in tree form is intuitive
and transforming it to a structure that is understandable [4].It and easy to understand.
is the computational process of discovering patterns in large
data sets involving methods at the intersection of artificial 2.2 Why use Decision Trees
intelligence, machine learning, statistics and database  Decision trees can be visualized and are simple to
systems. Data mining allows you to sift through all the chaotic understand and interpret.
and repetitive noise in your data. It also helps to understand
the relevant information and make good use of that  They require very little data preparation whereas other
information to assess likely outcomes. Thus data mining techniques often require data normalization, the creation
accelerates the pace of making informed decisions. There are of dummy variables and removal of blank values.
six classes in data mining namely Anomaly Detection,
Association Rule Learning, Clustering, Classification,  The cost of using the tree (for predicting data) is
Regression. Classification is a data mining function that logarithmic in the number of data points used to train
assigns items in a collection to target categories or classes. the tree.
Classification aims at predicting the target class for each case
 Decision trees can handle both categorical and
in the data. For example, a classification model can help to
numerical data whereas other techniques are specialized
identify bank loan applications as safe or risky. The various
for only one type of variable.
classification techniques used in the field of data mining are
decision tree induction, rule-based method, memory-based  Decision trees can handle multi-output problems.
learning, Bayesian networks, neural networks and support
vector machines.  Uses a white box model i.e. the explanation for the
condition can be explained easily by Boolean logic
The most widely applied supervised classification technique is because there are mostly two outputs. For example yes
Decision Tree. Decision Tree induction comprises of learning or no.
and classification. These steps are simple and fast and thus
Decision Tree can be applied to any domain [10]. The goal of
the decision tree is to form a model and then predict the value

15
 International Journal of Computer Applications (0975 – 8887)
Volume 163 – No 8, April 2017

 Decision trees can perform well even if assumptions are  Classifying the continuous data may prove to be
somewhat violated by the dataset from which the data is expensive in terms of computation, as many trees have
taken. to be generated to see where to break the continuum.
 One disadvantage of ID3 is that when given a large
2.3 Types of Decision Trees number of input values, it is overly sensitive to features
Decision trees used in data mining are mainly of two types: with a large number of values [2].
 Classification tree in which analysis is when the 2.3.2 C4.5
predicted outcome is the class to which the data C4.5 is an algorithm used to generate a decision tree which
belongs. For example outcome of loan application as was also developed by Ross Quinlan. It is an extension of
safe or risky. Quinlan’s ID3 algorithm. C4.5 generates decision trees which
 Regression tree in which analysis is when the predicted can be used for classification and therefore C4.5 is often
outcome can be considered a real number. For example referred to as statistical classifier [11]. It is better than the ID3
population of a state. algorithm because it deals with both continuous and discrete
Both the classification and regression trees have similarities as attributes and also with the missing values and pruning trees
well as differences, such as procedure used to determine after construction. C5.0 is the commercial successor of C4.5
where to split. because it is a lot faster, more memory efficient and used for
building smaller decision trees.C4.5 performs by default a tree
There are various decision trees algorithms namely pruning process. This leads to the formation of smaller trees,
ID3(Iterative Dichotomiser 3), C4.5, CART(Classification more simple rules and produces more intuitive interpretations.
and Regression Tree), CHAID(CHi- squared Automatic
Interaction Detector), MARS. Out of these, we will be C4.5 follows three steps in tree growth [3]:
discussing the more popular ones which are ID3, C4.5,  For splitting of categorical attributes, C4.5 follows the
CART. similar approach to ID3 algorithms. Continuous
attributes always generate binary splits.
2.3.1 ID3(Iterative Dichotomiser)  Selecting attribute with the highest gain ratio.
ID3 is an algorithm developed by Ross Quinlan used to  These steps are repeatedly applied to new tree branches
generate a decision tree from a dataset [12]. To construct a and growth of the tree is stopped after checking of stop
decision tree, ID3 uses a top-down, greedy search through the criterion. Information gain bias the attribute with more
given sets, where each attribute at every tree node is tested to number of values. Thus, C4.5 uses Gain Ratio which is
select the attribute that is best for classification of a given set a less biased selection criterion.
[10]. Therefore, the attribute with the highest information gain
can be selected as the test attribute of the current node. ID3 is 2.3.2.1 Advantages of C4.5
based on Occam’s razor. In this algorithm, small decision  C4.5 is easy to implement.
trees are preferred over the larger ones. However, it does not  C4.5 builds models that can be easily interpreted.
always construct the smallest tree and is, therefore, a heuristic
algorithm [6].  It can handle both categorical and continuous values.
 It can deal with noise and deal with missing value
For building a decision tree model, ID3 only accepts attributes.
categorical attributes. Accurate results are not given by ID3
when there is noise and when it is serially implemented. 2.3.2.2 Disadvantages of C4.5
Therefore data is preprocessed before constructing a decision  A small variation in data can lead to different decision
tree [1]. For constructing a decision tree information gain is trees when using C4.5.
calculated for each and every attribute and attribute with the  For a small training set, C4.5 does not work very well.
highest information gain becomes the root node. The rest
possible values are denoted by arcs. After that, all the 2.3.3 CART
outcome instances that are possible are examined whether It stands for Classification And Regression Trees. It was
they belong to the same class or not. For the instances of the introduced by Breiman in 1984. CART algorithm builds both
same class, a single name class is used to denote otherwise the classification and regression trees. The classification tree is
instances are classified on the basis of splitting attribute. constructed by CART by the binary splitting of the attribute.
Gini Index is used as selecting the splitting attribute. The
2.3.1.1 Advantages of ID3 CART is also used for regression analysis with the help of
 The training data is used to create understandable regression tree. The regression feature of CART can be used
prediction rules. in forecasting a dependent variable given a set of predictor
 It builds the fastest as well as a short tree. variable over a given period of time. CART have an average
 ID3 searches the whole dataset to create the whole tree. speed of processing and supports both continuous and
 It finds the leaf nodes thus enabling the test data to be nominal attribute data.
pruned and reducing the number of tests.
 The calculation time of ID3 is the linear function of the 2.3.3.1 Advantages of CART
product of the characteristic number and node number  CART can handle missing values automatically using
[9]. surrogate splits.

2.3.1.2 Disadvantages of ID3  Uses any combination of continuous/discrete variables.

 For a small sample, data may be over-fitted or over-  CART automatically performs variable selection.
classified.
 For making a decision, only one attribute is tested at an  CART can establish interactions among variables.
instant thus consuming a lot of time.

16
 International Journal of Computer Applications (0975 – 8887)
Volume 163 – No 8, April 2017

 CART does not vary according to the monotonic 3.1 Entropy

transformation of predictive variable [7]. Entropy is a measure of uncertainty associated with a random
variable. The entropy increases with the increase in
2.3.3.2 Disadvantages of CART uncertainty or randomness and decreases with a decrease in
 CART may have unstable decision trees. uncertainty or randomness. The value of entropy ranges from
 CART splits only by one variable. 0-1.
 Non-parametric. Entropy(D)= 𝑐𝑖=1 −pi log2(pi)
2.3.4 Random Forest where pi is the non-zero probability that an arbitrary tuple in
Random Forest decision tree was developed by Leo Breiman D belongs to class C and is estimated by |Ci,D|/|D|. A log
[13]. A Random Forest is a collection of simple tree function of base 2 is used because as stated above the entropy
predictors, such that each tree produces a response when a set is encoded in bits 0 and 1.
of predictor values are given as input. Similar to CART
algorithm. Random Forest also works both for classification 3.2 Information Gain
and regression problems. When solving classification ID3 uses information gain as its attribute selection measure.
problems, the response or the output appears in the form of a Claude Shannon studied the value or “information content” of
class membership, which associates or classifies, a set of messages and gave information gain as a measure in his
independent predictor values with the matching category Information Theory [5]. Information Gain is the difference
present in the dependent variable. When solving regression between the original information gain requirement (i.e. based
problems the output or response of the tree is an on just the proportion of classes) and the new requirement (i.e.
approximation of the dependent variables given the predictors. obtained after the partitioning of A).
Random Forest is a bagging tool that leverages the ability of 𝑣 |𝐷𝑗 |
Gain(D,A)=Entropy(D)- 𝑗 =1 |𝐷| Entropy(Dj)
multiple varied analyses, organization strategies, and
ensemble learning to supply correct models, perceptive Where,
variables, importance ranking and laser-sharp coverage on the
record-by-record basis for deep data understanding [8]. D: A given data partition

2.3.4.1 Advantages of Random Forest A: Attribute

 It recognizes outliers and anomalies in knowledgeable V: Suppose we partition the tuples in D on some attribute A
data. having v distinct values
 It is one of the most accurate learning algorithms
available. For many datasets, it produces highly D is split into v partition or subsets, {D1,D2,..Dj} were Dj
accurate classifiers. contains those tuples in D that have outcome aj of A.
 It gives an estimate of the important variables in The attribute that has the highest information gain is chosen.
classification.
3.3 Gain Ratio
2.3.4.2 Disadvantages of Random Forest The information gain measure is biased towards tests with
 Sometimes the classification made by Random Forests many outcomes. That is it prefers to select attributes having a
are difficult to be interpreted by humans. large number of values. As each partition is pure, the
 Random Forest sometimes overfits with datasets with information gain by partitioning is maximal. But such
noisy classification/regression tasks. partitioning cannot be used for classification.
3. ATTRIBUTE SELECTION C4.5(a successor of ID3) uses this attribute selection measure
named Gain Ratio which is an extension to the information
MEASURES gain. Gain Ratio differs from information gain, which
For selecting the splitting criterion that “best” separates the
measures the information with respect to a classification that
data partition, D, of class-labeled training tuples into
is acquired based on some partitioning [5]. Gain Ratio applies
individual classes, we used attribute selection measure which
kind of information gain using a “split information” value
is heuristic for such selection. If we were to split D into
defined as:
smaller partitions according to the outcomes of the splitting
criterion, ideally each partition would be pure(i.e., all the 𝑣 |𝐷𝑗 | |𝐷𝑗 |
SplitInfoA = - 𝑗 =1 |𝐷| log2 |𝐷|
tuples that fall into a given partition would belong into the
same class) [5]. The result of this scenario is actually the The Gain Ratio is then defined as:
“best” splitting criterion of all the criteria taken. Attribute
𝐺𝑎𝑖𝑛 (𝐴)
selection measure determines how to split the tuples at a given Gain Ratio(A)=𝑆𝑝𝑙𝑖𝑡𝐼𝑛𝑓𝑜𝐴 (𝐷)
node and are therefore also known as splitting rules.
A splitting attribute is selected which is the attribute having
The splitting attributes can be continuous-valued or it can be
the maximum Gain Ratio. The gain ratio becomes unstable if
restricted to binary trees. For continuous-valued attributes, a
the split information tends to 0. A constraint is added to avoid
split point must be determined as part of the splitting criterion
such condition, whereby the information gain of test selected
whereas for the binary trees a splitting subset must be
must be large- at least as great as the average gain over all
determined. The tree node for partition is labeled with the
tests examined [5].
splitting criterion, branches are grown for each outcome of
criterion and the tuples are partitioned accordingly. The most 3.4 Gini Index
popular attribute selection measures are – Entropy
Gini Index is an attribute selection measure used by the
(Information Gain), Gain Ratio and Gini Index.
CART decision tree algorithm. The Gini Index measures the
impurity D, a data partition or set of training tuples as:

17
 International Journal of Computer Applications (0975 – 8887)
Volume 163 – No 8, April 2017

𝑚 2
Gini(D)=1- 𝑖=1 𝑝i =0.443
Where pi is the probability that a tuple in D belongs to class =Ginisalary ϵ{high}(D).
Ci and is estimated by |Ci,D|/|D|. The sum is computed over m
We calculated the Gini index values for the other subsets also
classes. The attribute that reduces the impurity to the
and the result was 0.458 for the subset({low, high} and
maximum level (or has the minimum gini index) is selected as
{medium}) and it was 0.450 for the subset({medium, high}
the splitting attribute.
and {low}).
4. ILLUSTRATION SHOWING Therefore, the best binary split for salary attribute was found
ATTRIBUTESELECTION to be on ({low, medium} or {high}) because it minimizes the
MEASURES Gini index and has the value of 0.443.
In this paper, we have used the database of an Electronic store The attribute age when split over the subset({youth, senior})
to see whether a person buys a laptop or not. Figure 1 shows gives the minimum Gini index overall, with a reduction in
table having class-labeled training tuples from the electronic impurity of 0.459-0.357 = 0.102. Now according to the Gini
store. Each attribute taken is of a discrete value. The class- index, the binary split “age ∈ {youth, senior?} becomes the
labeled attribute buys _laptop, has two distinct values (yes, splitting criterion as it results in the maximum reduction in
no). Therefore there are two distinct classes and the value of impurities of tuples in D.
m is equal to 2.
Thus, the database of Electronics store shows that the attribute
We assume: age has the maximum or highest Information Gain and that
the age attribute also has the minimum Gini index, therefore,
 Class P: buys_laptop = “yes”
resulting in a maximum reduction in impurity of the tuple in
 Class N:buys_laptop=”no” this. Thus the decision tree for the given data is formed in
As there are 9 yes and 5 no in the buys_laptop attribute, Figure 3. by taking age as the splitting attribute.
therefore 9 tuples belong to class P and 5 tuples belong to
class N.
Entropy is calculated as:
9 9 5 5
Entropy(D)= -14 log2 - 14 log2 =0.940
14 14

Now the information gain is calculated as:

Gain(age,D)=Entropy(D)-
|𝑆𝑣|
𝑣∈{𝑦𝑜𝑢𝑡 𝑕,𝑚𝑖𝑑𝑑𝑙𝑒 −𝑎𝑔𝑒𝑑 ,𝑆𝑒𝑛𝑖𝑜𝑟 } |𝑆| Entropy(Sv)

5 4
=Entropy(D)- 14 Entropy(Syouth)- 14 Entropy(Smidddle-aged)-
5
Entropy(Ssenior)
14
Figure 1. Class‐labeled labeled training tuples from the
Gain(salary,D)=0.029 Electronics Store database.
Gain(graduate,D)=0.151
Gain(credit_raiting,D)=0.048

Calculation of Gain Ratio:

Firstly the SplitInfo is calculated. The salary attribute splits
the data in Figure 1 into three partitions high, low and
medium.
4 4 6 6 4 4
SplitInfosalary(D)=- 14 xlog2 -14 xlog2 - 14 xlog2
14 14 14
=1.557
As the Gain(salary)=0.029.
Therefore, GainRatio(salary)= 0.029/1.557 = 0.019
Calculation of Gini Index:
The Gini Index to compute the impurity of D for the database
in Figure 1 is:
9 5
Gini(D)=1- 14 2 − 14 2=0.459

The Gini Index value computed based on the salary attribute

partitioning is given as:
10 4
Ginisalary ϵ (low,medium)(D)= Gini(D1)+ 14 Gini(D2)
14
2 2 Figure 2. Tuples for the Electronic Store database
10 7 3 4 2 2 2 2
=14 1 − − + 14 1 − − partitioned according to the attribute age.
10 10 4 4

18
 International Journal of Computer Applications (0975 – 8887)
Volume 163 – No 8, April 2017

The attribute age has the highest information gain and thus We found that both C4.5 and CART are better than ID3 when
becomes the splitting attribute at the root node of the decision missing values are to be handled whereas ID3 cannot handle
tree. Branches are grown for each outcome of age. These missing or noisy data. But we also analyzed that ID3
tuples are shown partitioned according to the age. produces faster results. The paper also gives an idea of the
attribute selection measure used by various decision trees
A decision tree for the concept buys _laptop, indicating algorithms like ID3 algorithm uses information gain, the C4.5
whether a customer at an electronic store is likely to buy a algorithm uses gain ratio and CART algorithm uses GINI
laptop or not is shown in Figure 3. Each internal (non-leaf) Index as the attribute selection measure. The paper also gives
node of the decision tree represents a test on an attribute. Each the methods for calculation of these attribute selection
leaf node of the decision tree represents a class (either measures. In all, we find that these algorithms for decision
buys_laptop=”yes” or busy _laptop=” no” ). tree induction are to be used at different times according to
the situation.

7. REFERENCES
[1] Anuj Rathee and Robin Prakash Mathur, “Survey on
Decision Tree Classification algorithms for the
evaluation of Student Performance”,(IJCT),ISSN:2277-
3061,March-April,2013.
[2] Badr HSSINA, Abdelkarim MERBOUHA, Hanane
EZZIKOURI, Mohammed ERRITALI,“A comparative
study of decision tree ID3 and C4.5”, (IJACSA).
[3] Devinder Kaur, Rajiv Bedi and Dr. Sunil Kumar Gupta,
“Implementation of Enhanced Decision Tree Algorithm
on Traffic Accident Analysis”, (IJSRT), ISSN: 2379-
3686, 15th September 2015.
[4] G.Kesavaraj, Dr. S.Sukumaran, “A Study On
Figure 3. A decision tree for the concept buys _laptop in Classification Techniques in Data Mining”, IEEE-31661,
an electronic store. July 4-6, 2013.

5. APPLICATIONS OF DECISION [5] Han J., Kamber M., and Pei J. (2012) Data Mining:
Concepts and Techniques, 3rd edition. The Morgan
TREES IN VARIOUS AREAS OF Kaufmann Series in Data Management Systems, Jim
DATA MINING Gray, Series Editor.
The various decision tree algorithms find a large application
in real life. Some areas of application include: [6] Hemlata Chahal, “ID3 Modification and Implementation
in Data Mining”, International Journal of Computer
 E-Commerce: Used widely in the field of e-commerce, Applications (0975-8887), Volume 80-No7, October
decision tree helps to generate online catalog which is a 2013.
very important factor for the success of an e-commerce
website. [7] Jatinder Kaur and Jasmeet Singh Gurm, “Optimizing the
 Industry: Decision Tree algorithm is very useful for Accuracy of CART algorithm by Using Genetic
Algorithm”, (IJST), Volume 3 Issue 4, Jul-Aug, 2015.
producing quality control(faults identification) systems.
 Intelligent Vehicles: An important task for the [8] M.S. Mythili and Dr. A.R.Mohamed Shanavas, “An
development of intelligent vehicles is to find the lane Analysis of students’ performance using classification
boundaries of the road. Gonzalez and Ozguner have algorithms”, (IOSR-JCE), e-ISSN:2278-0661,p-
proposed lane detection for intelligent vehicles using ISSN:2278-8727, Jan. 2014.
decision trees.
 Medicine: Decision Tree is an important technique for [9] Qing-Yun Dai, Chun-ping Zhang and Hao Wu,“Research
medical research and practice. A decision tree is used of Decision Tree Classification Algorithm in Data
for diagnostic of various diseases. And is also used for Mining”, International Journal of Database Theory and
hard sound diagnosis. ApplicationVol.9, No.5(2016),pp.1-8.
 Business: Decision Trees also find use in the field of [10] T.Miranda Lakshmi, A.Martin, R.Mumtaj Begum, and
business where they are used for visualization of Dr. V.Prasanna Venkatesnan,“An Analysis on
probabilistic business models, used in CRM(Customer Performance of decision Tree Algorithms using
Relationship Management) and used for credit scoring Student’s Qualitative Data”, I.J. Modern Education and
for credit card users and for predicting loan risks in Computer Science, June 2013.
banks.
[11](2017, March 4), C4.5[Online].Available:
6. CONCLUSION http://en.wikipedia.org/wiki/C4.5_algorithm.
This paper analyses various decision tree algorithms that are [12](2017, March 4), ID3[Online].Available:
used in data mining. We found that each algorithm has got its http://en.wikipedia.org/wiki/ID3_algorithm.
own advantages and disadvantages as per our study. The
efficiency of various decision tree algorithms can be analyzed [13](2017, March 4), Random
based on their accuracy and the attribute selection measure Forest[Online].Available:http://en.wikipedia.org/wiki/Ra
used. The efficiency of the algorithms also depends on the ndom_Forest.
time taken information of the decision tree by the algorithm.

IJCATM : www.ijcaonline.org
19