Decision Trees

Jihoon Yang
Machine Learning Research Laboratory
Department of Computer Science & Engineering
Sogang University
Email: [email protected]
 Decision tree representation

• In the simplest case

– Each internal node tests on an attribute
– Each branch corresponds to an attribute value
– Each leaf node corresponds to a class label

• In general
– Each internal node corresponds to a test (on input instances)
with mutually exclusive and exhaustive outcomes – tests may be
univariate or multivariate
– Each branch corresponds to an outcome of a test
– Each leaf node corresponds to a class label

Department of Computer Science & Engineering

Machine Learning Research Laboratory 2
 Decision tree representation

11,01,10,00

Attributes Class 11,01,10,00 y

x y c x 1 0

1 0 11 10
1 1 1 A
E
01 00
x
2 0 1 B
11 01
1 x 0 1 x 0
a
m
10 00
p
l 3 1 0 A
e
c=A c=B 11 01 10 00
s
4 0 0 B
c=A c=B c=A c=B

Data set Tree 1 Tree 2

• Should we choose Tree 1 or Tree 2? Why?

Department of Computer Science & Engineering

Machine Learning Research Laboratory 3
 Learning decision tree classifiers

• Ockham’s razor recommends that we pick the simplest decision

tree that is consistent with the training set

• There are far too many trees that are consistent with a training set

• Searching for the simplest tree that is consistent with the training
set is not typically computationally feasible

• Solution
– Use a greedy algorithm – not guaranteed to find the simplest
tree – but works well in practice
– Or restrict the space of hypothesis to a subset of simple trees

Department of Computer Science & Engineering

Machine Learning Research Laboratory 4
 Information and Shannon Entropy

• Suppose we have a message that conveys the result of a random

experiment with m possible discrete outcomes, with probabilities
p1 , p2 ,..., pm

• The expected information content of such a message is called the

entropy of the probability distribution
m
H ( p1 , p2 ,..., pm )   pi I ( pi )
i 1

I ( pi )   log 2 pi provided pi  0
I ( pi )  0 otherwise

Department of Computer Science & Engineering

Machine Learning Research Laboratory 5
 Shannon’s entropy as a measure of information


Let P  ( p1 .... pn ) be a discrete probability distribution
The entropy of the distribution P is given by

 1
 n n
H P   pi log 2     pi log 2  pi 
i 1  pi  i 1

1 1 2
1 1 1 1
H  ,    pi log 2  pi     log 2      log 2    1 bit
2 2 i 1 2 2 2 2
2
H 0,1   pi log 2  pi   1I 1  0 I 0  0 bit
i 1

Department of Computer Science & Engineering

Machine Learning Research Laboratory 6
 The entropy for a binary variable

Department of Computer Science & Engineering

Machine Learning Research Laboratory 7
 Learning decision tree classifiers

• On average, the information needed to convey the class



membership of a random instance drawn from nature is H P

Nature
Training Data S S1 S2

Instance
Sm

Classifier  m     

H  P   i 1  pi  log 2  pi   H ( X )
     

Class label where P is an estimate of P
and X is a random variable with distribution P

Si is the multi-set of examples belonging to class Ci

Department of Computer Science & Engineering
Machine Learning Research Laboratory 8
 Learning decision tree classifiers

• The task of the learner then is to extract the needed information

from the training set and store it in the form of a decision tree for
classification

• Information gain based decision tree learner

Start with the entire training set at the root
Recursively add nodes to the tree
corresponding to tests that yield the
greatest expected reduction in entropy
(or the largest expected information gain)
until some termination criterion is met
( e.g. the training data at every leaf node has zero entropy )

Department of Computer Science & Engineering

Machine Learning Research Laboratory 9
 Learning decision tree classifiers – Example

Training Data
Instance Class label
Instances – I1 (t, d, l) +
ordered 3-tuples of attribute values I2 (s, d, l) +
corresponding to I3 (t, b, l) 
I4 (t, r, l) 
Height (tall, short) I5 (s, b, l) 
Hair (dark, blonde, red) I6 (t, b, w) +
Eye (blue, brown) I7 (t, d, w) +
I8 (s, b, w) +

Department of Computer Science & Engineering

Machine Learning Research Laboratory 10
 Learning decision tree classifiers – Example


I1…I8 3 3 5 5
H ( X )   log 2  log 2  0.954bits
8 8 8 8
Height
 3 3 2 2
H ( X | Height  t )   log 2  log 2  0.971bits
5 5 5 5
t s  2 2 1 1
H ( X | Height  s)   log 2  log 2  0.918bits
3 3 3 3

I1 I3 I4 I6 I7 I2 I5 I8
St Ss
 5  3  5 3
H ( X | Height )  H ( X | Height  t )  H ( X | Height  s)  (0.971)  (0.918)  0.95bits
8 8 8 8
   
Similarly, H ( X | Hair )  3 H ( X | Hair  d )  4 H ( X | Hair  b)  1 H ( Hair  r )  0.5bits and
8 8 8

H ( X | Eye )  0.607bits

Hair is the most informative because it yields the largest reduction in

entropy; Test on the value of Hair is chosen to correspond to the root of
the decision tree
Department of Computer Science & Engineering
Machine Learning Research Laboratory 11
 Learning decision tree classifiers – Example

Hair
d r
b
Compare the result with
+ Eye - Naïve Bayes

l w

- +

• In practice, we need some way to prune the tree to avoid overfitting

the training data

Department of Computer Science & Engineering

Machine Learning Research Laboratory 12
 Learning, generalization, overfitting

• Consider the error of a hypothesis h over

– Training data: ErrorTrain (h)
– Entire distribution D of data: ErrorD (h)

• Hypothesis h  H over fits training data if there is an alternative

hypothesis h' H such that

ErrorTrain( h)  ErrorTrain( h' )

and

ErrorD ( h)  ErrorD ( h' )

Department of Computer Science & Engineering

Machine Learning Research Laboratory 13
 Overfitting in decision tree learning
(e.g. diabetes dataset)

Department of Computer Science & Engineering

Machine Learning Research Laboratory 14
 Causes of overfitting

• As we move further away from the root, the data set used to choose
the best test becomes smaller  poor estimates of entropy

• Noisy examples can further exacerbate overfitting

Department of Computer Science & Engineering

Machine Learning Research Laboratory 15
 Minimizing overfitting

• Use roughly the same size sample at every node to estimate

entropy – when there is a large data set from which we can sample

• Stop when further split fails to yield statistically significant

information gain (estimated from validation set)

• Grow full tree, then prune

• Minimize size (tree) + size (exceptions (tree))

Department of Computer Science & Engineering

Machine Learning Research Laboratory 16
 Rule post-pruning

• Convert tree to equivalent

set of rules

IF (Outlook = Sunny)  (Humidity = High)

THEN PlayTennis = No
IF (Outlook = Sunny)  (Humidity = Normal)
THEN PlayTennis = Yes
...

Department of Computer Science & Engineering

Machine Learning Research Laboratory 17
 Rule post-pruning

1. Convert tree to equivalent set of rules

2. Prune each rule independently of others by dropping a condition

at a time if doing so does not reduce estimated accuracy (at the
desired confidence level)

3. Sort final rules in order of lowest to highest error for classifying

new instances

• Advantage – can potentially correct bad choices made close to the

root

• Post pruning based on validation set is the most commonly used

method in practice

Department of Computer Science & Engineering

Machine Learning Research Laboratory 18
 Classification of instances

• Unique classification – possible when each leaf has zero entropy

and there are no missing attribute values

• Most likely or probabilistic classification – based on distribution of

classes at a node when there are no missing attributes

Department of Computer Science & Engineering

Machine Learning Research Laboratory 19
 Handling different types of attribute values

• Types of attributes
– Nominal – values are names

– Ordinal – values are ordered

– Cardinal (numeric) – values are numbers (hence ordered)

– …

Department of Computer Science & Engineering

Machine Learning Research Laboratory 20
 Handling numeric attributes

Attribute T 40 48 50 54 60 70

Class N N Y Y Y N

48  50
T
60  70
Candidate splits T ? ?
2 2
2 4 3 3 1  1 
E( S | T  49 ?)  (0)      log2      log2   
6 6 4 4 4  4 

• Sort instances by value of numeric attribute under consideration

• For each attribute, find the test which yields the lowest entropy

• Greedily choose the best test across all attributes

Department of Computer Science & Engineering
Machine Learning Research Laboratory 21
 Handling numeric attributes

Axis-parallel split Oblique split

C2
C2 C1
C1

• Oblique splits cannot be realized by univariate tests

Department of Computer Science & Engineering

Machine Learning Research Laboratory 22
 Two-way versus multi-way splits

• Entropy criterion favors many-valued attributes

– Pathological behavior – what if in a medical diagnosis data set,
social security number is one of the candidate attributes?

• Solutions
– Only two-way splits (CART): A = value versus A = ~value

– Gain ratio (C4.5)

Gain(S , A)
GainRatio(S , A) 
SplitInformation(S , A)
Values ( A )
| Si | |S |
SplitInformation(S , A)   
i 1 |S |
log2 i
|S |

Department of Computer Science & Engineering

Machine Learning Research Laboratory 23
 See5/C5.0 [1997]

• Boosting

• New data types (e.g. dates), N/A values, variable misclassification

costs, attribute pre-filtering

• Unordered rulesets: all applicable rules are found and voted

• Improved scalability: multi-threading, multi-core/CPUs

Department of Computer Science & Engineering

Machine Learning Research Laboratory 24
 Summary of decision trees

• Simple

• Fast (linear in size of the tree, linear in the size of the training set,
linear in the number of attributes)

• Produce easy to interpret rules

• Good for generating simple predictive rules from data with lots of
attributes

• Popular extensions: GBDT(Friedman, 2001), XGBoost(Chen &

Guestrin, 2016), LightGBM(Ke et al., 2017)

Department of Computer Science & Engineering

Machine Learning Research Laboratory 25