CS685: Data Mining

Decision Trees and

Rule-Based Learners

Arnab Bhattacharya
[email protected]

Computer Science and Engineering,

Indian Institute of Technology, Kanpur
http://web.cse.iitk.ac.in/~cs685/

1st semester, 2018-19

Mon, Thu 1030-1145 at RM101

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 1 / 22

 Decision Trees
A decision tree is a tree structure used for classification
Each internal node represents a test on an attribute
Each branch represents an outcome of the test
Each leaf represents a class outcome
For a test object, its attributes are tested and a particular path is
followed to a leaf, which is deemed its class

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 2 / 22

 Decision Trees
A decision tree is a tree structure used for classification
Each internal node represents a test on an attribute
Each branch represents an outcome of the test
Each leaf represents a class outcome
For a test object, its attributes are tested and a particular path is
followed to a leaf, which is deemed its class
Really
interested
in change?

no yes

Can do
something Join politics
else?

no yes

Has someone Prefers money

in politics over shame
from family

yes no
no yes
Join politics No politics
No politics Join politics

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 2 / 22

 Constructing a Decision Tree
If all objects are in same class, label the leaf node with that class
The leaf is then pure

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 3 / 22

 Constructing a Decision Tree
If all objects are in same class, label the leaf node with that class
The leaf is then pure
Else, choose the “best” attribute to split
Determine splitting criterion based on splitting attribute
Indicates split point(s) or splitting subset(s)
Different measures of impurity to split a node
Separate objects into different branches according to split
Recursively, build tree for each split

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 3 / 22

 Constructing a Decision Tree
If all objects are in same class, label the leaf node with that class
The leaf is then pure
Else, choose the “best” attribute to split
Determine splitting criterion based on splitting attribute
Indicates split point(s) or splitting subset(s)
Different measures of impurity to split a node
Separate objects into different branches according to split
Recursively, build tree for each split
Stop when either
Leaf becomes pure
No more attributes to split – assign class through majority voting

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 3 / 22

 Constructing a Decision Tree
If all objects are in same class, label the leaf node with that class
The leaf is then pure
Else, choose the “best” attribute to split
Determine splitting criterion based on splitting attribute
Indicates split point(s) or splitting subset(s)
Different measures of impurity to split a node
Separate objects into different branches according to split
Recursively, build tree for each split
Stop when either
Leaf becomes pure
No more attributes to split – assign class through majority voting
Decision tree building is top-down and backtracking is not allowed

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 3 / 22

 Information Gain
Entropy impurity or information impurity

k
X
info(D) = − (pi log2 pi )
i=1

For n partitions into D1 , . . . , Dn , denoted by S

n
X
infoS (D) = (|Dj |/|D|)info(Dj )
j=1

Information gain is

gainS (D) = info(D) − infoS (D)

More the gain, better the split

Choose attribute and split point that maximizes gain
Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 4 / 22
 Gini Index
Variance impurity for two classes

var (D) = p1 .p2

For k classes, generalized to Gini index or Gini impurity

k
X k
X k
X
gini(D) = pi .pj = 1 − pi2
i=1 j=1,j6=i i=1

For n partitions into D1 , . . . , Dn , denoted by S

n
X
giniS (D) = (|Dj |/|D|)gini(Dj )
j=1

Less the gini index, better the split

Choose attribute and split point that minimizes gini index
Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 5 / 22
 Classification Error

Classification error or misclassification index

class(D) = 1 − max pi
i

This is the probability of misclassification when no more split is done

and majority voting is used
Find reduction in impurity by splitting
n
X
class(D) − classS (D) = class(D) − (|Dj |/|D|)class(Dj )
j=1

More the reduction in impurity, better the split

Choose attribute and split point that maximizes reduction

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 6 / 22

 Gain Ratio
Most impurity measures are biased towards multiway splits
Higher chance that a node becomes purer
Gain ratio counters it
For n partitions into D1 , . . . , Dn , denoted by S
Split information is defined as
n
X
splitinfoS (D) = − (|Dj |/|D|) log2 (|Dj |/|D|)
j=1

Similar to information measure, although just uses the number of

objects in each partition and not any class information
This is used to normalize information gain
gainratioS (D) = gainS (D)/splitinfoS (D)

Higher the gain ratio, better the split

Choose attribute and split point that maximizes gain ratio
Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 7 / 22
 Choosing a Split Point
If attribute is nominal

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Choosing a Split Point
If attribute is nominal
Each category denotes a new branch
If binary split is required,

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Choosing a Split Point
If attribute is nominal
Each category denotes a new branch
If binary split is required, use set membership testing
If attribute is ordinal

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Choosing a Split Point
If attribute is nominal
Each category denotes a new branch
If binary split is required, use set membership testing
If attribute is ordinal
Each category denotes a new branch
If binary split is required,

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Choosing a Split Point
If attribute is nominal
Each category denotes a new branch
If binary split is required, use set membership testing
If attribute is ordinal
Each category denotes a new branch
If binary split is required, use order information
If attribute is numeric

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Choosing a Split Point
If attribute is nominal
Each category denotes a new branch
If binary split is required, use set membership testing
If attribute is ordinal
Each category denotes a new branch
If binary split is required, use order information
If attribute is numeric
Sort all values and choose a (binary) split point
If multiway split is required,

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Choosing a Split Point
If attribute is nominal
Each category denotes a new branch
If binary split is required, use set membership testing
If attribute is ordinal
Each category denotes a new branch
If binary split is required, use order information
If attribute is numeric
Sort all values and choose a (binary) split point
If multiway split is required, choose multiple split points

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 8 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge
Under-fitting can also happen
Some thresholds are always needed to control these

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge
Under-fitting can also happen
Some thresholds are always needed to control these
Missing attributes

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge
Under-fitting can also happen
Some thresholds are always needed to control these
Missing attributes
Ignored while training
Most frequent class assumed when testing

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge
Under-fitting can also happen
Some thresholds are always needed to control these
Missing attributes
Ignored while training
Most frequent class assumed when testing
Node decisions are based on single attribute – monothetic trees

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge
Under-fitting can also happen
Some thresholds are always needed to control these
Missing attributes
Ignored while training
Most frequent class assumed when testing
Node decisions are based on single attribute – monothetic trees
Why not polythetic trees where decisions are based on multiple
attributes?

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Discussion

Over-fitting can happen

Tree needs to be pruned
Can use criteria such as chi-square test to stop splitting
Can use criteria such as information gain to merge
Under-fitting can also happen
Some thresholds are always needed to control these
Missing attributes
Ignored while training
Most frequent class assumed when testing
Node decisions are based on single attribute – monothetic trees
Why not polythetic trees where decisions are based on multiple
attributes?
Theoretically possible but practically too complex

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 9 / 22

 Variants of Decision Trees

Three main variants

ID3 (from Iterative Dichotomiser generation 3)
Multiway split
Uses information gain

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 10 / 22

 Variants of Decision Trees

Three main variants

ID3 (from Iterative Dichotomiser generation 3)
Multiway split
Uses information gain
C4.5
Evolved from ID3
Multiway split
Uses gain ratio

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 10 / 22

 Variants of Decision Trees

Three main variants

ID3 (from Iterative Dichotomiser generation 3)
Multiway split
Uses information gain
C4.5
Evolved from ID3
Multiway split
Uses gain ratio
CART (from Classification and Regression Trees)
Binary split
Uses gini index

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 10 / 22

 Rules
Rules are of the form
if condition then class
condition is a conjunct (i.e., logical AND) of tests on single attributes
If the condition holds, then the object is said to be from class
condition is called antecedent or precondition
class is called consequent
Example: if in family = yes AND can do something = no then
politics

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 11 / 22

 Rules
Rules are of the form
if condition then class
condition is a conjunct (i.e., logical AND) of tests on single attributes
If the condition holds, then the object is said to be from class
condition is called antecedent or precondition
class is called consequent
Example: if in family = yes AND can do something = no then
politics
Two important parameters of a rule
Coverage: Number of objects the rule applies to

coverage = |covers|/|D|

Accuracy: Number of correctly classified objects when rule is applied

accuracy = |correct|/|covers|

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 11 / 22

 Triggering and Firing of Rules

For every tuple, a rule that satisfies it is “triggered”

If for that tuple, it is the only rule, then it is “fired”

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 12 / 22

 Triggering and Firing of Rules

For every tuple, a rule that satisfies it is “triggered”

If for that tuple, it is the only rule, then it is “fired”
Otherwise, a conflict resolution strategy is devised

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 12 / 22

 Triggering and Firing of Rules

For every tuple, a rule that satisfies it is “triggered”

If for that tuple, it is the only rule, then it is “fired”
Otherwise, a conflict resolution strategy is devised
Size-based ordering: Rule with larger antecedent is invoked
More stringent, i.e., tougher

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 12 / 22

 Triggering and Firing of Rules

For every tuple, a rule that satisfies it is “triggered”

If for that tuple, it is the only rule, then it is “fired”
Otherwise, a conflict resolution strategy is devised
Size-based ordering: Rule with larger antecedent is invoked
More stringent, i.e., tougher
Class-based ordering: Two schemes
Consequent class is more frequent, i.e., according to order of prevalence
Consequent class has less misclassification
Within same class, there is arbitrary ordering

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 12 / 22

 Triggering and Firing of Rules

For every tuple, a rule that satisfies it is “triggered”

If for that tuple, it is the only rule, then it is “fired”
Otherwise, a conflict resolution strategy is devised
Size-based ordering: Rule with larger antecedent is invoked
More stringent, i.e., tougher
Class-based ordering: Two schemes
Consequent class is more frequent, i.e., according to order of prevalence
Consequent class has less misclassification
Within same class, there is arbitrary ordering
Rule-based ordering: Priority list according to some function based on
coverage, accuracy and size

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 12 / 22

 Triggering and Firing of Rules

For every tuple, a rule that satisfies it is “triggered”

If for that tuple, it is the only rule, then it is “fired”
Otherwise, a conflict resolution strategy is devised
Size-based ordering: Rule with larger antecedent is invoked
More stringent, i.e., tougher
Class-based ordering: Two schemes
Consequent class is more frequent, i.e., according to order of prevalence
Consequent class has less misclassification
Within same class, there is arbitrary ordering
Rule-based ordering: Priority list according to some function based on
coverage, accuracy and size
For a query tuple, the rule that satisfies it is invoked
If no such rule, then a default rule is invoked: if () then class i
Class i is the most abundant class

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 12 / 22

 Learning Rules from a Decision Tree

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 13 / 22

 Learning Rules from a Decision Tree

Every path is a rule

As verbose or complex as the decision tree itself

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 13 / 22

 Learning Rules from a Decision Tree

Every path is a rule

As verbose or complex as the decision tree itself
Rules are mutually exclusive and exhaustive

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 13 / 22

 Learning Rules from a Decision Tree

Every path is a rule

As verbose or complex as the decision tree itself
Rules are mutually exclusive and exhaustive
No need to order the rules

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 13 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?
Greedy learn-one-rule method learns the “best” rule given the current
set of tuples
General-to-specific strategy

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?
Greedy learn-one-rule method learns the “best” rule given the current
set of tuples
General-to-specific strategy
Starts with an empty antecedent
At each stage, every attribute (and every possible split) is considered
If the new rule has better quality than the old rule, it is retained
Decisions are greedy and are never backtracked

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?
Greedy learn-one-rule method learns the “best” rule given the current
set of tuples
General-to-specific strategy
Starts with an empty antecedent
At each stage, every attribute (and every possible split) is considered
If the new rule has better quality than the old rule, it is retained
Decisions are greedy and are never backtracked
Otherwise, the old rule is accepted

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?
Greedy learn-one-rule method learns the “best” rule given the current
set of tuples
General-to-specific strategy
Starts with an empty antecedent
At each stage, every attribute (and every possible split) is considered
If the new rule has better quality than the old rule, it is retained
Decisions are greedy and are never backtracked
Otherwise, the old rule is accepted
The next rule is then learnt

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?
Greedy learn-one-rule method learns the “best” rule given the current
set of tuples
General-to-specific strategy
Starts with an empty antecedent
At each stage, every attribute (and every possible split) is considered
If the new rule has better quality than the old rule, it is retained
Decisions are greedy and are never backtracked
Otherwise, the old rule is accepted
The next rule is then learnt
Rules are ordered according to their order of inception

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22

 Sequential Covering Algorithm
Sequential covering algorithm learns rules sequentially
Rules are learnt per class one-by-one
When a rule is learnt, all tuples covered by it are removed
Given a set of tuples, how is a rule learnt?
Greedy learn-one-rule method learns the “best” rule given the current
set of tuples
General-to-specific strategy
Starts with an empty antecedent
At each stage, every attribute (and every possible split) is considered
If the new rule has better quality than the old rule, it is retained
Decisions are greedy and are never backtracked
Otherwise, the old rule is accepted
The next rule is then learnt
Rules are ordered according to their order of inception
Variants are AQ, CN2 and RIPPER
Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 14 / 22
 Rule Quality
Accuracy is the most vital concern

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 15 / 22

 Rule Quality
Accuracy is the most vital concern
A rule with 90% accuracy and 80% coverage may not be better than
another rule with 95% accuracy and 10% coverage

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 15 / 22

 Rule Quality
Accuracy is the most vital concern
A rule with 90% accuracy and 80% coverage may not be better than
another rule with 95% accuracy and 10% coverage
Coverage also needs to be considered though

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 15 / 22

 Rule Quality
Accuracy is the most vital concern
A rule with 90% accuracy and 80% coverage may not be better than
another rule with 95% accuracy and 10% coverage
Coverage also needs to be considered though
Old rule R1 has a1 as antecedent
New rule R2 has a2 as antecedent
Let the number of tuples covered by a rule be denoted by Di
For the particular class in question, pi is the number of tuples
correctly classified, i.e., the consequent is this class
Correspondingly, ni is the number of negative tuples

Di = pi + ni

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 15 / 22

 Rule Quality
Accuracy is the most vital concern
A rule with 90% accuracy and 80% coverage may not be better than
another rule with 95% accuracy and 10% coverage
Coverage also needs to be considered though
Old rule R1 has a1 as antecedent
New rule R2 has a2 as antecedent
Let the number of tuples covered by a rule be denoted by Di
For the particular class in question, pi is the number of tuples
correctly classified, i.e., the consequent is this class
Correspondingly, ni is the number of negative tuples

Di = pi + ni

Four rule quality measures

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 15 / 22

 FOIL Gain

FOIL Gain measure proposed as part of the sequential covering

algorithm First Order Inductive Learner (FOIL) used in RIPPER
 
p2 p1
FOIL Gain(R1 → R2 ) = p2 × log2 − log2
D2 D1

Considers both coverage and accuracy

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 16 / 22

 Likelihood Ratio

Statistical test using the likelihood ratio statistic

m
X fi
LR = 2 fi log
ei
i=1

where m is the number of classes, fi and ei are the observed and

expected frequencies of tuples in each class
LR statistic has a chi-square distribution with m − 1 degrees of
freedom
The larger the statistic, the more deviated it is from the random rule,
and thus, the better

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 17 / 22

 Entropy

Entropy: Rule with less entropy is better

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 18 / 22

 M-Estimate

M-estimate measure considers the number of classes as well

pi + m.ci
m-estimate =
Di + m
where m is the number of classes and ci is the prior probability of
class Ci
If the prior probabilities are not known, replacing it by 1/m yields the
Laplacian estimate
pi + 1
Laplacian =
Di + m
The larger the estimate, the better is the rule

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 19 / 22

 Rule Pruning

General-to-specific strategy is susceptible to overfitting

Specific-to-general strategy first learns the most specific rule and then
prunes the antecedent
This is rule pruning

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 20 / 22

 Rule Pruning

General-to-specific strategy is susceptible to overfitting

Specific-to-general strategy first learns the most specific rule and then
prunes the antecedent
This is rule pruning
Each training instance starts as a rule
From a rule R1 , an antecedent is removed to yield rule R2
Measure of rule quality is FOIL Prune
pi − ni
FOIL Prune =
Di

If this measure is higher for R2 , then pruning is applied

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 20 / 22

 Frequent Pattern-Based Classifier

Uses the idea of frequent patterns from association rule mining

Assume a frequent pattern {A, B, C }
Suppose C is a class label
Suppose confidence of the rule {A, B} =⇒ C is high
Then presence of {A, B} is a good rule for classifying into class C

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 21 / 22

 Frequent Pattern-Based Classifier

Uses the idea of frequent patterns from association rule mining

Assume a frequent pattern {A, B, C }
Suppose C is a class label
Suppose confidence of the rule {A, B} =⇒ C is high
Then presence of {A, B} is a good rule for classifying into class C
This frequent pattern-based mining can be done for each class

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 21 / 22

 Discussion

Rules can be very verbose

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 22 / 22

 Discussion

Rules can be very verbose

Simple rules can only learn rectilinear boundaries

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 22 / 22

 Discussion

Rules can be very verbose

Simple rules can only learn rectilinear boundaries
Rules have an interpretation and can lead to descriptive models

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 22 / 22

 Discussion

Rules can be very verbose

Simple rules can only learn rectilinear boundaries
Rules have an interpretation and can lead to descriptive models
Can handle imbalance in class distribution very well

Arnab Bhattacharya ([email protected]) CS685: Decision Trees 2018-19 22 / 22