Decision Tree in Machine

Learning
Dr Abid Ali
Machine
Decision Learning
Trees Cours –
CS/SE 7th

PAF-IAST
Decision Trees
• Earlier, we decoupled the generation of the feature space from the
learning.
• Argued that we can map the given examples into another space, in
which the target functions are linearly separable.

• Do we always want to do it? Think about the Badges problem

• How do we determine what are good mappings?
What’s the best learning algorithm?
• The study of decision trees may shed some light on this.
• Learning is done directly from the given data representation.
• The algorithm ``transforms” the data itself. x2

x
This Lecture
• Decision trees for (binary) classification
– Non-linear classifiers

• Learning decision trees (ID3 algorithm)

– Greedy heuristic (based on information gain)
Originally developed for discrete features
– Some extensions to the basic algorithm

• Overfitting
– Some experimental issues
Introduction of Decision trees

4
Representing Data
• Think about a large table, N attributes, and assume you want to know
something about the people represented as entries in this table.
• E.g. own an expensive car or not;
• Simplest way: Histogram on the first attribute – own
• Then, histogram on first and second (own & gender)
• But, what if the # of attributes is larger: N=16
• How large are the 1-d histograms (contingency tables) ? 16 numbers
• How large are the 2-d histograms? 16-choose-2 = 120 numbers
• How many 3-d tables? 560 numbers
• With 100 attributes, the 3-d tables need 161,700 numbers
– We need to figure out a way to represent data in a better way, and figure
out what are the important attributes to look at first.
– Information theory has something to say about it – we will use it to better
represent the data.
Decision Trees
– A hierarchical data structure that represents data by
implementing a divide and conquer strategy
– Can be used as a non-parametric classification and
regression method
– Given a collection of examples, learn a decision tree that
represents it.
– Use this
C B
representation to classifyA new examples
The Representation
• Decision Trees are classifiers for instances represented as feature vectors
– color={red, blue, green} ; shape={circle, triangle, rectangle} ; label= {A, B, C}
• Nodes are tests for feature values Learning a
Evaluation of a
• There is one branch for each value of the feature
Decision Tree Decision Tree
• Leaves specify the category (labels)
• Can categorize instances into multiple disjoint categories Color

C B A
Shape B Shape

B A C B A
Expressivity of Decision Trees
• As Boolean functions they can represent any Boolean function.
• Can be rewritten as rules in Disjunctive Normal Form (DNF)
– Green ∧ Square  positive
– Blue ∧ Circle  positive
– Blue ∧ Square  positive Color
• The disjunction of these rules is equivalent to the Decision Tree
• What did we show? What is the hypothesis space here?
– 2 dimensions: color and shape
–
Shape
3 values each: color(red, blue, green), shape(triangle, square, circle)
B Shape
– |X| = 9: (red, triangle), (red, circle), (blue, square) …
– |Y| = 2: + and -
– |H| = 29
- + + + -
Decision Trees

• Output is a discrete category. Real

valued outputs are possible (regression
trees)
• There are efficient algorithms for Color
processing large amounts of data (but
not too many features)
• There are methods for handling noisy Shape B Shape
data (classification noise and attribute
noise) and for handling missing attribute
values
- + + + -
Decision Boundaries
• Usually, instances are represented as attribute-value pairs
(color=blue, shape = square, +)
• Numerical values can be used either by discretizing or by
using thresholds for splitting nodes
• In this case, the tree divides the features space into axis-
parallel rectangles, each labeled with one of the labels
Y
X<3
+ + +
no yes
7
Y>7 Y<5
+ + -
5
no yesno yes

- + - - + + X<1
no yes
1 3 X
+ -
Today’s key concepts
• Learning decision trees (ID3 algorithm)
– Greedy heuristic (based on information gain)
Originally developed for discrete features

• Overfitting
– What is it? How do we deal with it?

• Some extensions of DTs

• Principles of Experimental ML
Learning decision trees (ID3 algorithm)

12
Decision Trees
• Can represent any Boolean Function
• Can be viewed as a way to compactly represent a lot of
data.
• Natural representation: (20 questions)
• The evaluation of the Decision Tree Classifier is easy
Outlook
• Clearly, given data, there are
many ways to represent it as Sunny Overcast Rain
a decision tree. Humidity Wind
Yes
• Learning a good representation
from data is the challenge. High
No
Normal
Yes
StrongWeak
No Yes
Will I play tennis today?
• Features
– Outlook: {Sun, Overcast, Rain}
– Temperature: {Hot, Mild, Cool}
– Humidity: {High, Normal, Low}
– Wind: {Strong, Weak}

• Labels
– Binary classification task: Y = {+, -}
Will I play tennis today?
O T H W Play?
1 S H H W - Outlook: S(unny),
2 S H H S - O(vercast),
3 O H H W + R(ainy)
4 R M H W +
5 R C N W + Temperature: H(ot),
6 R C N S -
7 O C N S + M(edium),
8 S M H W -
C(ool)
9 S C N W +
1 R M N W + Humidity: H(igh),
0
1 S M N S +
N(ormal),
1 L(ow)
1 O M H S +
2 Wind: S(trong),
1 O H N W + W(eak)
3
1 R M H S -
Basic Decision Trees Learning Algorithm
O T H W Play?
1 S H H W - • Data is processed in Batch
2
3
S H
O H
H
H
S
W
-
+
(i.e. all the data available)
Algorithm?

4 R M H W + • Recursively build a decision

5 R C N W +
6 R C N S - tree top down.
7 O C N S + Outlook
8 S M H W -
9 S C N W +
1 R M N W +
Sunny Overcast Rain
0
1 S M N S + Humidity Yes Wind
1
1 O M H S + High Normal StrongWeak
2 No Yes No Yes
1 O H N W +
3
1 R M H S -
Basic Decision Tree Algorithm
• Let S be the set of Examples
– Label is the target attribute (the prediction)
– Attributes is the set of measured attributes
• ID3(S, Attributes, Label)
If all examples are labeled the same return a single node tree with Label
Otherwise Begin
A = attribute in Attributes that best classifies S (Create a Root node for tree)
for each possible value v of A
Add a new tree branch corresponding to A=v
Let Sv be the subset of examples in S with A=v
if Sv is empty: add leaf node with the common value of Label why?in S
Else: below this branch add the subtree For evaluation time
ID3(Sv, Attributes - {a}, Label)
End
Return Root
Picking the Root Attribute
• The goal is to have the resulting decision tree as
small as possible (Occam’s Razor)
– But, finding the minimal decision tree consistent with
the data is NP-hard
• The recursive algorithm is a greedy heuristic
search for a simple tree, but cannot guarantee
optimality.
• The main decision in the algorithm is the
selection of the next attribute to condition on.
Picking the Root Attribute
• Consider data with two Boolean attributes A
(A,B). 1 0
< (A=0,B=0), - >: 50 examples
< (A=0,B=1), - >: 50 examples
+ -
< (A=1,B=0), - >: 0 examples
< (A=1,B=1), + >: 100 examples
splitting on A
• What should be the first attribute we select?
– Splitting on A: we get purely labeled nodes. B
– Splitting on B: we don’t get purely labeled nodes. 1 0
– What if we have: <(A=1,B=0), - >: 3 examples?
A -
1 0
• (one way to think about it: # of queries required + -
to label a random data point)
splitting on B
Picking the Root Attribute
• Consider data with two Boolean attributes (A,B).
< (A=0,B=0), - >: 50 examples
< (A=0,B=1), - >: 50 examples
< (A=1,B=0), - >: 0 examples 3 examples
< (A=1,B=1), + >: 100 examples
• What should be the first attribute we select?
• Trees looks structurally similar; which attribute should we choose?

A B
1 0 1 0
Advantage A. But…
Need a way to quantify -
B things A -
1 0 100 1 0 5
• One way to think about it: # of queries required to
label a random data point. + - + - 3
• If we choose A we have less uncertainty about
the labels. 100 3 100 50
splitting on A splitting on B
Picking the Root Attribute
• The goal is to have the resulting decision tree as
small as possible (Occam’s Razor)
– The main decision in the algorithm is the selection of the
next attribute to condition on.
• We want attributes that split the examples to sets
that are relatively pure in one label; this way we
are closer to a leaf node.
– The most popular heuristics is based on information
gain, originated with the ID3 system of Quinlan.
Entropy
• Entropy (impurity, disorder) of a set of examples, S, relative to a
binary classification is:

• is the proportion of positive examples in S and

• is the proportion of negatives examples in S
– If all the examples belong to the same category: Entropy = 0
– If all the examples are equally mixed (0.5, 0.5): Entropy = 1
– Entropy = Level of uncertainty.
• In general, when pi is the fraction of examples labeled i:

• Entropy can be viewed as the number of bits required, on average,

to encode the class of labels. If the probability for + is 0.5, a single
bit is required for each example; if it is 0.8 – can use less then 1 bit.
22
Entropy
• Entropy (impurity, disorder) of a set of examples, S, relative to a binary
classification is:

• is the proportion of positive examples in S and

• is the proportion of negatives examples in S
– If all the examples belong to the same category: Entropy = 0
– If all the examples are equally mixed (0.5, 0.5): Entropy = 1
– Entropy = Level of uncertainty.

1 1 1

-- + -- + -- +
Entropy
(Convince yourself that the max value would be )
(Also note that the base of the log only introduce a constant factor; therefore, we’ll think
about base 2)

1 1 1
Information Gain
High Entropy – High level of
Uncertainty
Low Entropy – No Uncertainty.

• The information gain of an attribute a is the expected

reduction in entropy caused by partitioning on this
attribute
Outlook

• Where:
– Sv is the subset of S for which attribute a has value v, andSunny Overcast Rain
– the entropy of partitioning the data is calculated by weighing
the entropy of each partition by its size relative to the
original set

• Partitions of low entropy (imbalanced splits) lead to high

gain
• Go back to check which of the A, B splits is better
Will I play tennis today?
O T H W Play?
1 S H H W -
Outlook: S(unny),
2 S H H S - O(vercast),
3 O H H W + R(ainy)
4 R M H W +
5 R C N W + Temperature: H(ot),
6 R C N S -
7 O C N S + M(edium),
8 S M H W - C(ool)
9 S C N W +
1 R M N W + Humidity: H(igh),
0 N(ormal),
1 S M N S +
1
L(ow)
1 O M H S + Wind: S(trong),
2
1 O H N W + W(eak)
3
1 R M H S -
Will I play tennis today?
O T H W Play?
1 S H H W -
2 S H H S -
3 O H H W + calculate current entropy
4 R M H W +
5 R C N W + •
6 R C N S -
7 O C N S +
8 S M H W -
9 S C N W +  0.94
1 R M N W +
0
1 S M N S +
1
1 O M H S +
2
1 O H N W +
3
1 R M H S -
Information Gain: Outlook
¿
1
O
S
T
H
H
H
W
W
Play?
-
𝐺𝑎𝑖𝑛 ( 𝑆,𝑎 )=𝐸𝑛𝑡𝑟𝑜𝑝𝑦 ( 𝑆 ) − ∑ ¿ 𝑆𝑣 ∨
¿𝑆∨¿ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑆𝑣 )
¿¿
2 S H H S - 𝑣∈𝑣𝑎𝑙𝑢𝑒𝑠(𝑆)
3 O H H W + Outlook = sunny:
4 R M H W + Entropy(O = S) = 0.971
5 R C N W + Outlook = overcast:
6 R C N S -
Entropy(O = O) = 0
7 O C N S +
8 S M H W -
Outlook = rainy:
9 S C N W + Entropy(O = R) = 0.971
1 R M N W +
0 Expected entropy
1 S M N S + =
1 = (5/14)×0.971 + (4/14)×0 + (5/14)×0.971 = 0.694
1 O M H S +
2
1 O H N W + Information gain = 0.940 – 0.694 = 0.246
3
1 R M H S -
Information Gain: Humidity
¿
1
O
S
T
H
H
H
W
W
Play?
-
𝐺𝑎𝑖𝑛 ( 𝑆,𝑎 )=𝐸𝑛𝑡𝑟𝑜𝑝𝑦 ( 𝑆 ) − ∑ ¿ 𝑆𝑣 ∨
¿𝑆∨¿ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑆𝑣 )
¿¿
2 S H H S - 𝑣∈𝑣𝑎𝑙𝑢𝑒𝑠(𝑆)
3 O H H W + Humidity = high:
4 R M H W + Entropy(H = H) = 0.985
5 R C N W + Humidity = Normal:
6 R C N S -
Entropy(H = N) = 0.592
7 O C N S +
8 S M H W -
9 S C N W + Expected entropy
1 R M N W + =
0 = (7/14)×0.985 + (7/14)×0.592 = 0.7785
1 S M N S +
1 Information gain = 0.940 – 0.694 = 0.246
1 O M H S +
2
1 O H N W +
3
1 R M H S -
Which feature to split on?
O T H W Play?
1 S H H W -
2 S H H S -
3 O H H W +
4 R M H W + Information gain:
5 R C N W + Outlook: 0.246
6 R C N S - Humidity: 0.151
7 O C N S + Wind: 0.048
8 S M H W -
Temperature: 0.029
9 S C N W +
1 R M N W +
0 → Split on Outlook
1 S M N S +
1
1 O M H S +
2
1 O H N W +
3
1 R M H S -
An Illustrative Example (III)
Gain(S,Humidity)=0.151
Outlook Gain(S,Wind) = 0.048
Gain(S,Temperature) = 0.029
Gain(S,Outlook) = 0.246
An Illustrative Example (III)
O T H W Play?
Outlook 1S H H W -
2S H H S -
3O H H W +
4R M H W +
Sunny Overcast Rain 5R C N W +
1,2,8,9,11 3,7,12,13 4,5,6,10,14 6R C N S -
2+,3- 4+,0- 3+,2- 7O C N S +
8S M H W -
? Yes ?
9S C N W +
1R M N W +
0
1 S M N S +
1
1 O M H S +
2
1 O H N W +
3
An Illustrative Example (III)
O T H W Play?
Outlook
1S H H W -
2S H H S -
3O H H W +
4R M H W +
Sunny Overcast Rain 5R C N W +
1,2,8,9,11 3,7,12,13 4,5,6,10,14 6R C N S -
2+,3- 4+,0- 3+,2- 7O C N S +
? Yes ? 8S M H W -
9S C N W +
1R M N W +
Continue until:
• Every attribute is included in path, or,
0
• All examples in the leaf have same label 1 S M N S +
1
1 O M H S +
2
1 O H N W +
3
An Illustrative Example (IV)
Outlook
O T H W Play?
1 S H H W -
2 S H H S -
Sunny Overcast Rain 4 R M H W +
5 R C N W +
1,2,8,9,11 3,7,12,13 4,5,6,10,14 6 R C N S -
2+,3- 4+,0- 3+,2- 7 O C N S +
? Yes ? 8 S M H W -
9 S C N W +
1 R M N W +

Gain(S sunny , Humidity) .97-(3/5) 0-(2/5) 0 = .97 0
Gain(S sunny , Temp) .97- 0-(2/5) 1 = .57 1 S M N S +
1
Gain(S sunny , Wind) 
.97-(2/5) 1 - (3/5) .92= .02 1 O M H S +
2
Split on Humidity 1 O H N W +
3
1 R M H S -
An Illustrative Example (V)
Outlook

Sunny Overcast Rain

1,2,8,9,11 3,7,12,13 4,5,6,10,14
2+,3- 4+,0- 3+,2-
? Yes ?
An Illustrative Example (V)
Outlook

Sunny Overcast Rain

1,2,8,9,113,7,12,13 4,5,6,10,14
2+,3- 4+,0- 3+,2-
Humidity Yes ?

High Normal
No Yes
induceDecisionTree(S)
• 1. Does S uniquely define a class?
if all s ∈ S have the same label y: return S;

• 2. Find the feature with the most information gain:

i = argmax i Gain(S, Xi)

• 3. Add children to S:
for k in Values(Xi):
Sk = {s ∈ S | xi = k}
addChild(S, Sk)
induceDecisionTree(Sk)
return S;
An Illustrative Example (VI)
Outlook

Sunny Overcast Rain

1,2,8,9,113,7,12,13 4,5,6,10,14
2+,3- 4+,0- 3+,2-
Humidity Yes Wind

High Normal Strong Weak

No Yes No Yes
Hypothesis Space in Decision Tree Induction

• Conduct a search of the space of decision trees which

can represent all possible discrete functions. (pros and
cons)
• Goal: to find the best decision tree
– Best could be “smallest depth”
– Best could be “minimizing the expected number of tests”
• Finding a minimal decision tree consistent with a set of
data is NP-hard.
• Performs a greedy heuristic search: hill climbing without
backtracking
• Makes statistically based decisions using all data
History of Decision Tree Research
• Hunt and colleagues in Psychology used full search decision
tree methods to model human concept learning in the 60s
– Quinlan developed ID3, with the information gain heuristics in the late
70s to learn expert systems from examples
– Breiman, Freidman and colleagues in statistics developed CART
(classification and regression trees simultaneously)
• A variety of improvements in the 80s: coping with noise,
continuous attributes, missing data, non-axis parallel etc.
– Quinlan’s updated algorithm, C4.5 (1993) is commonly used (New: C5)
• Boosting (or Bagging) over DTs is a very good general
purpose algorithm
Overfitting

41
Example
Outlook

• Outlook = Sunny,
• Temp = Hot Sunny Overcast Rain
• Humidity = Normal 1,2,8,9,113,7,12,13 4,5,6,10,14
• Wind = Strong 2+,3- 4+,0- 3+,2-
• label: NO Humidity Yes Wind
• this example doesn’t exist in the
tree
High Normal Strong Weak
No Yes No Yes
Overfitting - Example
This can always be Outlook
done – may fit noise
or other
coincidental
• Outlook = Sunny, regularities Sunny Overcast Rain
• Temp = Hot 1,2,8,9,11 3,7,12,13 4,5,6,10,14
• Humidity = Normal 2+,3- 4+,0- 3+,2-
• Wind = Strong Humidity Yes Wind
• label: NO
• this example doesn’t exist in the
tree High Normal Strong Weak
No Wind No Yes

Strong Weak
No Yes
Our training data
The instance space
Overfitting the Data
• Learning a tree that classifies the training data perfectly may not lead to the
tree with the best generalization performance.
– There may be noise in the training data the tree is fitting
– The algorithm might be making decisions based on very little data
• A hypothesis h is said to overfit the training data if there is another hypothesis
h’, such that h has a smaller error than h’ on the training data but h has larger
error on the test data than h’.

accuracy
On training

On testing

Complexity of tree
Reasons for overfitting
• Too much variance in the training data
– Training data is not a representative sample
of the instance space
– We split on features that are actually irrelevant

• Too much noise in the training data

– Noise = some feature values or class labels are incorrect
– We learn to predict the noise

• In both cases, it is a result of our will to minimize the

empirical error when we learn, and the ability to do it (with
DTs)
Pruning a decision tree
• Prune = remove leaves and assign majority label
of the parent to all items
• Prune the children of node s if:
– all children are leaves, and
– the accuracy on the validation set does not decrease if
we assign the most frequent class label to all items at s.
Avoiding Overfitting
How can this be avoided with linear classifiers?
• Two basic approaches
– Pre-pruning: Stop growing the tree at some point during construction when it is
determined that there is not enough data to make reliable choices.
– Post-pruning: Grow the full tree and then remove nodes that seem not to have
sufficient evidence.
• Methods for evaluating subtrees to prune
– Cross-validation: Reserve hold-out set to evaluate utility
– Statistical testing: Test if the observed regularity can be dismissed as likely to occur
by chance
– Minimum Description Length: Is the additional complexity of the hypothesis smaller
than remembering the exceptions?
• This is related to the notion of regularization that we will see in other contexts –
keep the hypothesis simple.
Hand waving, for
now.
Next: a brief detour into explaining generalization and overfitting
Preventing Overfitting

h1 h2
The i.i.d. assumption
• Training and test items are independently and
identically distributed (i.i.d.):
– There is a distribution P(X, Y) from which the data D = {(x, y)}
is generated.
• Sometimes it’s useful to rewrite P(X, Y) as P(X)P(Y|X)
Usually P(X, Y) is unknown to us (we just know it exists)
– Training and test data are samples drawn from the same P(X,
Y): they are identically distributed
– Each (x, y) is drawn independently from P(X, Y)
Overfitting
On training data
Accurac
y this
Why
shape of On test data
curves?

Size of tree

• A decision tree overfits the training data when its

accuracy on the training data goes up but its
accuracy on unseen data goes down
Overfitting
Empirical
Error

??

Model complexity

• Empirical error (= on a given data set):

The percentage of items in this data set are
misclassified by the classifier f.
Overfitting

Empirical
Error

Model complexity

• Model complexity (informally):

How many parameters do we have to learn?
• Decision trees: complexity = #nodes
Overfitting

Expected
Error

Model complexity

• Expected error:
What percentage of items drawn from P(x,y) do
we expect to be misclassified by f?
• (That’s what we really care about –
generalization)
Variance of a learner (informally)

Variance

Model complexity

• How susceptible is the learner to minor changes in the training data?

– (i.e. to different samples from P(X, Y))
• Variance increases with model complexity
– Think about extreme cases: a hypothesis space with one function vs. all functions.
– Or, adding the “wind” feature in the DT earlier.
– The larger the hypothesis space is, the more flexible the selection of the chosen
hypothesis is as a function of the data.
– More accurately: for each data set D, you will learn a different hypothesis h(D), that
will have a different true error e(h); we are looking here at the variance of this
random variable.
Bias of a learner (informally)

Bias Model complexity

• How likely is the learner to identify the target hypothesis?

• Bias is low when the model is expressive (low empirical error)
• Bias is high when the model is (too) simple
– The larger the hypothesis space is, the easiest it is to be close to the true
hypothesis.
– More accurately: for each data set D, you learn a different hypothesis h(D), that has
a different true error e(h); we are looking here at the difference of the mean of this
random variable from the true error.
Impact of bias and variance

Expected
Error
Variance

Bias

Model complexity

• Expected error ≈ bias + variance

Model complexity

Expected
Error
Variance

Bias

Model complexity

Simple models: Complex models:

High bias and low variance High variance and low bias
Underfitting and Overfitting
Underfitti Overfittin
Expected ng g
Error
Variance

Bias
Model complexity

Simple models: Complex models:

High bias and low variance High variance and low bias
This can be made more accurate for some loss functions.
We will discuss a more precise and general theory that
trades expressivity of models with empirical error
Avoiding Overfitting
How can this be avoided with linear classifiers?
• Two basic approaches
– Pre-pruning: Stop growing the tree at some point during construction when
it is determined that there is not enough data to make reliable choices.
– Post-pruning: Grow the full tree and then remove nodes that seem not to
have sufficient evidence.
• Methods for evaluating subtrees to prune
– Cross-validation: Reserve hold-out set to evaluate utility
– Statistical testing: Test if the observed regularity can be dismissed as likely
to occur by chance
– Minimum Description Length: Is the additional complexity of the
hypothesis smaller than remembering the exceptions?
• This is related to the notion of regularization that we will see in
other contexts – keep the hypothesis simple.
Trees and Rules
• Decision Trees can be
represented as Rules Outlook
– (outlook = sunny) and
(humidity = normal) then YES
Sunny Overcast Rain
– (outlook = rain) and (wind = 1,2,8,9,113,7,12,13 4,5,6,10,14
strong) then NO 2+,3- 4+,0- 3+,2-
Humidity Yes Wind
• Sometimes Pruning can be
done at the rules level High Normal Strong Weak
– Rules are generalized by No Yes No Yes

erasing a condition (different!)

DT Extensions: Continuous
Attributes and Missing
Values

66
Continuous Attributes
• Real-valued attributes can, in advance, be discretized into
ranges, such as big, medium, small
• Alternatively, one can develop splitting nodes based on
thresholds of the form A<c that partition the data into examples
that satisfy A<c and A>=c.
– The information gain for these splits is calculated in the same way and
compared to the information gain of discrete splits.
• How to find the split with the highest gain?
• For each continuous feature A:
– Sort examples according to the value of A
– For each ordered pair (x,y) with different labels
• Check the mid-point as a possible threshold, i.e.
• Sa < x Sa >= y
Continuous Attributes
• Example:
– Length (L): 10 15 21 28 32 40 50
– Class: - + + - + + -
– Check thresholds: L < 12.5; L < 24.5; L < 45
– Subset of Examples= {…}, Split= k+,j-

• How to find the split with the highest gain ?

– For each continuous feature A:
• Sort examples according to the value of A
• For each ordered pair (x,y) with different labels
– Check the mid-point as a possible threshold. I.e,
– Sa < x, Sa >= y
Missing Values
• Diagnosis = < fever, blood_pressure,…,
blood_test=?,…>

• Many times values are not available for all

attributes during training or testing (e.g., medical
diagnosis)

• Training: evaluate Gain(S,a) where in some of the

examples a value for a is not given
 𝐺𝑎𝑖𝑛 ( 𝑆,𝑎 )=𝐸𝑛𝑡𝑟𝑜𝑝𝑦 ( 𝑆 ) − ∑ ¿ 𝑆𝑣 ∨ ¿
Missing Values ¿ 𝑆∨¿ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑆𝑣 )
¿¿

Other suggestions? Outlook Gain(S sunny ,Temp) .97- 0-(2/5) 1 = .57

Gain(S sunny , Humidity) 
 Fill in: assign the most likely value of Xi to s:
Sunny Overcast Rain argmax k P( Xi = k ):
1,2,8,9,11
3,7,12,13
4,5,6,10,14 Normal
2+,3- 4+,0- 3+,2-
 97-(3/5) Ent[+0,-3] -(2/5) Ent[+2,-0]
= .97
? Yes ?  Assign fractional counts P(Xi =k)
for each value of Xi to s
Humidity.97-(2.5/5)
Wind Ent[+0,-2.5] - (2.5/5)

Day Outlook Temperature PlayTennis
Ent[+2,-.5] < .97
1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
8 Sunny Mild ??? Weak No
9 Sunny Cool Normal Weak Yes
11 Sunny Mild Normal Strong Yes
Missing Values
• Diagnosis = < fever, blood_pressure,…,
blood_test=?,…>

• Many times values are not available for all

attributes during training or testing (e.g., medical
diagnosis)

• Training: evaluate Gain(S,a) where in some of the

examples a value for a is not given
• Testing: classify an example without knowing the
Missing Values
Outlook = Sunny, Temp = Hot, Humidity = ???, Wind = Strong, label = ??
Normal/High

Outlook = ???, Temp = Hot, Humidity = Normal, Wind = Strong, label = ??

Outlook1/3 Yes + 1/3 Yes +1/3 No = Yes

Sunny Overcast Rain

1,2,8,9,113,7,12,13 4,5,6,10,14
2+,3- 4+,0- 3+,2-
Humidity Yes Wind

High Normal Strong Weak

No Yes No Yes
72
Other Issues
• Attributes with different costs
– Change information gain so that low cost attribute are preferred
• Dealing with features with different # of values
• Alternative measures for selecting attributes
– When different attributes have different number of values
information gain tends to prefer those with many values
• Oblique Decision Trees
– Decisions are not axis-parallel
• Incremental Decision Trees induction
– Update an existing decision tree to account for new examples
incrementally (Maintain consistency?)
Summary: Decision Trees
• Presented the hypothesis class of Decision Trees
– Very expressive, flexible, class of functions

• Presented a learning algorithm for Decision Tress

– Recursive algorithm.
– Key step is based on the notion of Entropy

• Discussed the notion of overfitting and ways to address it within DTs

– In your problem set – look at the performance on the training vs. test

• Briefly discussed some extensions

– Real valued attributes
– Missing attributes

• Evaluation in machine learning

– Cross validation
– Statistical significance
Decision Trees as Features
• Rather than using decision trees to represent the target function it is becoming
common to use small decision trees as features
• When learning over a large number of features, learning decision trees is
difficult and the resulting tree may be very large
 (over fitting)
• Instead, learn small decision trees, with limited depth.
• Treat them as “experts”; they are correct, but only on a small region in the
domain. (what DTs to learn? same every time?)
• Then, learn another function, typically a linear function, over these as features.
• Boosting (but also other linear learners) are used on top of the small decision
trees. (Either Boolean, or real valued features)

• In HW1 you learn a linear classifier over DTs.

– Not learning the DTs sequentially; all are learned at once.
• How can you learn multiple DTs?
– Combining them using an SGD algorithm.
Experimental Machine Learning
• Machine Learning is an Experimental Field and we will spend
some time (in Problem sets) learning how to run experiments
and evaluate results
– First hint: be organized; write scripts
• Basics:
– Split your data into three sets:
• Training data (often 70-90%)
• Test data (often 10-20%)
• Development data (10-20%)
• You need to report performance on test data, but you are not
allowed to look at it.
– You are allowed to look at the development data (and use it to tune
parameters)
 Metrics
Methodologies
Statistical Significance

77
Metrics
• We train on our training data Train = {xi, yi}1,m
• We test on Test data.
• We often set aside part of the training data as a development set,
especially when the algorithms require tuning.
– In the HW we asked you to present results also on the Training; why?
• When we deal with binary classification we often measure
performance simply using Accuracy:

• Any possible problems with it?

78
Alternative Metrics
Positive negative
• If the Binary classification problem is biased
– In many problems most examples are negative
• Or, in multiclass classification
– The distribution over labels is often non-uniform
• Simple accuracy is not a useful metric.
– Often we resort to task specific metrics
• However one important example that is being used
often involves Recall and Precision

• Recall: # (positive identified = true

positives) Predicted positive
# (all positive)

• Precision: # (positive identified = true

positives)
# (predicted positive)
79
Example
Positive negative
• 100 examples, 5% are positive.

• Just say NO: your accuracy is 95%

– Recall = precision = 0
• Predict 4+, 96-; 2 of the +s are indeed positive
– Recall:2/5; Precision: 2/4

• Recall: # (positive identified = true

positives)
# (all positive)

• Precision: # (positive identified = true positives)

# (predicted positive)
80
 Confusion Matrix
• Given a dataset of P positive instances and N
negative instances:
Predicted Class
Yes No

Actual Class
The notion of a
confusion matrix Yes TP FN
can be usefully
extended to the No FP TN
multiclass case
(i,j) cell indicate • Imagine using classifier to identify positive cases
how many of the (i.e., for information retrieval)
i-labeled
examples were
predicted to be j
Probability that a Probability that a
randomly selected randomly selected
positive prediction is positive is identified
indeed positive
81
Relevant Metrics
• It makes sense to consider Recall
and Precision together or combine
them into a single metric.

• Recall-Precision Curve:

• F-Measure:
– A measure that combines precision
and recall is the harmonic mean of
precision and recall.

– F1 is the most commonly used

metric.
 Comparing Classifiers
Say we have two classifiers, C1 and C2, and want to
choose the best one to use for future predictions
Can we use training accuracy to choose between them?
• No!

• What about accuracy on test data?

83
N-fold cross validation
• Instead of a single test-training split:
train test

• Split data into N equal-sized parts

• Train and test N different classifiers

• Report average accuracy and standard deviation
of the accuracy
Evaluation: significance tests
• You have two different classifiers, A and B
• You train and test them on the same data set
using N-fold cross-validation
• For the n-th fold:
accuracy(A, n), accuracy(B, n)
pn = accuracy(A, n) - accuracy(B, n)
• Is the difference between A and B’s accuracies
significant?
Hypothesis testing
• You want to show that hypothesis H is true, based
on your data
– (e.g. H = “classifier A and B are different”)

• Define a null hypothesis H0

– (H0 is the contrary of what you want to show)

• H0 defines a distribution P(m |H0) over some

statistic
– e.g. a distribution over the difference in accuracy
between A and B
• Can you refute (reject) H0?
Rejecting H0
• H0 defines a distribution P(M |H0) over some statistic M
– (e.g. M= the difference in accuracy between A and B)
• Select a significance value S
– (e.g. 0.05, 0.01, etc.)
– You can only reject H0 if P(m |H0) ≤ S
• Compute the test statistic m from your data
– e.g. the average difference in accuracy over your N folds
• Compute P(m |H0)
• Refute H0 with p ≤ S if P(m |H0) ≤ S
Paired t-test
• Null hypothesis (H0; to be refuted):
– There is no difference between A and B, i.e. the
expected accuracies of A and B are the same
• That is, the expected difference (over all possible
data sets) between their accuracies is 0:
H0: E[pD] = 0

•We don’t know the true E[pD]

•N-fold cross-validation gives us N samples of pD
Paired t-test
• Null hypothesis H0: E[diffD] = μ = 0
• m: our estimate of μ based on N samples of diffD
m = 1/N n diffn
•The estimated variance S2:
S2 = 1/(N-1) 1,N (diffn – m)2
•Accept Null hypothesis at significance level a if the

following statistic lies in (-ta/2, N-1, +ta/2, N-1)

Decision Trees - Summary
• Hypothesis Space:
– Variable size (contains all functions)
– Deterministic; Discrete and Continuous attributes
• Search Algorithm
– ID3 - batch
– Extensions: missing values
• Issues:
– What is the goal?
– When to stop? How to guarantee good generalization?
• Did not address:
– How are we doing? (Correctness-wise, Complexity-wise)