Outline Training Examples for EnjoySport

[read Chapter 2]
[suggested exercises 2.2, 2.3, 2.4, 2.6]
Sky Temp Humid Wind Water Forecst EnjoySpt
Sunny Warm Normal Strong Warm Same Yes
• Learning from examples Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
• General-to-specific ordering over hypotheses Sunny Warm High Strong Cool Change Yes

• Version spaces and candidate elimination algorithm

• Picking new examples What is the general concept?

• The need for inductive bias

Note: simple approach assuming no noise, illustrates key concepts

1 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 2 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Representing Hypotheses Prototypical Concept Learning Task

Sky Temp Humid Wind Water Forecst EnjoySpt Sky Temp Humid Wind Water Forecst EnjoySpt
Sunny Warm Normal Strong Warm Same Yes Sunny Warm Normal Strong Warm Same Yes
Sunny Warm High Strong Warm Same Yes Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No Rainy Cold High Strong Warm Change No
Sunny Warm High Strong Cool Change Yes Sunny Warm High Strong Cool Change Yes

Many possible representations

• Given:
Here, h is conjunction of constraints on attributes
– Instances X: Possible days, each described by the attributes Sky, AirTemp, Humidity, Wind,
Each constraint can be Water, Forecast
– Target function c: EnjoySport : X → {0, 1}
• a specfic value (e.g., W ater = W arm)
– Hypotheses H: Conjunctions of literals. E.g.
• don’t care (e.g., “W ater =?”)
⟨?, Cold, High, ?, ?, ?⟩.
• no value allowed (e.g.,“Water=∅”)
For example, – Training examples D: Positive and negative examples of the target function
Sky AirTemp Humid Wind Water Forecst
⟨Sunny ? ? Strong ? Same⟩ ⟨x1 , c(x1 )⟩, . . . ⟨xm , c(xm )⟩

Classify everything negative • Determine: A hypothesis h in H such that h(x) = c(x) for all x in D.
⟨φ, φ, φ, φ, φ, φ⟩

Classify everything positive

⟨?, ?, ?, ?, ?, ?⟩

3 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 4 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Instance, Hypotheses, and More-
General-Than

Sky Temp Humid Wind Water Forecst EnjoySpt

Sunny Warm Normal Strong Warm Same Yes
Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
The inductive learning hypothesis: Any hypothesis found to approximate the target function well Sunny Warm High Strong Cool Change Yes
over a sufficiently large set of training examples will also approximate the target function well over
other unobserved examples.
Instances X Hypotheses H

Specific

h h
x1 1 3

h
x 2
2
General

x1= <Sunny, Warm, High, Strong, Cool, Same> h 1= <Sunny, ?, ?, Strong, ?, ?>
x = <Sunny, Warm, High, Light, Warm, Same> h = <Sunny, ?, ?, ?, ?, ?>
2 2
h = <Sunny, ?, ?, ?, Cool, ?>
3

5 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 6 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Find-S Algorithm Hypothesis Space Search by Find-S

Sky Temp Humid Wind Water Forecst EnjoySpt

1. Initialize h to the most specific hypothesis in H
Sunny Warm Normal Strong Warm Same Yes
2. For each positive training instance x Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
• For each attribute constraint ai in h Sunny Warm High Strong Cool Change Yes
If the constraint ai in h is satisfied by x
Then do nothing Instances X Hypotheses H
Else replace ai in h by the next more general constraint that is satisfied by x

3. Output hypothesis h
-
h0 Specific
x3
h1

h 2,3
x 1+ x+
2

x4+ h General
4

h = <∅, ∅, ∅, ∅, ∅, ∅>
0
x 1 = <Sunny Warm Normal Strong Warm Same>, + h1 = <Sunny Warm Normal Strong Warm Same>
x 2 = <Sunny Warm High Strong Warm Same>, + h2 = <Sunny Warm ? Strong Warm Same>
x 3 = <Rainy Cold High Strong Warm Change>, - h = <Sunny Warm ? Strong Warm Same>
3
x = <Sunny Warm High Strong Cool Change>, + h = <Sunny Warm ? Strong ? ? >
4 4

Generalisation only as far as necessary!

IF hypothesis is conjuction of attribute constraints

THEN guarantees most specific hypothesis in H that is consistent with the positive examples

also consistent with the negative examples provided target concept ∈ H, training examples correct

7 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 8 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Complaints about Find-S Version Spaces

• Can’t tell whether it has learned concept A hypothesis h is consistent with a set of training examples D of target concept c if and only
if h(x) = c(x) for each training example ⟨x, c(x)⟩ in D.
• Can’t tell when training data inconsistent
Consistent(h, D) ≡ (∀⟨x, c(x)⟩ ∈ D) h(x) = c(x)
• Picks a maximally specific h (why?)
• Depending on H, there might be several!

The version space, V SH,D , with respect to hypothesis space H and training examples D, is
the subset of hypotheses from H consistent with all training examples in D.

V SH,D ≡ {h ∈ H|Consistent(h, D)}

9 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 10 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 The List-Then-Eliminate Algorithm: Example Version Space

Sky Temp Humid Wind Water Forecst EnjoySpt

Sunny Warm Normal Strong Warm Same Yes
1. V ersionSpace ← a list containing every hypothesis in H Sunny Warm High Strong Warm Same Yes
2. For each training example, ⟨x, c(x)⟩ Rainy Cold High Strong Warm Change No
Sunny Warm High Strong Cool Change Yes
remove from V ersionSpace any hypothesis h for which h(x) ̸= c(x)
3. Output the list of hypotheses in V ersionSpace
S: { <Sunny, Warm, ?, Strong, ?, ?> }

<Sunny, ?, ?, Strong, ?, ?> <Sunny, Warm, ?, ?, ?, ?> <?, Warm, ?, Strong, ?, ?>

G: { <Sunny, ?, ?, ?, ?, ?>, <?, Warm, ?, ?, ?, ?> }

11 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 12 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Concept Learning As Search Representing Version Spaces

Sky Temp Humid Wind Water Forecst EnjoySpt The General boundary, G, of version space V SH,D is the set of its maximally general members
Sunny Warm Normal Strong Warm Same Yes
Sunny Warm High Strong Warm Same Yes The Specific boundary, S, of version space V SH,D is the set of its maximally specific members
Rainy Cold High Strong Warm Change No
Sunny Warm High Strong Cool Change Yes
Every member of the version space lies between these boundaries

V SH,D = {h ∈ H|(∃s ∈ S)(∃g ∈ G)(g ≥ h ≥ s)}

• # instances
where x ≥ y means x is more general or equal to y
– 3 × 2 × 2 × 2 × 2 × 2 = 96

• # classifications

– 296

• # syntactically distinct hypotheses

– 5120

• ⟨., ., φ, ., ., .⟩

– represents no instance

• # semantically distinct hypotheses

– 975

When large hypothesis space H (possibly infinite) then efficient search strategies required to find
hypothesis that best fits the data.

13 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 14 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Candidate Elimination Algorithm Example Trace

G ← maximally general hypotheses in H, ⟨?, ?, ?, ?, ?, ?⟩

S ← maximally specific hypotheses in H, ⟨φ, φ, φ, φ, φ, φ⟩
For each training example d, do S : {<Ø, Ø, Ø, Ø, Ø, Ø>}
• If d is a positive example
0
– Remove from G any hypothesis inconsistent with d
– For each hypothesis s in S that is not consistent with d
∗ Remove s from S
∗ Add to S all minimal generalizations h of s such that
1. h is consistent with d, and
2. some member of G is more general than h
∗ Remove from S any hypothesis that is more general than another hypothesis in S

• If d is a negative example

– Remove from S any hypothesis inconsistent with d

– For each hypothesis g in G that is not consistent with d
∗ Remove g from G
∗ Add to G all minimal specializations h of g such that
1. h is consistent with d, and
2. some member of S is more specific than h
∗ Remove from G any hypothesis that is less general than another hypothesis in G

G 0: {<?, ?, ?, ?, ?, ?>}

15 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 16 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 What Next Training Example? How Should These Be Classified?

S: { <Sunny, Warm, ?, Strong, ?, ?> } S: { <Sunny, Warm, ?, Strong, ?, ?> }

<Sunny, ?, ?, Strong, ?, ?> <Sunny, Warm, ?, ?, ?, ?> <?, Warm, ?, Strong, ?, ?> <Sunny, ?, ?, Strong, ?, ?> <Sunny, Warm, ?, ?, ?, ?> <?, Warm, ?, Strong, ?, ?>

G: { <Sunny, ?, ?, ?, ?, ?>, <?, Warm, ?, ?, ?, ?> } G: { <Sunny, ?, ?, ?, ?, ?>, <?, Warm, ?, ?, ?, ?> }

What query is best, most informative?

A query is satisfied by # / 2 of the hypotheses, and not satisfied by the other half ⟨Sunny W arm N ormal Strong Cool Change⟩

For example ⟨Rainy Cool N ormal Light W arm Same⟩

⟨Sunny, W arm, N ormal, Light, W arm, Same⟩

Note: Answer comes from nature or teacher ⟨Sunny W arm N ormal Light W arm Same⟩
# experiments: log2 |V S|

⟨Sunny Cold N ormal Strong W arm Same⟩

17 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 18 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 A Biased Hypothesis Space What Justifies this Inductive Leap?

Example Sky Temp Humid Wind Water Forecst EnjoySpt + ⟨Sunny W arm N ormal Strong Cool Change⟩
1 Sunny Warm Normal Strong Cool Change Yes
2 Cloudy Warm Normal Strong Cool Change Yes + ⟨Sunny W arm N ormal Light W arm Same⟩
3 Rainy Warm Normal Strong Cool Change No
S : ⟨Sunny W arm N ormal ? ? ?⟩

S2 : ⟨? W arm N ormal Strong Cool Change⟩

Why believe we can classify the unseen

⟨Sunny W arm N ormal Strong W arm Same⟩

19 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 20 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 An UNBiased Learner Inductive Bias

Consider
Idea: Choose H that expresses every teachable concept (i.e., H is the power set of X)
• concept learning algorithm L
Consider H ′ = disjunctions, conjunctions, negations over previous H. E.g.,
• instances X, target concept c
⟨Sunny W arm N ormal ? ? ?⟩ ∨ ¬⟨? ? ? ? ? Change⟩
• training examples Dc = {⟨x, c(x)⟩}
What are S, G in this case? • let L(xi , Dc ) denote the classification assigned to the instance xi by L after training on data Dc .
S←
Definition:
G←
The inductive bias of L is any minimal set of assertions B such that for any target concept c
and corresponding training examples Dc

(∀xi ∈ X)[(B ∧ Dc ∧ xi ) ⊢ L(xi , Dc )]

where A ⊢ B means A logically entails B

21 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 22 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Inductive Systems and Equivalent De- Three Learners with Different Biases
ductive Systems

Inductive system 1. Rote learner: Store examples, Classify x iff it matches previously observed example.

Classification of 2. Version space candidate elimination algorithm

Training examples Candidate new instance, or 3. Find-S

Elimination "don’t know"
Algorithm
New instance Using Hypothesis
Space H

Equivalent deductive system

Classification of
Training examples new instance, or
"don’t know"
Theorem Prover
New instance

Assertion " H contains

the target concept"

Inductive bias
made explicit

23 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 24 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Summary Points Decision Tree Learning

[read Chapter 3]
1. Concept learning as search through H [recommended exercises 3.1, 3.4]
2. General-to-specific ordering over H • Decision tree representation
3. Version space candidate elimination algorithm • ID3 learning algorithm
4. S and G boundaries characterize learner’s uncertainty • Entropy, Information gain
5. Learner can generate useful queries • Overfitting
6. Inductive leaps possible only if learner is biased
7. Inductive learners can be modelled by equivalent deductive systems

25 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 26 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Decision Tree for P layT ennis A Tree to Predict C-Section Risk

Learned from medical records of 1000 women

Outlook Negative examples are C-sections

[833+,167-] .83+ .17-

Fetal_Presentation = 1: [822+,116-] .88+ .12-
| Previous_Csection = 0: [767+,81-] .90+ .10-
Sunny Overcast Rain | | Primiparous = 0: [399+,13-] .97+ .03-
| | Primiparous = 1: [368+,68-] .84+ .16-
| | | Fetal_Distress = 0: [334+,47-] .88+ .12-
| | | | Birth_Weight < 3349: [201+,10.6-] .95+ .05-
Humidity Yes Wind | | | | Birth_Weight >= 3349: [133+,36.4-] .78+ .22-
| | | Fetal_Distress = 1: [34+,21-] .62+ .38-
| Previous_Csection = 1: [55+,35-] .61+ .39-
Fetal_Presentation = 2: [3+,29-] .11+ .89-
Fetal_Presentation = 3: [8+,22-] .27+ .73-

High Normal Strong Weak

No Yes No Yes

The instance
• Outlook = Sunny
• Temperature = Hot
• Humidity = High
• Wind = Strong
→ No

27 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 28 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Decision Trees When to Consider Decision Trees

Decision tree representation:

• Instances describable by attribute–value pairs
• Each internal node tests an attribute
• Target function is discrete valued
• Each branch corresponds to attribute value
• Disjunctive hypothesis may be required
• Each leaf node assigns a classification
• Possibly noisy training data

How would we represent:

Examples:
• ∧, ∨, XOR
• Equipment or medical diagnosis
• (A ∧ B) ∨ (C ∧ ¬D ∧ E)
• Credit risk analysis
• M of N
• Modeling calendar scheduling preferences

29 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 30 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Top-Down Induction of Decision Entropy
Trees
1.0
Main loop:
1. A ← the “best” decision attribute for next node
2. Assign A as decision attribute for node

Entropy(S)
3. For each value of A, create new descendant of node
4. Sort training examples to leaf nodes 0.5
5. If training examples perfectly classified, Then STOP, Else iterate over new leaf nodes

Which attribute is best?

[29+,35-] A1=? [29+,35-] A2=?

0.0 0.5 1.0
p
t f t f +

• S is a sample of training examples

• p⊕ is the proportion of positive examples in S
[21+,5-] [8+,30-] [18+,33-] [11+,2-]
• p⊖ is the proportion of negative examples in S
• Entropy measures the impurity of S

Entropy(S) ≡ −p⊕ log2 p⊕ − p⊖ log2 p⊖

31 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 32 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Entropy Information Gain

Entropy(S) = expected number of bits needed to encode class (⊕ or ⊖) of randomly drawn member of S Gain(S, A) = expected reduction in entropy due to sorting on A
(under the optimal, shortest-length code)

! |Sv |
Why? Gain(S, A) ≡ Entropy(S) − Entropy(Sv )
|S|
v∈V alues(A)
Information theory: optimal length code assigns − log2 p bits to message having probability p.

So, expected number of bits to encode ⊕ or ⊖ of random member of S:

[29+,35-] A1=? [29+,35-] A2=?
p⊕ (− log2 p⊕ ) + p⊖ (− log2 p⊖ )
t f t f
Entropy(S) ≡ −p⊕ log2 p⊕ − p⊖ log2 p⊖

with c-wise classification, we get [21+,5-] [8+,30-] [18+,33-] [11+,2-]

c
!
Entropy(S) = −pi log2 pi
i=1

33 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 34 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Training Examples Selecting the Next Attribute

Day Outlook Temperature Humidity Wind PlayTennis

Day Outlook Temperature Humidity Wind PlayTennis D1 Sunny Hot High Weak No
D1 Sunny Hot High Weak No D2 Sunny Hot High Strong No
D2 Sunny Hot High Strong No D3 Overcast Hot High Weak Yes
D3 Overcast Hot High Weak Yes D4 Rain Mild High Weak Yes
D4 Rain Mild High Weak Yes D5 Rain Cool Normal Weak Yes
D5 Rain Cool Normal Weak Yes D6 Rain Cool Normal Strong No
D6 Rain Cool Normal Strong No D7 Overcast Cool Normal Strong Yes
D7 Overcast Cool Normal Strong Yes D8 Sunny Mild High Weak No
D8 Sunny Mild High Weak No D9 Sunny Cool Normal Weak Yes
D9 Sunny Cool Normal Weak Yes D10 Rain Mild Normal Weak Yes
D10 Rain Mild Normal Weak Yes D11 Sunny Mild Normal Strong Yes
D11 Sunny Mild Normal Strong Yes D12 Overcast Mild High Strong Yes
D12 Overcast Mild High Strong Yes D13 Overcast Hot Normal Weak Yes
D13 Overcast Hot Normal Weak Yes D14 Rain Mild High Strong No
D14 Rain Mild High Strong No
Which attribute is the best classifier?

S: [9+,5-] S: [9+,5-]
E =0.940 E =0.940

Humidity Wind

High Normal Weak Strong

[3+,4-] [6+,1-] [6+,2-] [3+,3-]

E =0.985 E =0.592 E =0.811 E =1.00

Gain (S, Humidity ) Gain (S, Wind )

= .940 - (7/14).985 - (7/14).592 = .940 - (8/14).811 - (6/14)1.0
= .151 = .048

Gain(s, Outlook) = 0.246

Gain(s, T emperature) = 0.029

35 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 36 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 {D1, D2, ..., D14} Hypothesis Space Search by ID3
[9+,5−]

Outlook

Sunny Overcast Rain

+ – +

{D1,D2,D8,D9,D11} {D3,D7,D12,D13} {D4,D5,D6,D10,D14}

...
[2+,3−] [4+,0−] [3+,2−]
A2
A1
? Yes ? + – + + + – + –

...

A2 A2
Which attribute should be tested here?
+ – + – + – + –
A3 A4
Ssunny = {D1,D2,D8,D9,D11} –
+
Gain (Ssunny , Humidity) = .970 − (3/5) 0.0 − (2/5) 0.0 = .970

Gain (Ssunny , Temperature) = .970 − (2/5) 0.0 − (2/5) 1.0 − (1/5) 0.0 = .570 ... ...

Gain (Ssunny , Wind) = .970 − (2/5) 1.0 − (3/5) .918 = .019 ID3 algorithms perform a single to complex hill-climbing searching

The evaluation function = information gain

37 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 38 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Hypothesis Space Search by ID3 Inductive Bias in ID3

Note H is the power set of instances X

• Hypothesis space is complete!
→Unbiased?
– Target function surely in there...
Not really...
• Outputs a single hypothesis (which one?)
• Preference for short trees, and for those with high information gain attributes near the root
– Can’t play 20 questions...
• Bias is a preference for some hypotheses, rather than a restriction of hypothesis space H
• No back tracking
• Occam’s razor: prefer the shortest hypothesis that fits the data
– Local minima...

• Statisically-based search choices

– Robust to noisy data...

• Inductive bias: approx “prefer shortest tree”

39 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 40 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Occam’s Razor Overfitting in Decision Trees

Why prefer short hypotheses? Consider adding noisy training example #15:

Argument in favor: Sunny, Hot, N ormal, Strong, P layT ennis = N o

• Fewer short hyps. than long hyps.
What effect on earlier tree?
→ a short hyp that fits data unlikely to be coincidence
→ a long hyp that fits data might be coincidence
Outlook

Argument opposed:
• There are many ways to define small sets of hyps Sunny Overcast Rain
• e.g., all trees with a prime number of nodes that use attributes beginning with “Z”
• What’s so special about small sets based on size of hypothesis?? Humidity Yes Wind

High Normal Strong Weak

No Yes No Yes

41 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 42 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Overfitting Overfitting in Decision Tree Learning

Consider error of hypothesis h over

• training data: errortrain (h)
0.9
• entire distribution D of data: errorD (h)

Hypothesis h ∈ H overfits training data if there is an alternative hypothesis h′ ∈ H such that

0.85

errortrain (h) < errortrain (h′ ) 0.8

and
0.75
errorD (h) > errorD (h′ )

Accuracy
0.7

0.65

0.6 On training data

On test data
0.55

0.5
0 10 20 30 40 50 60 70 80 90 100
Size of tree (number of nodes)

43 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 44 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Avoiding Overfitting Reduced-Error Pruning

How can we avoid overfitting? Split data into training and validation set

• stop growing when data split not statistically significant Do until further pruning is harmful:
• grow full tree, then post-prune 1. Evaluate impact on validation set of pruning each possible node (plus those below it)
2. Greedily remove the one that most improves validation set accuracy
How to select “best” tree:
• Measure performance over training data
• produces smallest version of most accurate subtree
• Measure performance over separate validation data set
• What if data is limited?
• MDL: minimize size(tree) + size(misclassif ications(tree))

45 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 46 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Effect of Reduced-Error Pruning Rule Post-Pruning

0.9
1. Convert tree to equivalent set of rules

0.85 2. Prune each rule independently of others

3. Sort final rules into desired sequence for use
0.8

0.75 Perhaps most frequently used method (e.g., C4.5)

Accuracy

0.7

0.65

0.6 On training data

On test data
0.55 On test data (during pruning)

0.5
0 10 20 30 40 50 60 70 80 90 100
Size of tree (number of nodes)

47 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 48 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Converting A Tree to Rules Continuous Valued Attributes

Create a discrete attribute to test continuous

Outlook
• T emperature = 82.5
• (T emperature > 72.3) = t, f

Sunny Overcast Rain

Temperature: 40 48 60 72 80 90
PlayTennis: No No Yes Yes Yes No
Humidity Yes Wind

High Normal Strong Weak

No Yes No Yes

IF (Outlook = Sunny) ∧ (Humidity = High)

THEN P layT ennis = N o

IF (Outlook = Sunny) ∧ (Humidity = N ormal)

THEN P layT ennis = Y es

...

49 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 50 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Attributes with Many Values Attributes with Costs

Problem: Consider

• If attribute has many values, Gain will select it • medical diagnosis, BloodT est has cost $150

• Imagine using Date = Jun 3 1996 as attribute • robotics, W idth f rom 1f t has cost 23 sec.

How to learn a consistent tree with low expected cost?

One approach: use GainRatio instead One approach: replace gain by

Gain(S, A) • Tan and Schlimmer (1990)

GainRatio(S, A) ≡ Gain2 (S, A)
SplitInf ormation(S, A) .
Cost(A)
v

SplitInf ormation(S, A) ≡ −
-c
! |Si |
log2
|Si | • Nunez (1988)
|S| |S| 2Gain(S,A) − 1
i=1
(Cost(A) + 1)w
where Si is subset of S for which A has value vi where w ∈ [0, 1] determines importance of cost

51 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997 52 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997
 Unknown Attribute Values

What if some examples missing values of A?

Use training example anyway, sort through tree
• If node n tests A, assign most common value of A among other examples sorted to node n
• assign most common value of A among other examples with same target value
• assign probability pi to each possible value vi of A

– assign fraction pi of example to each descendant in tree

Classify new examples in same fashion

53 c
lecture slides for textbook Machine Learning, ⃝Tom M. Mitchell, McGraw Hill, 1997