RV College of Go, change the

RV College
Engineering world
of
Engineering
UNIT -3

▪ Supervised Learning
▪ Decision Tree Classifier

▪ Model Overfitting

▪ Model Selection

R V College of Engineering 1
RV College of
Engineering

UNIT-3 Supervised Learning

Basic Concepts, General
Framework for Classification

R V College of Engineering 2
 Classification: Definition
RV College of
Engineering

l Given a collection of records (training set )

– Each record is by characterized by a tuple
(x,y), where x is the attribute set and y is the
class label
◆ x: attribute, predictor, independent variable, input
◆ y: class, response, dependent variable, output

l Task:
– Learn a model that maps each attribute set x
into one of the predefined class labels y

R V College of Engineering 3
 RV College of
Engineering Examples of Classification Task

Task Attribute set, x Class label, y

Categorizing Features extracted from spam or non-spam

email email message header
messages and content
Identifying Features extracted from malignant or benign
tumor cells x-rays or MRI scans cells

Cataloging Features extracted from Elliptical, spiral, or

galaxies telescope images irregular-shaped
galaxies

R V College of Engineering 4
RV College of
General Approach for Building
Engineering Classification Model

R V College of Engineering 5
RV College of
General Approach for Building
Engineering Classification Model

R V College of Engineering 6
 Classification Techniques
RV College of
Engineering

Base Classifiers
– Decision Tree based Methods
– Rule-based Methods
– Nearest-neighbor
– Naïve Bayes and Bayesian Belief Networks
– Support Vector Machines
– Neural Networks, Deep Neural Nets

Ensemble Classifiers
– Boosting, Bagging, Random Forests

R V College of Engineering 7
 RV College of
Engineering

UNIT-3 : Decision Tree Classifier

A Basic Algorithm to Build a Decision Tree,
Methods for Expressing Attribute Test
Conditions, Measures for Selecting an
Attribute Test Condition, Algorithm for
Decision Tree Induction, Characteristics of
Decision Tree Classifiers,

R V College of Engineering 8
 RV College of
Engineering
Example of a Decision Tree

Splitting Attributes
Home Marital Annual Defaulted
ID
Owner Status Income Borrower
1 Yes Single 125K No Home
2 No Married 100K No Owner
Yes No
3 No Single 70K No
4 Yes Married 120K No NO MarSt
5 No Divorced 95K Yes Single, Divorced Married
6 No Married 60K No
Income NO
7 Yes Divorced 220K No
< 80K > 80K
8 No Single 85K Yes
9 No Married 75K No NO YES
10 No Single 90K Yes
10

Training Data Model: Decision Tree

R V College of Engineering 9
RV College of
Engineering Apply Model to Test Data
Test Data
Start from the root of tree.
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Single, Divorced Married

Income NO
< 80K > 80K

NO YES

R V College of Engineering 10
 Apply Model to Test Data
RV College of
Engineering

Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Single, Divorced Married

Income NO
< 80K > 80K

NO YES

R V College of Engineering 11
RV College of
Engineering
Apply Model to Test Data

Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Single, Divorced Married

Income NO
< 80K > 80K

NO YES

R V College of Engineering 12
 Apply Model to Test Data
RV College of
Engineering

Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Single, Divorced Married

Income NO
< 80K > 80K

NO YES

R V College of Engineering 13
 Apply Model to Test Data
RV College of
Engineering

Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Single, Divorced Married

Income NO
< 80K > 80K

NO YES

R V College of Engineering 14
 Apply Model to Test Data
RV College of
Engineering

Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Single, Divorced Married Assign Defaulted to
“No”
Income NO
< 80K > 80K

NO YES

R V College of Engineering 15
 RV College of
Another Example of Decision
Engineering Tree

MarSt Single,
Married Divorced
Home Marital Annual Defaulted
ID
Owner Status Income Borrower
NO Home
1 Yes Single 125K No
Yes Owner No
2 No Married 100K No
3 No Single 70K No NO Income
4 Yes Married 120K No < 80K > 80K
5 No Divorced 95K Yes
NO YES
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No There could be more than one tree that
fits the same data!
10 No Single 90K Yes
10

R V College of Engineering 16
RV College of
Decision Tree Classification
Engineering Task

Tid Attrib1 Attrib2 Attrib3 Class

Tree
1 Yes Large 125K No Induction
2 No Medium 100K No algorithm
3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply Decision
Tid Attrib1 Attrib2 Attrib3 Class
Model Tree
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ?

Deduction
14 No Small 95K ?

15 No Large 67K ?
10

Test Set

R V College of Engineering 17
 RV College of
Engineering Decision Tree Induction
Many Algorithms:
– Hunt’s Algorithm (one of the earliest)
– CART
– ID3, C4.5
– SLIQ,SPRINT

R V College of Engineering 18
 RV College of
General Structure of Hunt’s
Engineering Algorithm
Home Marital Annual Defaulted
l Let Dt be the set of training ID
Owner Status Income Borrower
records that reach a node t 1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
l General Procedure:
4 Yes Married 120K No
– If Dt contains records that 5 No Divorced 95K Yes

belong the same class yt, 6 No Married 60K No

then t is a leaf node 7 Yes Divorced 220K No

8 No Single 85K Yes
labeled as yt
9 No Married 75K No
– If Dt contains records that 10
10 No Single 90K Yes

belong to more than one

Dt
class, use an attribute test
to split the data into smaller
subsets. Recursively apply ?
the procedure to each
subset.

R V College of Engineering 19
 RV College of
Engineering
Hunt’s Algorithm
Home Marital Annual Defaulted
Home ID
Owner Status Income Borrower
Owner
1 Yes Single 125K No
Yes No
Defaulted = No 2 No Married 100K No
Defaulted = No Defaulted = No 3 No Single 70K No
(7,3)
(3,0) (4,3) 4 Yes Married 120K No

(a) (b) 5 No Divorced 95K Yes

6 No Married 60K No
7 Yes Divorced 220K No
Home
8 No Single 85K Yes
Owner
Home Yes No 9 No Married 75K No
Owner 10 No Single 90K Yes
Defaulted = No Marital 10

Yes No
Status
(3,0) Single,
Married
Defaulted = No Marital Divorced
Status
Defaulted = No
(3,0) Single,
Married
Annual
Divorced Income
(3,0)
Defaulted = Yes Defaulted = No < 80K >= 80K

Defaulted = No Defaulted = Yes

(1,3) (3,0)
(1,0) (0,3)
(c) (d)
R V College of Engineering 20
 RV College of
Engineering
Hunt’s Algorithm
Home Marital Annual Defaulted
Home ID
Owner Status Income Borrower
Owner
1 Yes Single 125K No
Yes No
Defaulted = No 2 No Married 100K No
Defaulted = No Defaulted = No 3 No Single 70K No
(7,3)
(3,0) (4,3) 4 Yes Married 120K No

(a) (b) 5 No Divorced 95K Yes

6 No Married 60K No
7 Yes Divorced 220K No
Home
8 No Single 85K Yes
Owner
Home Yes No 9 No Married 75K No
Owner 10 No Single 90K Yes
Defaulted = No Marital 10

Yes No
Status
(3,0) Single,
Married
Defaulted = No Marital Divorced
Status
Defaulted = No
(3,0) Single,
Married
Annual
Divorced Income
(3,0)
Defaulted = Yes Defaulted = No < 80K >= 80K

Defaulted = No Defaulted = Yes

(1,3) (3,0)
(1,0) (0,3)
(c) (d)
R V College of Engineering 21
 RV College of
Engineering
Hunt’s Algorithm
Home Marital Annual Defaulted
Home ID
Owner Status Income Borrower
Owner
1 Yes Single 125K No
Yes No
Defaulted = No 2 No Married 100K No
Defaulted = No Defaulted = No 3 No Single 70K No
(7,3)
(3,0) (4,3) 4 Yes Married 120K No

(a) (b) 5 No Divorced 95K Yes

6 No Married 60K No
7 Yes Divorced 220K No
Home
8 No Single 85K Yes
Owner
Home Yes No 9 No Married 75K No
Owner 10 No Single 90K Yes
Defaulted = No Marital 10

Yes No
Status
(3,0) Single,
Married
Defaulted = No Marital Divorced
Status
Defaulted = No
(3,0) Single,
Married
Annual
Divorced Income
(3,0)
Defaulted = Yes Defaulted = No < 80K >= 80K

Defaulted = No Defaulted = Yes

(1,3) (3,0)
(1,0) (0,3)
(c) (d)
R V College of Engineering 22
 RV College of
Engineering
Hunt’s Algorithm
Home Marital Annual Defaulted
Home ID
Owner Status Income Borrower
Owner
1 Yes Single 125K No
Yes No
Defaulted = No 2 No Married 100K No
Defaulted = No Defaulted = No 3 No Single 70K No
(7,3)
(3,0) (4,3) 4 Yes Married 120K No

(a) (b) 5 No Divorced 95K Yes

6 No Married 60K No
7 Yes Divorced 220K No
Home
8 No Single 85K Yes
Owner
Home Yes No 9 No Married 75K No
Owner 10 No Single 90K Yes
Defaulted = No Marital 10

Yes No
Status
(3,0) Single,
Married
Defaulted = No Marital Divorced
Status
Defaulted = No
(3,0) Single,
Married
Annual
Divorced Income
(3,0)
Defaulted = Yes Defaulted = No < 80K >= 80K

Defaulted = No Defaulted = Yes

(1,3) (3,0)
(1,0) (0,3)
(c) (d)
R V College of Engineering 23
 RV College of
Design Issues of Decision Tree
Engineering Induction
l How should training records be split?
– Method for expressing test condition
◆ depending on attribute types
– Measure for evaluating the goodness of a test
condition

l How should the splitting procedure stop?

– Stop splitting if all the records belong to the
same class or have identical attribute values
– Early termination
R V College of Engineering 24
 RV College of
Methods for Expressing Test
Engineering Conditions
l Depends on attribute types
– Binary
– Nominal
– Ordinal
– Continuous

R V College of Engineering 25
 RV College of
Test Condition for Nominal
Engineering Attributes
Multi-way split:
Marital
– Use as many partitions as Status
distinct values.

Single Divorced Married

Binary split:
– Divides values into two subsets

Marital Marital Marital

Status Status Status
OR OR

{Married} {Single, {Single} {Married, {Single, {Divorced}

Divorced} Divorced} Married}

R V College of Engineering 26
 RV College of
Test Condition for Ordinal
Engineering Attributes
l Multi-way split: Shirt
Size
– Use as many partitions
as distinct values
Small
Medium Large Extra Large

Shirt Shirt
l Binary split: Size Size

– Divides values into two

subsets
– Preserve order {Small,
Medium}
{Large,
Extra Large}
{Small} {Medium, Large,
Extra Large}

property among Shirt

attribute values Size
This grouping
violates order
property

{Small, {Medium,
Large} Extra Large}
R V College of Engineering 27
RV College of
Test Condition for
Engineering Continuous Attributes

Annual Annual
Income Income?
> 80K?
< 10K > 80K
Yes No

[10K,25K) [25K,50K) [50K,80K)

(i) Binary split (ii) Multi-way split

R V College of Engineering 28
 RV College of
Splitting Based on Continuous
Engineering Attributes
Different ways of handling
– Discretization to form an ordinal categorical
attribute
Ranges can be found by equal interval bucketing,
equal frequency bucketing (percentiles), or
clustering.
◆ Static – discretize once at the beginning

◆ Dynamic – repeat at each node

– Binary Decision: (A < v) or (A  v)

◆ consider all possible splits and finds the best cut
◆ can be more compute intensive
R V College of Engineering 29
 RV College of
How to determine the Best
Engineering Split

Before Splitting: 10 records of class 0,

10 records of class 1

Gender Car Customer

Type ID

Yes No Family Luxury c1 c20

c10 c11
Sports
C0: 6 C0: 4 C0: 1 C0: 8 C0: 1 C0: 1 ... C0: 1 C0: 0 ... C0: 0
C1: 4 C1: 6 C1: 3 C1: 0 C1: 7 C1: 0 C1: 0 C1: 1 C1: 1

Which test condition is the best?

R V College of Engineering 30
 RV College of
How to determine the Best
Engineering Split
l Greedy approach:
– Nodes with purer class distribution are
preferred

l Need a measure of node impurity:

C0: 5 C0: 9
C1: 5 C1: 1

High degree of impurity Low degree of impurity

R V College of Engineering 31
 RV College of
Engineering Measures of Node Impurity
l Gini Index 𝑐−1 Where 𝒑𝒊 𝒕 is the frequency
2 of class 𝒊 at node t, and 𝒄 is
𝐺𝑖𝑛𝑖 𝐼𝑛𝑑𝑒𝑥 = 1 − ෍ 𝑝𝑖 𝑡
the total number of classes
𝑖=0

l Entropy 𝑐−1

𝐸𝑛𝑡𝑟𝑜𝑝𝑦 = − ෍ 𝑝𝑖 𝑡 𝑙𝑜𝑔2 𝑝𝑖 (𝑡)

𝑖=0

l Misclassification error
𝐶𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑒𝑟𝑟𝑜𝑟 = 1 − max[𝑝𝑖 (𝑡)]

R V College of Engineering 32
 RV College of
Engineering Finding the Best Split
1. Compute impurity measure (P) before splitting
2. Compute impurity measure (M) after splitting
l Compute impurity measure of each child node
l M is the weighted impurity of child nodes

3. Choose the attribute test condition that

produces the highest gain

Gain = P - M

or equivalently, lowest impurity measure after splitting

(M)

R V College of Engineering 33
 RV College of
Engineering Finding the Best Split
Before Splitting: C0 N00
P
C1 N01

A? B?
Yes No Yes No

Node N1 Node N2 Node N3 Node N4

C0 N10 C0 N20 C0 N30 C0 N40

C1 N11 C1 N21 C1 N31 C1 N41

M11 M12 M21 M22

M1 M2
Gain = P – M1 vs P – M2
R V College of Engineering 34
 RV College of
Engineering Measure of Impurity: GINI
Gini Index for a given node 𝒕
𝑐−1

𝐺𝑖𝑛𝑖 𝐼𝑛𝑑𝑒𝑥 = 1 − ෍ 𝑝𝑖 𝑡 2

𝑖=0
Where 𝒑𝒊 𝒕 is the frequency of class 𝒊 at node 𝒕, and 𝒄 is the total
number of classes

– Maximum of 1 − 1/𝑐 when records are equally

distributed among all classes, implying the least
beneficial situation for classification
– Minimum of 0 when all records belong to one class,
implying the most beneficial situation for classification
– Gini index is used in decision tree algorithms such as
CART, SLIQ, SPRINT

R V College of Engineering 35
 RV College of
Engineering Measure of Impurity: GINI
Gini Index for a given node t :
𝑐−1

𝐺𝑖𝑛𝑖 𝐼𝑛𝑑𝑒𝑥 = 1 − ෍ 𝑝𝑖 𝑡 2

𝑖=0

– For 2-class problem (p, 1 – p):

◆ GINI = 1 – p2 – (1 – p)2 = 2p (1-p)

C1 0 C1 1 C1 2 C1 3
C2 6 C2 5 C2 4 C2 3
Gini=0.000 Gini=0.278 Gini=0.444 Gini=0.500

R V College of Engineering 36
 Computing Gini Index of a
Single Node
RV College of
Engineering

𝑐−1

𝐺𝑖𝑛𝑖 𝐼𝑛𝑑𝑒𝑥 = 1 − ෍ 𝑝𝑖 𝑡 2

𝑖=0

C1 0 P(C1) = 0/6 = 0 P(C2) = 6/6 = 1

C2 6 Gini = 1 – P(C1)2 – P(C2)2 = 1 – 0 – 1 = 0

C1 1 P(C1) = 1/6 P(C2) = 5/6

C2 5 Gini = 1 – (1/6)2 – (5/6)2 = 0.278

C1 2 P(C1) = 2/6 P(C2) = 4/6

C2 4 Gini = 1 – (2/6)2 – (4/6)2 = 0.444

R V College of Engineering 37
 RV College of
Computing Gini Index for a
Engineering Collection of Nodes
l When a node 𝑝 is split into 𝑘 partitions (children)
𝑘
𝑛𝑖
𝐺𝐼𝑁𝐼𝑠𝑝𝑙𝑖𝑡 = ෍ 𝐺𝐼𝑁𝐼(𝑖)
𝑛
𝑖=1

where, 𝑛𝑖 = number of records at child 𝑖,

𝑛 = number of records at parent node 𝑝.

R V College of Engineering 38
 RV College of Binary Attributes: Computing
Engineering
GINI Index
Splits into two partitions (child nodes)
Effect of Weighing partitions:
– Larger and purer partitions are sought
Parent
B? C1 7
Yes No C2 5
Gini = 0.486
Node N1 Node N2
Gini(N1)
= 1 – (5/6)2 – (1/6)2 N1 N2 Weighted Gini of N1 N2
= 0.278
C1 5 2 = 6/12 * 0.278 +
Gini(N2) C2 1 4 6/12 * 0.444
= 1 – (2/6)2 – (4/6)2 = 0.361
Gini=0.361
= 0.444 Gain = 0.486 – 0.361 = 0.125

R V College of Engineering 39
 RV College of
Categorical Attributes:
Engineering Computing Gini Index
l For each distinct value, gather counts for each class in
the dataset
l Use the count matrix to make decisions

Multi-way split Two-way split

(find best partition of values)

CarType CarType CarType

{Sports, {Family,
Family Sports Luxury {Family} {Sports}
Luxury} Luxury}
C1 1 8 1 C1 9 1 C1 8 2
C2 3 0 7 C2 7 3 C2 0 10
Gini 0.163 Gini 0.468 Gini 0.167

Which of these is the best?

R V College of Engineering 40
 RV College of
Continuous Attributes:
Engineering Computing Gini Index
l Use Binary Decisions based on one ID
Home
Owner
Marital
Status
Annual
Income
Defaulted
value
1 Yes Single 125K No
l Several Choices for the splitting value 2 No Married 100K No

– Number of possible splitting values 3 No Single 70K No

= Number of distinct values 4 Yes Married 120K No
5 No Divorced 95K Yes
l Each splitting value has a count matrix
6 No Married 60K No
associated with it
7 Yes Divorced 220K No
– Class counts in each of the 8 No Single 85K Yes
partitions, A ≤ v and A > v
9 No Married 75K No
l Simple method to choose best v 10 No Single 90K Yes

– For each v, scan the database to

10

Annual Income ?
gather count matrix and compute
its Gini index
≤ 80 > 80
– Computationally Inefficient!
Repetition of work. Defaulted Yes 0 3
Defaulted No 3 4

R V College of Engineering 41
 RV College of
Continuous Attributes: Computing
Engineering Gini Index...
l For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and computing gini index
– Choose the split position that has the least gini index

Cheat No No No Yes Yes Yes No No No No

Annual Income
Sorted Values 60 70 75 85 90 95 100 120 125 220
55 65 72 80 87 92 97 110 122 172 230
<= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= >
Yes 0 3 0 3 0 3 0 3 1 2 2 1 3 0 3 0 3 0 3 0 3 0

No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0

Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420

R V College of Engineering 42
 RV College of
Continuous Attributes: Computing
Engineering Gini Index...
l For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and computing gini index
– Choose the split position that has the least gini index

Cheat No No No Yes Yes Yes No No No No

Annual Income
Sorted Values 60 70 75 85 90 95 100 120 125 220
Split Positions 55 65 72 80 87 92 97 110 122 172 230
<= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= >
Yes 0 3 0 3 0 3 0 3 1 2 2 1 3 0 3 0 3 0 3 0 3 0

No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0

Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420

R V College of Engineering 43
 RV College of
Continuous Attributes:
Engineering Computing Gini Index...
l For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and computing gini index
– Choose the split position that has the least gini index

Cheat No No No Yes Yes Yes No No No No

Annual Income
Sorted Values 60 70 75 85 90 95 100 120 125 220
Split Positions 55 65 72 80 87 92 97 110 122 172 230
<= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= >
Yes 0 3 0 3 0 3 0 3 1 2 2 1 3 0 3 0 3 0 3 0 3 0

No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0

Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420

R V College of Engineering 44
 RV College of
Continuous Attributes: Computing
Engineering Gini Index...
l For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and computing gini index
– Choose the split position that has the least gini index

Cheat No No No Yes Yes Yes No No No No

Annual Income
Sorted Values 60 70 75 85 90 95 100 120 125 220
Split Positions 55 65 72 80 87 92 97 110 122 172 230
<= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= >
Yes 0 3 0 3 0 3 0 3 1 2 2 1 3 0 3 0 3 0 3 0 3 0

No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0

Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420

R V College of Engineering 45
 RV College of Continuous Attributes: Computing
Engineering
Gini Index...
l For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and computing gini index
– Choose the split position that has the least gini index

Cheat No No No Yes Yes Yes No No No No

Annual Income
Sorted Values 60 70 75 85 90 95 100 120 125 220
Split Positions 55 65 72 80 87 92 97 110 122 172 230
<= > <= > <= > <= > <= > <= > <= > <= > <= > <= > <= >
Yes 0 3 0 3 0 3 0 3 1 2 2 1 3 0 3 0 3 0 3 0 3 0

No 0 7 1 6 2 5 3 4 3 4 3 4 3 4 4 3 5 2 6 1 7 0

Gini 0.420 0.400 0.375 0.343 0.417 0.400 0.300 0.343 0.375 0.400 0.420

R V College of Engineering 46
 RV College of
Engineering Measure of Impurity: Entropy
l Entropy at a given node 𝒕
𝑐−1

𝐸𝑛𝑡𝑟𝑜𝑝𝑦 = − ෍ 𝑝𝑖 𝑡 𝑙𝑜𝑔2 𝑝𝑖 (𝑡)

𝑖=0
Where 𝒑𝒊 𝒕 is the frequency of class 𝒊 at node 𝒕, and 𝒄 is the total number
of classes

◆ Maximum of log 2 𝑐 when records are equally distributed

among all classes, implying the least beneficial situation for
classification
◆ Minimum of 0 when all records belong to one class,
implying most beneficial situation for classification

– Entropy based computations are quite similar to the GINI

index computations
R V College of Engineering 47
 Computing Entropy of a
Single Node
RV College of
Engineering

𝑐−1

𝐸𝑛𝑡𝑟𝑜𝑝𝑦 = − ෍ 𝑝𝑖 𝑡 𝑙𝑜𝑔2 𝑝𝑖 (𝑡)

𝑖=0

C1 0 P(C1) = 0/6 = 0 P(C2) = 6/6 = 1

C2 6 Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0

C1 1 P(C1) = 1/6 P(C2) = 5/6

C2 5 Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65

C1 2 P(C1) = 2/6 P(C2) = 4/6

C2 4 Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92

R V College of Engineering 48
 RV College of
Computing Information Gain
Engineering After Splitting
l Information Gain:
𝑘
𝑛𝑖
𝐺𝑎𝑖𝑛𝑠𝑝𝑙𝑖𝑡 = 𝐸𝑛𝑡𝑟𝑜𝑝𝑦 𝑝 − ෍ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑖)
𝑛
𝑖=1

Parent Node, 𝑝 is split into 𝑘 partitions (children)

𝑛𝑖 is number of records in child node 𝑖

– Choose the split that achieves most reduction (maximizes

GAIN)

– Used in the ID3 and C4.5 decision tree algorithms

– Information gain is the mutual information between the class

variable and the splitting variable

R V College of Engineering 49
 RV College of
Problem with large number of
Engineering partitions
Node impurity measures tend to prefer splits that
result in large number of partitions, each being
small but pure
Gender Car Customer
Type ID

Yes No Family Luxury c1 c20

c10 c11
Sports
C0: 6 C0: 4 C0: 1 C0: 8 C0: 1 C0: 1 ... C0: 1 C0: 0 ... C0: 0
C1: 4 C1: 6 C1: 3 C1: 0 C1: 7 C1: 0 C1: 0 C1: 1 C1: 1

– Customer ID has highest information gain

because entropy for all the children is zero
R V College of Engineering 50
 RV College of
Engineering
Gain Ratio

l Gain Ratio:
𝑘
𝐺𝑎𝑖𝑛𝑠𝑝𝑙𝑖𝑡 𝑛𝑖 𝑛𝑖
𝐺𝑎𝑖𝑛 𝑅𝑎𝑡𝑖𝑜 = 𝑆𝑝𝑙𝑖𝑡 𝐼𝑛𝑓𝑜 = − ෍ 𝑙𝑜𝑔2
𝑆𝑝𝑙𝑖𝑡 𝐼𝑛𝑓𝑜 𝑛 𝑛
𝑖=1

Parent Node, 𝑝 is split into 𝑘 partitions (children)

𝑛𝑖 is number of records in child node 𝑖

– Adjusts Information Gain by the entropy of the partitioning

(𝑆𝑝𝑙𝑖𝑡 𝐼𝑛𝑓𝑜).
◆ Higher entropy partitioning (large number of small partitions) is
penalized!
– Used in C4.5 algorithm
– Designed to overcome the disadvantage of Information Gain

R V College of Engineering 51
 RV College of
Engineering
Gain Ratio

l Gain Ratio:
𝑘
𝐺𝑎𝑖𝑛𝑠𝑝𝑙𝑖𝑡 𝑛𝑖 𝑛𝑖
𝐺𝑎𝑖𝑛 𝑅𝑎𝑡𝑖𝑜 = 𝑆𝑝𝑙𝑖𝑡 𝐼𝑛𝑓𝑜 = ෍ 𝑙𝑜𝑔2
𝑆𝑝𝑙𝑖𝑡 𝐼𝑛𝑓𝑜 𝑛 𝑛
𝑖=1

Parent Node, 𝑝 is split into 𝑘 partitions (children)

𝑛𝑖 is number of records in child node 𝑖

CarType CarType CarType

{Sports, {Family,
Family Sports Luxury {Family} {Sports}
Luxury} Luxury}
C1 1 8 1 C1 9 1 C1 8 2
C2 3 0 7 C2 7 3 C2 0 10
Gini 0.163 Gini 0.468 Gini 0.167

SplitINFO = 1.52 SplitINFO = 0.72 SplitINFO = 0.97

R V College of Engineering 52
 RV College of
Measure of Impurity:
Engineering Classification Error

l Classification error at a node 𝑡

𝐸𝑟𝑟𝑜𝑟 𝑡 = 1 − max[𝑝𝑖 𝑡 ]
𝑖

– Maximum of 1 − 1/𝑐 when records are equally

distributed among all classes, implying the least
interesting situation
– Minimum of 0 when all records belong to one class,
implying the most interesting situation

R V College of Engineering 53
 RV College of
Computing Error of a Single
Engineering Node

𝐸𝑟𝑟𝑜𝑟 𝑡 = 1 − max[𝑝𝑖 𝑡 ]
𝑖

C1 0 P(C1) = 0/6 = 0 P(C2) = 6/6 = 1

C2 6 Error = 1 – max (0, 1) = 1 – 1 = 0

C1 1 P(C1) = 1/6 P(C2) = 5/6

C2 5 Error = 1 – max (1/6, 5/6) = 1 – 5/6 = 1/6

C1 2 P(C1) = 2/6 P(C2) = 4/6

C2 4 Error = 1 – max (2/6, 4/6) = 1 – 4/6 = 1/3

R V College of Engineering 54
 Comparison among Impurity
Measures
RV College of
Engineering

For a 2-class problem:

R V College of Engineering 55
 RV College of Misclassification Error vs Gini
Engineering
Index
A? Parent
C1 7
Yes No
C2 3
Node N1 Node N2 Gini = 0.42

Gini(N1) N1 N2
= 1 – (3/3)2 – (0/3)2 Gini(Children)
C1 3 4 = 3/10 * 0
=0
C2 0 3 + 7/10 * 0.489
Gini(N2) Gini=0.342 = 0.342
= 1 – (4/7)2 – (3/7)2
= 0.489 Gini improves but
error remains the
same!!

R V College of Engineering 56
RV College of Misclassification Error vs Gini
Engineering
Index
A? Parent
C1 7
Yes No
C2 3
Node N1 Node N2 Gini = 0.42

N1 N2 N1 N2
C1 3 4 C1 3 4
C2 0 3 C2 1 2
Gini=0.342 Gini=0.416

Misclassification error for all three cases = 0.3 !

R V College of Engineering 57
 RV College of
Decision Tree Based
Engineering Classification
l Advantages:
– Relatively inexpensive to construct
– Extremely fast at classifying unknown records
– Easy to interpret for small-sized trees
– Robust to noise (especially when methods to avoid overfitting are
employed)
– Can easily handle redundant attributes
– Can easily handle irrelevant attributes (unless the attributes are
interacting)
l Disadvantages: .
– Due to the greedy nature of splitting criterion, interacting attributes (that
can distinguish between classes together but not individually) may be
passed over in favor of other attributed that are less discriminating.
– Each decision boundary involves only a single attribute

R V College of Engineering 58
 RV College of
Engineering
Handling interactions

+ : 1000 instances Entropy (X) : 0.99

Entropy (Y) : 0.99
o : 1000 instances
Y

R V College of Engineering 59
RV College of Handling interactions
Engineering

R V College of Engineering 60
 RV College of
Handling interactions given irrelevant
Engineering
attributes

+ : 1000 instances Entropy (X) : 0.99

Entropy (Y) : 0.99
o : 1000 instances Entropy (Z) : 0.98
Y
Adding Z as a noisy Attribute Z will be
attribute generated chosen for splitting!
from a uniform
distribution
X

R V College of Engineering 61
RV College of
Engineering

UNIT-3: Model Overfitting

Reasons for Model Overfitting
 RV College of Classification Errors
Engineering

Training errors: Errors committed on the training set

Test errors: Errors committed on the test set

Generalization errors: Expected error of a model over random selection of records from same distribution

Tid Attrib1 Attrib2 Attrib3 Class Learning

1 Yes Large 125K No
algorithm
2 No Medium 100K No

3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply
Tid Attrib1 Attrib2 Attrib3 Class Model
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ? Deduction

14 No Small 95K ?

15 No Large 67K ?
10

Test Set
RV College of
Engineering
Example Data Set

Two class problem:

+ : 5400 instances
• 5000 instances generated
from a Gaussian centered at
(10,10)

• 400 noisy instances added

o : 5400 instances
• Generated from a uniform
distribution

10 % of the data used for

training and 90% of the
data used for testing
Increasing number of nodes in Decision Trees
RV College of
Engineering
RV College of
Engineering
Decision Tree with 4 nodes

Decision Tree

Decision boundaries on Training data

Decision Tree with 50 nodes
RV College of
Engineering

Decision Tree

Decision boundaries on Training data

RV College of
Engineering Which tree is better?

Decision Tree with 4 nodes

Which tree is better ?

Decision Tree with 50 nodes
 RV College of
Model Underfitting and
Engineering Overfitting

•As the model becomes more and more complex, test errors can start increasing even
though training error may be decreasing

Underfitting: when model is too simple, both training and test errors are large
Overfitting: when model is too complex, training error is small but test error is large
 RV College of
Model Overfitting – Impact of Training
Engineering
Data Size

Using twice the number of data instances

• Increasing the size of training data reduces the difference between training and
testing errors at a given size of model
 RV College of
Model Overfitting – Impact of Training
Engineering
Data Size

Decision Tree with 50 nodes Decision Tree with 50 nodes

Using twice the number of data instances

• Increasing the size of training data reduces the difference between training and
testing errors at a given size of model
 RV College of
Engineering Reasons for Model Overfitting
Not enough training data

High model complexity

– Multiple Comparison Procedure
 RV College of Effect of Multiple Comparison
Procedure
Engineering

Consider the task of predicting whether Day 1 Up

stock market will rise/fall in the next 10 Day 2 Down
trading days
Day 3 Down
Day 4 Up
Random guessing:
Day 5 Down
P(correct) = 0.5 Day 6 Down
Day 7 Up
Make 10 random guesses in a row: Day 8 Up
Day 9 Up
10  10  10 
  +   +   Day 10 Down
P(# correct  8) =  8   9  10 
= 0.0547
10
2
 RV College of
Effect of Multiple
Engineering Comparison Procedure
Approach:
– Get 50 analysts
– Each analyst makes 10 random guesses
– Choose the analyst that makes the most
number of correct predictions

Probability that at least one analyst makes at

least 8 correct predictions
P(# correct  8) = 1 − (1 − 0.0547 )50 = 0.9399
 RV College of Effect of Multiple
Engineering
Comparison Procedure
Many algorithms employ the following greedy strategy:
– Initial model: M
– Alternative model: M’ = M  ,
where  is a component to be added to the model
(e.g., a test condition of a decision tree)
– Keep M’ if improvement, (M,M’) > 

Often times,  is chosen from a set of alternative

components,  = {1, 2, …, k}

If many alternatives are available, one may inadvertently

add irrelevant components to the model, resulting in
model overfitting
 RV College of
Effect of Multiple
Engineering Comparison - Example

Use additional 100 noisy variables

generated from a uniform distribution
along with X and Y as attributes.

Use 30% of the data for training and

70% of the data for testing
Using only X and Y as attributes
 RV College of
Engineering
Notes on Overfitting

Overfitting results in decision trees that are more

complex than necessary

Training error does not provide a good estimate

of how well the tree will perform on previously
unseen records

Need ways for estimating generalization errors

RV College of
Engineering

UNIT-3 : Model Selection

Using a Validation Set, Incorporating
Model Complexity, Estimating
Statistical Bounds, Model Selection
for Decision Trees, Model Evaluation

R V College of Engineering 78
 RV College of
Engineering Model Selection

Performed during model building

Purpose is to ensure that model is not overly
complex (to avoid overfitting)
Need to estimate generalization error
– Using Validation Set
– Incorporating Model Complexity
 Model Selection:
RV College of
Engineering Using Validation Set
Divide training data into two parts:
– Training set:
◆ use for model building
– Validation set:
◆ use for estimating generalization error
◆ Note: validation set is not the same as test set

Drawback:
– Less data available for training
 RV College of Model Selection:
Engineering Incorporating Model Complexity

Rationale: Occam’s Razor

– Given two models of similar generalization errors,
one should prefer the simpler model over the more
complex model

– A complex model has a greater chance of being fitted

accidentally

– Therefore, one should include model complexity when

evaluating a model

Gen. Error(Model) = Train. Error(Model, Train. Data) +

x Complexity(Model)
 RV College of
Estimating the Complexity of
Engineering Decision Trees

Pessimistic Error Estimate of decision tree T

with k leaf nodes:

– err(T): error rate on all training records

– : trade-off hyper-parameter (similar to )
◆ Relative cost of adding a leaf node
– k: number of leaf nodes
– Ntrain: total number of training records
 RV College of Estimating the Complexity of Decision
Engineering Trees: Example

e(TL) = 4/24

+: 3 +: 5 +: 1 +: 3 +: 3 e(TR) = 6/24
-: 0 -: 2 -: 4 -: 0 -: 6

+: 3 +: 2 +: 0 +: 1 +: 3 +: 0 =1
-: 1 -: 1 -: 2 -: 2 -: 1 -: 5

Decision Tree, TL Decision Tree, TR

egen(TL) = 4/24 + 1*7/24 = 11/24 = 0.458

egen(TR) = 6/24 + 1*4/24 = 10/24 = 0.417

 RV College of
Estimating the Complexity of
Engineering Decision Trees

Resubstitution Estimate:
– Using training error as an optimistic estimate of
generalization error
– Referred to as optimistic error estimate

e(TL) = 4/24

e(TR) = 6/24

+: 3 +: 5 +: 1 +: 3 +: 3
-: 0 -: 2 -: 4 -: 0 -: 6

+: 3 +: 2 +: 0 +: 1 +: 3 +: 0
-: 1 -: 1 -: 2 -: 2 -: 1 -: 5

Decision Tree, TL Decision Tree, TR

 RV College of
Minimum Description Length
Engineering (MDL)
A?
X y Yes No
X y
X1 1 0 B? X1 ?
X2 0 B1 B2
X2 ?
X3 0 C? 1
A C1 C2 B X3 ?
X4 1
0 1 X4 ?
… …
Xn
… …
1
Xn ?

Cost(Model,Data) = Cost(Data|Model) + x Cost(Model)

– Cost is the number of bits needed for encoding.
– Search for the least costly model.
Cost(Data|Model) encodes the misclassification errors.
Cost(Model) uses node encoding (number of children)
plus splitting condition encoding.
 RV College of
Model Selection for Decision
Engineering Trees
Pre-Pruning (Early Stopping Rule)
– Stop the algorithm before it becomes a fully-grown tree
– Typical stopping conditions for a node:
◆ Stop if all instances belong to the same class
◆ Stop if all the attribute values are the same
– More restrictive conditions:
◆ Stop if number of instances is less than some user-specified
threshold
◆ Stop if class distribution of instances are independent of the
available features (e.g., using  2 test)
◆ Stop if expanding the current node does not improve impurity
measures (e.g., Gini or information gain).
◆ Stop if estimated generalization error falls below certain threshold
 RV College of
Model Selection for Decision
Engineering Trees
Post-pruning
– Grow decision tree to its entirety
– Subtree replacement
◆ Trim the nodes of the decision tree in a bottom-up
fashion
◆ If generalization error improves after trimming,
replace sub-tree by a leaf node
◆ Class label of leaf node is determined from
majority class of instances in the sub-tree
 Example of Post-Pruning
RV College of
Engineering

Training Error (Before splitting) = 10/30

Class = Yes 20 Pessimistic error = (10 + 0.5)/30 = 10.5/30

Class = No 10 Training Error (After splitting) = 9/30

Error = 10/30 Pessimistic error (After splitting)

= (9 + 4  0.5)/30 = 11/30
PRUNE!
A?

A1 A4
A2 A3

Class = Yes 8 Class = Yes 3 Class = Yes 4 Class = Yes 5

Class = No 4 Class = No 4 Class = No 1 Class = No 1
RV College of
Engineering
Examples of Post-pruning
Decision Tree:
depth = 1 :
| breadth > 7 : class 1
| breadth <= 7 :
| | breadth <= 3 : Simplified Decision Tree:
| | | ImagePages > 0.375 : class 0
| | | ImagePages <= 0.375 : depth = 1 :
| | | | totalPages <= 6 : class 1
| ImagePages <= 0.1333 : class 1
| | | | totalPages > 6 :
| | | | | breadth <= 1 : class 1
Subtree | ImagePages > 0.1333 :
| | | | | breadth > 1 : class 0 Raising | | breadth <= 6 : class 0
| | width > 3 : | | breadth > 6 : class 1
| | | MultiIP = 0:
| | | | ImagePages <= 0.1333 : class 1 depth > 1 :
| | | | ImagePages > 0.1333 : | MultiAgent = 0: class 0
| | | | | breadth <= 6 : class 0 | MultiAgent = 1:
| | | | | breadth > 6 : class 1
| | | MultiIP = 1:
| | totalPages <= 81 : class 0
| | | | TotalTime <= 361 : class 0 | | totalPages > 81 : class 1
| | | | TotalTime > 361 : class 1
depth > 1 :
| MultiAgent = 0:
| | depth > 2 : class 0
| | depth <= 2 : Subtree
| | | MultiIP = 1: class 0
| | | MultiIP = 0:
Replacement
| | | | breadth <= 6 : class 0
| | | | breadth > 6 :
| | | | | RepeatedAccess <= 0.0322 : class 0
| | | | | RepeatedAccess > 0.0322 : class 1
| MultiAgent = 1:
| | totalPages <= 81 : class 0
| | totalPages > 81 : class 1
 RV College of
Engineering
Model Evaluation

Purpose:
– To estimate performance of classifier on previously
unseen data (test set)

Holdout
– Reserve k% for training and (100-k)% for testing
– Random subsampling: repeated holdout
Cross validation
– Partition data into k disjoint subsets
– k-fold: train on k-1 partitions, test on the remaining one
– Leave-one-out: k=n
 RV College of
Engineering Cross-validation Example
3-fold cross-validation
 RV College of
Engineering
Variations on Cross-validation

Repeated cross-validation
– Perform cross-validation a number of times
– Gives an estimate of the variance of the
generalization error
Stratified cross-validation
– Guarantee the same percentage of class
labels in training and test
– Important when classes are imbalanced and
the sample is small
Use nested cross-validation approach for model
selection and evaluation