1604C331 Data Mining

Week 4:
Classification:
Decision Tree 1

Odd Semester 2024-2025

20102620240919
Informatics Engineering
Faculty of Engineering | Universitas Surabaya
Introduction to Classification

2
Informatics Engineering | Universitas Surabaya
What is Classification
• Given a collection of records (training set)
– Each record is characterized by a tuple (x, y), where:
• x is the attribute set, predictor, independent variable, input
• y is the class label, response, dependent variable, output

• Task:
– Build a model that maps each attribute set x into one of the predefined
class label y.
Supervised Learning
• Classification is supervised
learning
– Supervision: the training data
such as observations or
measurements are accompanied
by labels indicating the classes
which they belong to.
– New data is classified based on
the models built from the training
set
Classification Techniques
• Base Classifiers
– Decision tree
– Rule-based
– Nearest-neighbor
– Naïve Bayes and Bayesian Belief Networks
– Support Vector Machines
– Neural Networks, Deep Neural Networks

• Ensemble Classifiers
– Boosting, Bagging, Random Forests
Decision Tree

6
Informatics Engineering | Universitas Surabaya
Decision Tree Induction
Splitting Attributes
Home Marital Annual Defaulted
ID
Owner Status Income Borrower
1 Yes Single 125K No Home
2 No Married 100K No Owner
Yes No
3 No Single 70K No
4 Yes Married 120K No NO MarSt
5 No Divorced 95K Yes Single, Divorced Married
6 No Married 60K No
Income NO
7 Yes Divorced 220K No
8 No Single 85K
< 80K > 80K
Yes
9 No Married 75K No NO YES
10 No Single 90K Yes
10

Training Data Model: Decision Tree

Apply Model to Test Data (1)
Test Data
Start from the root of tree.
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt

Single, Divorced Married

Income NO
< 80K > 80K

NO YES
Apply Model to Test Data (2)
Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt

Single, Divorced Married

Income NO
< 80K > 80K

NO YES
Apply Model to Test Data (3)
Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt

Single, Divorced Married

Income NO
< 80K > 80K

NO YES
Apply Model to Test Data (4)
Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt

Single, Divorced Married

Income NO
< 80K > 80K

NO YES
Apply Model to Test Data (5)
Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt

Single, Divorced Married

Income NO
< 80K > 80K

NO YES
Apply Model to Test Data (5)
Test Data
Home Marital Annual Defaulted
Owner Status Income Borrower
No Married 80K ?
Home 10

Yes Owner No

NO MarSt
Assign defaulted to “No”
Single, Divorced Married

Income NO
< 80K > 80K

NO YES
Another Example of Decision Tree
MarSt Single,
Married Divorced
Home Marital Annual Defaulted
ID
Owner Status Income Borrower
NO Home
1 Yes Single 125K No
Yes Owner No
2 No Married 100K No
3 No Single 70K No NO Income
4 Yes Married 120K No < 80K > 80K
5 No Divorced 95K Yes
NO YES
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes There could be more than one tree that fits the same data!
9 No Married 75K No
10 No Single 90K Yes
10
Decision Tree Classification Task
Tid Attrib1 Attrib2 Attrib3 Class
Tree
1 Yes Large 125K No Induction
2 No Medium 100K No algorithm
3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply Decision Tree
Tid Attrib1 Attrib2 Attrib3 Class
Model
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ?

Deduction
14 No Small 95K ?

15 No Large 67K ?
10

Test Set
Decision Tre Induction Algorithms
• Many algorithms:
– Hunt’s algorithm (one of the earliest)
– CART (Classification and Regression Trees – generation of binary decision tree)
– ID3 (Iterative Dichotomizer), C4.5 (became benchmark to which newer supervised learning
algorithms are compared)
– SLIQ, SPRINT

• Why is decision tree induction popular?

– Relatively faster learning speed (compared to other classification methods)
– Convertible to simple and easy to understand classification rules.
General Structure of Hunt’s Algorithm
Home Marital Annual Defaulted
ID
Owner Status Income Borrower

Let Dt be the set of training records that reach a 1 Yes Single 125K No

node t 2 No Married 100K No

3 No Single 70K No
4 Yes Married 120K No
General Procedure: 5 No Divorced 95K Yes

– If Dt contains records that belong the 6 No Married 60K No

same class yt, then t is a leaf node 7 Yes Divorced 220K No

8 No Single 85K Yes
labeled as yt
9 No Married 75K No
– If Dt contains records that belong to 10 No Single 90K Yes
more than one class, use an attribute
10

test to split the data into smaller subsets. Dt

Recursively apply the procedure to each
subset. ?
Hunt’s Algorithm Home
Owner
Yes No Home Marital Annual Defaulted
ID
Defaulted = No Owner Status Income Borrower
Defaulted = No Defaulted = No 1 Yes Single 125K No
(7,3)
(3,0) (4,3) 2 No Married 100K No

(a) (b) 3 No Single 70K No

4 Yes Married 120K No
5 No Divorced 95K Yes
Home
6 No Married 60K No
Owner
Yes No 7 Yes Divorced 220K No
Home
Owner 8 No Single 85K Yes
Defaulted = No Marital
Yes No 9 No Married 75K No
Status
(3,0) Single,
Married
Defaulted = No Marital Divorced 10 No Single 90K Yes
10

Status
Defaulted = No
(3,0) Single,
Married
Annual
Divorced Income
(3,0)
Defaulted = Yes Defaulted = No < 80K >= 80K

Defaulted = No Defaulted = Yes

(1,3) (3,0)
(1,0) (0,3)
(c) (d)
Design Issues of Decision Tree Induction
• How should training records be split?
– Method for expressing test condition
• Depends on attribute types: binary, nominal, ordinal, continuous
– Measure for evaluating the goodness of a test condition

• How should the splitting procedure stop?

– Stop splitting if all records belong to the same class or have identical attribute
values
– There are no samples left.
– Early termination:
• There are no remaining attributes for further partitioning: majority voting is
employed for classifying the leaf.
Test Condition for Nominal Attributes
• Multi-way split: use as many partitions as distinct values.
Marital
Status

Single Divorced Married

• Binary split: divides values into 2 subsets.

Marital Marital Marital
Status Status Status
OR OR

{Married} {Single, {Single} {Married, {Single, {Divorced}

Divorced} Divorced} Married}
Test Condition for Ordinal Attributes
• Multi-way split: use as many partitions as distinct values.
Shirt
Size

Small
Medium Large Extra Large

• Binary split: divides values into 2 subsets, & preserves order

property among attribute values.
Shirt
Shirt Shirt Size
Size Size This grouping
violates order
property

{Small, {Medium,
{Small, {Large, {Small} {Medium, Large, Large} Extra Large}
Medium} Extra Large} Extra Large}
Test Condition for Continuous Attributes
• Multi-way split: use as many partitions as distinct values.

• Binary split: divides values into 2 subsets.

Splitting based on Continuous Attributes
• Different ways of handling:
– Discretization to form an ordinal categorical attribute
• Ranges can be found by equal interval bucketing, equal frequency bucketing
(percentiles), or clustering.
• Static: discretize once at the beginning
• Dynamic: repeat at each node

– Binary Decision: (A < v) or (A  v)

• Consider all possible splits and finds the best cut
• Can be more compute intensive
How to determine the Best Split
Before Splitting:
10 records of class 0,
10 records of class 1

Gender Car Customer

Type ID

Yes No Family Luxury c1 c20

c10 c11
Sports
C0: 6 C0: 4 C0: 1 C0: 8 C0: 1 C0: 1 ... C0: 1 C0: 0 ... C0: 0
C1: 4 C1: 6 C1: 3 C1: 0 C1: 7 C1: 0 C1: 0 C1: 1 C1: 1

Which test condition is the best?

Method to determine the Best Split
• Greedy approach: nodes with purer class distribution are preferred.

• Need a measure of node impurity.

C0: 5 C0: 9
C1: 5 C1: 1

High degree of impurity Low degree of impurity

Measures of Node Impurity
• Gini Index
– Used in decision tree algorithms such as: CART, SLIQ, SPRINT, IBM
IntelligentMiner
𝑐−1
2 Where 𝒑𝒊 𝒕 is the frequency of class 𝒊 at node t,
𝐺𝑖𝑛𝑖 𝐼𝑛𝑑𝑒𝑥 = 1 − ෍ 𝑝𝑖 𝑡
and 𝒄 is the total number of classes
𝑖=0
• Entropy
– ID3, C4.5 𝑐−1

𝐸𝑛𝑡𝑟𝑜𝑝𝑦 = − ෍ 𝑝𝑖 𝑡 𝑙𝑜𝑔2 𝑝𝑖 (𝑡)

𝑖=0
• Misclassification error
𝐶𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑒𝑟𝑟𝑜𝑟 = 1 − max[𝑝𝑖 (𝑡)]
Finding the Best Split (1)
• Compute impurity measure before splitting (P)

• Compute impurity measure after splitting (M)

– Compute impurity measure of each child node
– M is the weighted impurity of child nodes

• Choose the attribute test condition that produces the highest gain:

Gain = P – M

or equivalently, lowest impurity measure after splitting (M).

Finding the Best Split (2)
Before Splitting: C0 N00
P
C1 N01

A? B?
Yes No Yes No

Node N1 Node N2 Node N3 Node N4

C0 N10 C0 N20 C0 N30 C0 N40

C1 N11 C1 N21 C1 N31 C1 N41

M11 M12 M21 M22

M1 M2
Gain = P – M1 vs P – M2
Problem with Large Number of Partitions
Node impurity measures tend to prefer splits that result in large number
of partitions, each being small but pure.
Gender Car Customer
Type ID

Yes No Family Luxury c1 c20

c10 c11
Sports
C0: 6 C0: 4 C0: 1 C0: 8 C0: 1 C0: 1 ... C0: 1 C0: 0 ... C0: 0
C1: 4 C1: 6 C1: 3 C1: 0 C1: 7 C1: 0 C1: 0 C1: 1 C1: 1

Customer ID has highest information gain because impurity for all the
children is zero.
Gini Index, Gini Split, Gain
• If a dataset D contains records from n classes, gini(D) is defined as:
𝑛

𝑔𝑖𝑛𝑖 𝐷 = 1 − ෍ 𝑝𝑗 2 where pj is the relative frequency of class j in D.

𝑗=1
• If dataset D is split on A into two subsets D1 and D2, ginisplit(D) is defined as:
|D1| |D |
gini A ( D) = gini( D1) + 2 gini( D 2)
|D| |D|
• Reduction in impurity (Gain):

gini( A) = gini( D) − giniA ( D)

• The attribute provides the smallest ginisplit(D) or the larget reduction in impurity (Gain) is chosen to split
the node.

• Maximum of 1-1/n when records are equally distributed among all classes, implying the least
beneficial situation for classification.
• Minimum of – when all records belong to one class, implying the most beneficial situation for
classification.
Examples: Gini Index
• Gini index for a given node t:
𝑛
2
𝑔𝑖𝑛𝑖(𝐷𝑡) = 1 − ෍ 𝑝𝑗 𝑡
𝑗=1

• For 2-class problem (n = 2): (p, 1-p)

– Gini = 1 – p2 – (1 – p)2
= 2p (1-p)

C1 0 C1 1 C1 2 C1 3
C2 6 C2 5 C2 4 C2 3
Gini=0.000 Gini=0.278 Gini=0.444 Gini=0.500
Computing Gini Index of a Single Node
𝑛
2
𝑔𝑖𝑛𝑖(𝐷𝑡) = 1 − ෍ 𝑝𝑗 𝑡
𝑗=1

C1 0 P(C1) = 0/6 = 0 P(C2) = 6/6 = 1

C2 6 Gini = 1 – P(C1)2 – P(C2)2 = 1 – 0 – 1 = 0

C1 1 P(C1) = 1/6 P(C2) = 5/6

C2 5 Gini = 1 – (1/6)2 – (5/6)2 = 0.278

C1 2 P(C1) = 2/6 P(C2) = 4/6

C2 4 Gini = 1 – (2/6)2 – (4/6)2 = 0.444
Computing Gini split
• When a node p is split into k partitions (children):
𝑘
𝑛𝑖
𝑔𝑖𝑛𝑖𝑠𝑝𝑙𝑖𝑡 = ෍ 𝑔𝑖𝑛𝑖(𝑖)
𝑛
𝑖=1

where, 𝑛𝑖 = number of records at child 𝑖,

𝑛 = number of records at parent node 𝑝.
Binary Attributes: Computing Gini Index
• Splits into 2 partitions (children)
• Effect of weighing partitions: larger and purer partitions are sought.
Parent
B? C1 7
Yes No C2 5
Gini = 0.486
Node N1 Node N2
Gini(N1)
= 1 – (5/6)2 – (1/6)2 N1 N2 Weighted Gini of N1 N2 (Gini split):
= 0.278
C1 5 2 Ginisplit(B) = 6/12 * 0.278 + 6/12 * 0.444
Gini(N2) C2 1 4 = 0.361
= 1 – (2/6)2 – (4/6)2 Gini=0.361
= 0.444 Gain = 0.486 – 0.361 = 0.125
Exercises

35
Informatics Engineering | Universitas Surabaya
Tutorial

Calculate the gain in the Gini index when splitting on A and B. Which
attribute would the decision tree induction algorithm choose?
Homework
• Do all the exercises.
• You can write the solution on papers or you can use tools like Excel
or Python and explain in detail step-by-step of your work until it finds
the solution.
• Create a pdf file for your solution and submit it to ULS
• You can upload one more file that you use to do the computation
(.xlsx or .ipynb) along with your .pdf file. Upload those files
separately.
• Note: do not forget to put your Student ID and name at the first page
of the pdf file.
Exercise 1: Loan Borrower
Compute Gini index for the Home owner and Marital status attributes
and find the best split. ID Home owner Marital status Defaulted?

1 Yes Single No

2 No Married No

3 No Single No

4 Yes Married No

5 No Divorced Yes

6 No Single No

7 Yes Divorced No

8 No Single Yes

9 No Married No

10 No Single Yes
Exercise 2: Customer DB training examples
a) Compute Gini index for the overall
collection of training examples
b) Compute Gini index for the Gender
attribute.
c) Compute Gini index for the Car Type
attribute using multiway split.
d) Compute Gini index for the Shirt
Size attribute using multiway split.
e) Which attribute is better? Explain
your answer.
Question?

40
Informatics Engineering | Universitas Surabaya