Summer 2021

Data Mining and Machine

Learning (CSE 321)
Topic – 5: Classification
(Alternative Techniques)
Course Teacher:
Md. Tarek Habib
Assistant Professor
Department of Computer Science and Engineering
Daffodil International University
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1
 Topic Contents
• Nearest-Neighbor Classifier

• Bayesian Classifier

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 2

3

Recommended Reading

• “Introduction to Data Mining,” Pang-Ning

Tan, Michael Steinbach and Vipin Kumar,
Addison Wesley, 2006.
☞ Chapter 5 (Classification: Alternative
Techniques)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 3

3
 Nearest Neighbor Classifiers

● Basic idea:
– If it walks like a duck, quacks like a duck, then
it’s probably a duck

Compute
Distance Test
Record

Training Choose k of the

Records “nearest” records

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 4

 Nearest-Neighbor Classifiers

● Requires three things

– The set of stored records
– Distance Metric to compute
distance between records
– The value of k, the number of
nearest neighbors to retrieve

● To classify an unknown record:

– Compute distance to other
training records
– Identify k nearest neighbors
– Use class labels of nearest
neighbors to determine the
class label of unknown record
(e.g., by taking majority vote)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 5

 Definition of Nearest Neighbor

K-nearest neighbors of a record x are data points

that have the k smallest distance to x

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 6

 Nearest Neighbor Classification

● Compute distance between two points:

– Euclidean distance

● Determine the class from nearest neighbor list

– take the majority vote of class labels among
the k-nearest neighbors
– Weigh the vote according to distance
◆ weight factor, w = 1/d2

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 7

 Nearest Neighbor Classification…

● Choosing the value of k:

– If k is too small, sensitive to noise points
– If k is too large, neighborhood may include points from
other classes

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 8

 Nearest Neighbor Classification…

● Scaling issues
– Attributes may have to be scaled to prevent
distance measures from being dominated by
one of the attributes
– Example:
◆ height of a person may vary from 1.5m to 1.8m
◆ weight of a person may vary from 90lb to 300lb

◆ income of a person may vary from $10K to $1M

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 9

 Nearest neighbor Classification…

● k-NN classifiers are lazy learners

– It does not build models explicitly
– Unlike eager learners such as decision tree
induction and rule-based systems
– Classifying unknown records are relatively
expensive

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 10

 Bayes Classifier

● The task of predicting whether a user is in the risk of heart

disease depends on healthy diet and exercise.
● There might be other factors like heridity, smoking or
alcohol abuse which may affect it
● A probabilistic framework for solving classification
problems
● Conditional Probability:

● Bayes theorem:

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 11

 Example of Bayes Theorem

● Given:
– A doctor knows that meningitis causes stiff neck 50% of the
time
– Prior probability of any patient having meningitis is 1/50,000
– Prior probability of any patient having stiff neck is 1/20

● If a patient has stiff neck, what’s the probability

he/she has meningitis?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 12

 Bayesian Classifiers

● Consider each attribute and class label as random

variables

● Given a record with attributes (A1, A2,…,An)

– Goal is to predict class C
– Specifically, we want to find the value of C that
maximizes P(C| A1, A2,…,An )

● Can we estimate P(C| A1, A2,…,An ) directly from

data?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 13

 Bayesian Classifiers

● Approach:
– compute the posterior probability P(C | A1, A2, …, An) for
all values of C using the Bayes theorem

– Choose value of C that maximizes

P(C | A1, A2, …, An)

– Equivalent to choosing value of C that maximizes

P(A1, A2, …, An|C) P(C)

● How to estimate P(A1, A2, …, An | C )?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 14
 Naïve Bayes Classifier

● Assume independence among attributes Ai when class is

given:
– P(A1, A2, …, An |C) = P(A1| Cj) P(A2| Cj)… P(An| Cj)

– Can estimate P(Ai| Cj) for all Ai and Cj.

– New point is classified to Cj if P(Cj) Π P(Ai| Cj) is

maximal.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 15

How to Estimate Probabilities from Data?

● Class: P(C) = Nc/N

– e.g., P(No) = 7/10,
P(Yes) = 3/10

● For discrete attributes:

P(Ai | Ck) = |Aik|/ Nc k
– where |Aik| is number of
instances having attribute
Ai and belongs to class Ck
– Examples:
P(Status=Married|No) = 4/7
P(Refund=Yes|Yes)=0

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 16

How to Estimate Probabilities from Data?

● For continuous attributes:

– Discretize the range into bins
◆ one ordinal attribute per bin
◆ violates independence assumption k

– Two-way split: (A < v) or (A > v)

◆ choose only one of the two splits as new attribute
– Probability density estimation:
◆ Assume attribute follows a normal distribution
◆ Use data to estimate parameters of distribution
(e.g., mean and standard deviation)
◆ Once probability distribution is known, can use it to
estimate the conditional probability P(Ai|c)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 17
How to Estimate Probabilities from Data?

● Normal distribution:

– One for each (Ai,ci) pair

● For (Income, Class=No):

– If Class=No
◆ sample mean = 110
◆ sample variance = 2975

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 18

 Example of Naïve Bayes Classifier
Given a Test Record:

● P(X|Class=No) = P(Refund=No|Class=No)
× P(Married| Class=No)
× P(Income=120K| Class=No)
= 4/7 × 4/7 × 0.0072 = 0.0024

● P(X|Class=Yes) = P(Refund=No| Class=Yes)

× P(Married| Class=Yes)
× P(Income=120K| Class=Yes)
= 1 × 0 × 1.2 × 10-9 = 0

Since P(X|No)P(No) > P(X|Yes)P(Yes)

Therefore P(No|X) > P(Yes|X)
=> Class = No

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 19

 Naïve Bayes Classifier

● If one of the conditional

probability is zero, then
the entire expression
becomes zero
● Probability estimation:

n: number of instances
from class yj
p: prior probability
m: parameter
nc: is the number of
training examples from
class yj

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 20

 Example of Naïve Bayes Classifier

A: attributes
M: mammals
N: non-mammals

P(A|M)P(M) > P(A|N)P(N)

=> Mammals

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 21

 Naïve Bayes (Summary)

● Robust to isolated noise points

● Handle missing values by ignoring the instance

during probability estimate calculations

● Robust to irrelevant attributes

● Independence assumption may not hold for some

attributes
– Use other techniques such as Bayesian Belief
Networks (BBN)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 22
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 23