Scribd
Upload
61 views17 pages

Lecture Notes For Chapter 4 Instance-Based Learning Introduction To Data Mining, 2 Edition

The document discusses instance-based learning classification techniques, specifically nearest neighbor classifiers. It explains that nearest neighbor classifiers determine the class of an unknown record based on the classes of the k nearest training records, as measured by a distance metric like Euclidean distance. The document covers key aspects of nearest neighbor classifiers like choosing k, handling scaling issues, and techniques for improving efficiency like using proximity graphs.

Uploaded by

Usman Ali
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
61 views17 pages

Lecture Notes For Chapter 4 Instance-Based Learning Introduction To Data Mining, 2 Edition

The document discusses instance-based learning classification techniques, specifically nearest neighbor classifiers. It explains that nearest neighbor classifiers determine the class of an unknown record based on the classes of the k nearest training records, as measured by a distance metric like Euclidean distance. The document covers key aspects of nearest neighbor classifiers like choosing k, handling scaling issues, and techniques for improving efficiency like using proximity graphs.

Uploaded by

Usman Ali
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 17

Data Mining

Classification: Alternative Techniques

Lecture Notes for Chapter 4

Instance-Based Learning

Introduction to Data Mining , 2nd Edition


by
Tan, Steinbach, Karpatne, Kumar

02/14/2018 Introduction to Data Mining, 2nd Edition 1


Instance Based Classifiers

 Examples:
– Rote-learner
 Memorizes entire training data and performs
classification only if attributes of record match one
of the training examples exactly

– Nearest neighbor
 Uses k “closest” points (nearest neighbors) for
performing classification

02/14/2018 Introduction to Data Mining, 2nd Edition 2


Nearest Neighbor Classifiers

 Basic idea:
– If it walks like a duck, quacks like a duck, then
it’s probably a duck

Compute
Distance Test
Record

Training Choose k of the


Records “nearest” records

02/14/2018 Introduction to Data Mining, 2nd Edition 3


Nearest-Neighbor Classifiers
Unknown record  Requires three things
– The set of labeled records
– Distance Metric to compute
distance between records
– The value of k, the number of
nearest neighbors to retrieve

 To classify an unknown record:


– Compute distance to other
training records
– Identify k nearest neighbors
– Use class labels of nearest
neighbors to determine the
class label of unknown record
(e.g., by taking majority vote)

02/14/2018 Introduction to Data Mining, 2nd Edition 4


Definition of Nearest Neighbor

X X X

(a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor

K-nearest neighbors of a record x are data points


that have the k smallest distances to x

02/14/2018 Introduction to Data Mining, 2nd Edition 5


1 nearest-neighbor

Voronoi Diagram

02/14/2018 Introduction to Data Mining, 2nd Edition 6


Nearest Neighbor Classification

 Compute distance between two points:


– Euclidean distance

d ( p, q )   ( pi
i
q )
i
2

 Determine the class from nearest neighbor list


– Take the majority vote of class labels among
the k-nearest neighbors
– Weigh the vote according to distance
 weight factor, w = 1/d2
02/14/2018 Introduction to Data Mining, 2nd Edition 7
Nearest Neighbor Classification…

 Choosing the value of k:


– If k is too small, sensitive to noise points
– If k is too large, neighborhood may include points from
other classes

02/14/2018 Introduction to Data Mining, 2nd Edition 8


Nearest Neighbor Classification…

 Scaling issues
– Attributes may have to be scaled to prevent
distance measures from being dominated by
one of the attributes
– Example:
 height of a person may vary from 1.5m to 1.8m
 weight of a person may vary from 90lb to 300lb
 income of a person may vary from $10K to $1M

02/14/2018 Introduction to Data Mining, 2nd Edition 9


Nearest Neighbor Classification…

 Selection of the right similarity measure is critical:

111111111110 000000000001
vs
011111111111 100000000000

Euclidean distance = 1.4142 for both pairs

02/14/2018 Introduction to Data Mining, 2nd Edition 10


Nearest neighbor Classification…

 k-NN classifiers are lazy learners since they do not build


models explicitly
– Does not learn a distinguishing function from the data
– Memorizes the training dataset
– In contrast, logistic regression learns its model
weights during training phase
 Classifying unknown((prediction) records are relatively
expensive
– Each time k-NN has to search nearest neighbors in
the entire training set
 Tricks such as BallTress and KDtress can help in speed up the
search
02/14/2018 Introduction to Data Mining, 2nd Edition 11
Nearest neighbor Classification…

 Can produce arbitrarily shaped decision boundaries


 Selection of right proximity measure (amount of k) is
essential
 Superfluous or redundant attributes can create problems
– Only distinguishing attributes make the difference
 Missing attributes are hard to handle

02/14/2018 Introduction to Data Mining, 2nd Edition 12


Improving KNN Efficiency

 Avoid computing distance to all objects in the


training set
– Multi-dimensional access methods (k-d trees)
– Fast approximate similarity search
– Locality Sensitive Hashing (LSH)
 Condensing

– Determine a smaller set of objects that give


the same performance
 Editing

– Remove objects to improve efficiency


02/14/2018 Introduction to Data Mining, 2nd Edition 13
KNN and Proximity Graphs

 Proximity graphs
– a graph in which two vertices are connected
by an edge if and only if the vertices satisfy
particular geometric requirements
– nearest neighbor graphs
– minimum spanning trees
– Delaunay triangulations
– relative neighborhood graphs
– Gabriel graphs

02/14/2018 Introduction to Data Mining, 2nd Edition 14


Measure Proximity in a Graph

 Two different algorithms that measures the


proximity between vertices of a graph
– Shared Nearest Neighbor (SNN)
 Defines proximity or similarity between two vertices in
terms of the number of neighbors (i.e., directly
connected vertices) they have in common

02/14/2018 Introduction to Data Mining, 2nd Edition 15


Measure Proximity in a Graph

 The Neumann Kernal


 Models the relatedness of vertices in a graph based
on the immediate and more distant connections
between vertices
 Use a tunable parameter to control how much weight
should be given to more distant connections
 Produce results based entirely on immediate
neighbors or results that take the full structure of a
graph into consideration

02/14/2018 Introduction to Data Mining, 2nd Edition 16


KNN and Proximity Graphs

 See recent papers by Toussaint


– G. T. Toussaint. Proximity graphs for nearest neighbor
decision rules: recent progress. In Interface-2002, 34th
Symposium on Computing and Statistics, ontreal, Canada,
April 17–20 2002.
– G. T. Toussaint. Open problems in geometric methods for
instance based learning. In Discrete and Computational
Geometry, volume 2866 of Lecture Notes in Computer
Science, pages 273–283, December 6-9, 2003.
– G. T. Toussaint. Geometric proximity graphs for improving
nearest neighbor methods in instance-based learning and
data mining. Int. J. Comput. Geometry Appl., 15(2):101–
150, 2005.

02/14/2018 Introduction to Data Mining, 2nd Edition 17

You might also like