Scribd
Upload
101 views13 pages

Instance-Based Learning: Slides Provided by Introduction To Data Mining, 2 Edition

The document discusses K-nearest neighbors (KNN) classification, an instance-based learning method. KNN classifiers assign unlabeled instances to the class that is most common among its K nearest labeled neighbors. The value of K determines how many neighbors to consider. Distance metrics like Euclidean distance are used to find the nearest neighbors. Issues like selecting K, scaling attributes, and efficiency improvements like k-d trees are also summarized.

Uploaded by

Priti Yadav
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
101 views13 pages

Instance-Based Learning: Slides Provided by Introduction To Data Mining, 2 Edition

The document discusses K-nearest neighbors (KNN) classification, an instance-based learning method. KNN classifiers assign unlabeled instances to the class that is most common among its K nearest labeled neighbors. The value of K determines how many neighbors to consider. Distance metrics like Euclidean distance are used to find the nearest neighbors. Issues like selecting K, scaling attributes, and efficiency improvements like k-d trees are also summarized.

Uploaded by

Priti Yadav
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Intelligent Systems

Classification: K Nearest Neighbuors

Instance-Based Learning

Slides provided by Introduction to Data Mining , 2nd Edition


by
Tan, Steinbach, Karpatne, Kumar
Instance Based Classifiers

Examples:
– Rote-learner
 Memorizes entire training data and performs
classification only if attributes of record match one
of the training examples exactly

– Nearest neighbor
 Uses k “closest” points (nearest neighbors) for
performing classification

02/14/2018 Introduction to Data Mining, 2nd Edition 2


Nearest Neighbor Classifiers

Basic idea:
– If it walks like a duck, quacks like a duck, then
it’s probably a duck

Compute
Distance Test
Record

Training Choose k of the


Records “nearest” records

02/14/2018 Introduction to Data Mining, 2nd Edition 3


Nearest-Neighbor Classifiers
Unknown record Requires three things
– The set of labeled records
– Distance Metric to compute
distance between records
– The value of k, the number of
nearest neighbors to retrieve

To classify an unknown record:


– Compute distance to other
training records
– Identify k nearest neighbors
– Use class labels of nearest
neighbors to determine the
class label of unknown record
(e.g., by taking majority vote)

02/14/2018 Introduction to Data Mining, 2nd Edition 4


Definition of Nearest Neighbor

X X X

(a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor

K-nearest neighbors of a record x are data points


that have the k smallest distances to x

02/14/2018 Introduction to Data Mining, 2nd Edition 5


1 nearest-neighbor

Voronoi Diagram

02/14/2018 Introduction to Data Mining, 2nd Edition 6


Nearest Neighbor Classification

Compute distance between two points:


– Euclidean distance

d ( p, q )   ( pi
i
q ) i
2

Determine the class from nearest neighbor list


– Take the majority vote of class labels among
the k-nearest neighbors
– Weigh the vote according to distance
 weight factor, w = 1/d2
02/14/2018 Introduction to Data Mining, 2nd Edition 7
Nearest Neighbor Classification…

Choosing the value of k:


– If k is too small, sensitive to noise points
– If k is too large, neighborhood may include points from
other classes

02/14/2018 Introduction to Data Mining, 2nd Edition 8


Nearest Neighbor Classification…

Scaling issues
– Attributes may have to be scaled to prevent
distance measures from being dominated by
one of the attributes
– Example:
 height of a person may vary from 1.5m to 1.8m
 weight of a person may vary from 90lb to 300lb

 income of a person may vary from $10K to $1M

02/14/2018 Introduction to Data Mining, 2nd Edition 9


Nearest Neighbor Classification…

Selection of the right similarity measure is critical:

111111111110 000000000001
vs
011111111111 100000000000

Euclidean distance = 1.4142 for both pairs

02/14/2018 Introduction to Data Mining, 2nd Edition 10


Nearest neighbor Classification…

k-NN classifiers are lazy learners since they do not build


models explicitly
Classifying unknown records are relatively expensive
Can produce arbitrarily shaped decision boundaries
Easy to handle variable interactions since the decisions
are based on local information
Selection of right proximity measure is essential
Superfluous or redundant attributes can create problems
Missing attributes are hard to handle

02/14/2018 Introduction to Data Mining, 2nd Edition 11


Improving KNN Efficiency

Avoid having to compute distance to all objects in


the training set
– Multi-dimensional access methods (k-d trees)
– Fast approximate similarity search
– Locality Sensitive Hashing (LSH)
Condensing
– Determine a smaller set of objects that give
the same performance
Editing
– Remove objects to improve efficiency
02/14/2018 Introduction to Data Mining, 2nd Edition 12
KNN and Proximity Graphs

Proximity graphs
– a graph in which two vertices are connected by an edge if and
only if the vertices satisfy particular geometric requirements
– nearest neighbor graphs,
– minimum spanning trees
– Delaunay triangulations
– relative neighborhood graphs
– Gabriel graphs
See recent papers by Toussaint
– G. T. Toussaint. Proximity graphs for nearest neighbor decision rules: recent progress.
In Interface-2002, 34th Symposium on Computing and Statistics, ontreal, Canada,
April 17–20 2002.
– G. T. Toussaint. Open problems in geometric methods for instance based learning. In
Discrete and Computational Geometry, volume 2866 of Lecture Notes in Computer
Science, pages 273–283, December 6-9, 2003.
– G. T. Toussaint. Geometric proximity graphs for improving nearest neighbor methods
in instance-based learning and data mining. Int. J. Comput. Geometry Appl.,
15(2):101–150, 2005.

02/14/2018 Introduction to Data Mining, 2nd Edition 13

You might also like