Nearest Neighbour Algorithm

The nearest neighbor algorithm classifies new data based on similarity: it identifies the k training samples that are closest in distance, and assigns the new data the majority class of those neighbors. It requires a training set, distance calculation between samples, and selection of k. Distance is typically Euclidean. Choosing k involves a tradeoff between noise and including samples from other classes. Scaling may be needed to avoid dominance by any single attribute.

Uploaded by

varunsingh214761

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPTX, PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

202 views20 pages

Nearest Neighbour Algorithm

Uploaded by

varunsingh214761

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPTX, PDF, TXT or read online on Scribd

You are on page 1/ 20

Nearest Neighbor Algorithm

Nearest Neighbor Classifiers
 Basic idea:
 If it walks like a duck, quacks like a duck, then

it’s probably a duck

Compute
Distance Test
Record

Training Choose k of the

Records “nearest” records

1/7/19 Nearest Neighbour Algorithm 3

Nearest neighbor method
 Majority vote within the k nearest neighbors

new

K= 1: brown
K= 3: green

1/7/19 Nearest Neighbour Algorithm 4

Nearest-Neighbor Classifiers
 Requires three things
– The set of stored records
– Distance Matrix to compute
distance between records
– The value of k, the number of
nearest neighbors to retrieve
 To classify an unknown record:
– Compute distance to other
training records
– Identify k nearest neighbors
– Use class labels of nearest
neighbors to determine the
class label of unknown record
(e.g., by taking majority vote)

1/7/19 Nearest Neighbour Algorithm 5

Definition of Nearest Neighbor

X X X

(a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor

K-nearest neighbors of a record x are data points

that have the k smallest distance to x

1/7/19 Nearest Neighbour Algorithm 6

1 nearest-neighbor
Voronoi Diagram

1/7/19 Nearest Neighbour Algorithm 7

Nearest Neighbor Classification
 Compute distance between two points:
 Euclidean distance

d ( p, q )   ( pi
i
q ) i
2

 Determine the class from nearest neighbor list

 take the majority vote of class labels among

the k-nearest neighbors

 Weigh the vote according to distance

 weight factor, w = 1/d2

1/7/19 Nearest Neighbour Algorithm 8

Nearest Neighbor Classification…
 Choosing the value of k:
 If k is too small, sensitive to noise points
 If k is too large, neighborhood may include points from
other classes

1/7/19 Nearest Neighbour Algorithm 9

Nearest Neighbor Classification…
 Scaling issues
 Attributes may have to be scaled to prevent

distance measures from being dominated by

one of the attributes
 Example:

 height of a person may vary from 1.5m to 1.8m

 weight of a person may vary from 90lb to 300lb
 income of a person may vary from $10K to $1M