Multi-dimensional

Search Trees
CS 302 Data Structures
Dr. George Bebis
Query Types

Exact match query: Asks for the object(s)

whose key matches query key exactly.

Range query: Asks for the objects whose

key lies in a specified query range (interval).

Nearest-neighbor query: Asks for the

objects whose key is close to query key.

2
Exact Match Query

Suppose that we store employee records in a

database:

ID Name Age Salary #Children

Example:
key=ID: retrieve the record with ID=12345

3
Range Query

Example:
key=Age: retrieve all records satisfying
20 < Age < 50
key= #Children: retrieve all records satisfying
1 < #Children < 4

ID Name Age Salary #Children

4
Nearest-Neighbor(s) (NN)
Query
Example:
key=Salary: retrieve the employee whose salary
is closest to $50,000 (i.e., 1-NN).
key=Age: retrieve the 5 employees whose age is
closest to 40 (i.e., k-NN, k=5).

ID Name Age Salary #Children

5
Nearest Neighbor(s) Query
What is the closest restaurant to my hotel?

6
Nearest Neighbor(s) Query
(contd)
Find the 4 closest restaurants to my hotel

7
Multi-dimensional Query

Inpractice, queries might involve multi-

dimensional keys.
key=(Name, Age): retrieve all records with
Name=George and 50 <= Age <= 70

ID Name Age Salary #Children

8
 Nearest Neighbor Query in High
Dimensions
Very important and practical problem!
Image retrieval

find N closest
matches (i.e., N
nearest neighbors)
(f1,f2, .., fk)

9
Nearest Neighbor Query in High
Dimensions
Face recognition

find closest match

(i.e., nearest neighbor)

10
We will discuss

Range trees
KD-trees
Quadtrees

11
Interpreting Queries
Geometrically
Multi-dimensional keys can be thought as
points in high dimensional spaces.

Queries about records Queries about points

12
Example 1- Range Search in
2D

age = 10,000 x year + 100 x month + day

13
Example 2 Range Search in
3D

14
Example 3 Nearest
Neighbors Search

Query
Point

15
1D Range Search

16
1D Range Search
Range: [x, x]

Updates take O(n) time

Does not generalize well to high dimensions.

Example: retrieve all points in [25, 90]

17
1D Range Search
Data Structure 2: BST
Search using binary search property.
Some subtrees are eliminated during search.
Search using:
Range:[l,r]
x if
if l x r>x
search
search

Example: retrieve all points in [25, 90]

18
1D Range Search
Data Structure 3: BST with data stored in leaves
Internal nodes store splitting values (i.e., not
necessarily same as data).
Data points are stored in the leaf nodes.

19
BST with data stored in
leaves

0 100
25 50 75

50

Data: 10, 39, 55, 120

25 75

10 39 55 120

20
1D Range Search
Retrieving data in [x, x]
Perform binary search twice, once using x and the other using x
Suppose binary search ends at leaves l and l
The points in [x, x] are the ones stored between l and l plus,
possibly, the points stored in l and l

21
1D Range Search
Example: retrieve all points in [25, 90]
The search path for 25 is:

22
1D Range Search
The search for 90 is:

23
 1D Range Search
Examine the leaves in the sub-trees between the
two traversing paths from the root.

split node

retrieve all points in [25, 90] 24

1D Range Search Another
Example

25
1D Range Search
How do we find the leaves of interest?
Find split node (i.e., node where the
paths to x and x split).

Left turn: report leaves in right subtrees

Right turn: report leaves in left substrees

O(logn + k) time where

k is the number of
items reported.

26
1D Range Search
Speed-up search by keeping the leaves in
sorted order using a linked-list.

27
2D Range Search

28
2D Range Search (contd)
A 2D range query can be decomposed in two 1D
range queries:
One on the x-coordinate of the points.
The other on the y-coordinates of the points.

29
2D Range Search (contd)
Store a primary 1D range tree for all the points
based on x-coordinate.
For each node, store a secondary 1D range tree based
on y-coordinate.

30
2D Range Search (contd)

Range Tree

Space requirements: O(nlogn)

31
2D Range Search (contd)
Search using the x-coordinate only.
How to restrict to points with proper y-coordinate?

32
2D Range Search (contd)
Recursively search within each subtree using
the y-coordinate.

33
Range Search in d
dimensions
1D query time: O(logn + k)

2D query time: O(log2n + k)

d dimensions:

34
KD Tree
A binary search tree where every node is a
k-dimensional point.

Example: k=2 53, 14

27, 28 65, 51

30, 11 31, 85 70, 3 99, 90

40, 26 7, 39 32, 29 82, 64

29, 16

38, 23 55,62 73, 75

15, 61
KD Tree (contd)
Example: data stored at the leaves
KD Tree (contd)
Every node (except leaves) represents a hyperplane
that divides the space into two parts.
Points to the left (right) of this hyperplane represent the
left (right) sub-tree of that node.

Pleft Pright
KD Tree (contd)
As we move down the tree, we divide the space along
alternating (but not always) axis-aligned hyperplanes:

Split by x-coordinate: split by a vertical line that

has (ideally) half the points left or on, and half
right.

Split by y-coordinate: split by a horizontal line

that has (ideally) half the points below or on and
half above.
KD Tree - Example
Split by x-coordinate: split by a vertical line that
has approximately half the points left or on, and
half right.

x
KD Tree - Example
Split by y-coordinate: split by a horizontal line that
has half the points below or on and half above.

y y
KD Tree - Example
Split by x-coordinate: split by a vertical line that
has half the points left or on, and half right.

y y

x
x x x
KD Tree - Example
Split by y-coordinate: split by a horizontal line that
has half the points below or on and half above.

y y

x
x x x

y y
Node Structure
A KD-tree node has 5 fields
Splitting axis
Splitting value
Data
Left pointer
Right pointer
Splitting Strategies
Divide based on order of point insertion
Assumes that points are given one at a time.

Divide by finding median

Assumes all the points are available ahead of time.

Divide perpendicular to the axis with widest

spread
Split axes might not alternate
and more!
Example using order of point
insertion
(data stored at nodes
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
 Example using median
(data stored at the leaves)
Another Example using
median
Another Example - using
median
Another Example - using
median
Another Example - using
median
Another Example - using
median
Another Example - using
median
Another Example - using
median
Example split perpendicular to
the axis with widest spread

62
KD Tree (contd)

Lets discuss
Insert
Delete
Search

63
Insert new data
55 > 53, move right

Insert (55, 53, 14 x 62 > 51, move right

62) 27, 28 65, 51 y

70, 3 99, 90 x
30, 11 31, 85
55 < 99, move left

40, 26 7, 39 32, 29 82, 64

29, 16 y

38, 23 55,62 73, 75

15, 61
62 < 64, move left
Null pointer, attach
Delete data
Suppose we need to remove p = (a, b)
Find node t which contains p
If t is a leaf node, replace it by null
Otherwise, find a replacement node r = (c, d) see below!
Replace (a, b) by (c, d)
Remove r
Finding the replacement r = (c, d)
If t has a right child, use the successor*
Otherwise, use node with minimum value* in the left
subtree
*
(depending on what axis the node discriminates)
Delete data (contd)
Delete data (contd)
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
KD Tree Exact Search
 KD Tree - Range Range:[l,r]
x
r>x
Search [35, 40] x [23, 30]
In range? If so, print cell
l x
low[level]<=data[level] search t.left
high[level] >= data[level] search t.right x 53, 14

y 65, 51
27, 28

x 70, 3 99, 90
30, 11 31, 85

y 40, 26 7, 39 32, 29 82, 64

29, 16

x 38, 23 73, 75
15, 61

low[0] = 35, high[0] = 40; This sub-tree is never searched.

Searching is preorder. Efficiency is obtained by
low[1] = 23, high[1] = 30; pruning subtrees from the search.

79
KD Tree - Range Search
Consider a KD Tree where the data is stored
at the leaves, how do we perform range
search?
KD Tree Region of a node
The region region(v) corresponding to a node
v is a rectangle, which is bounded by
splitting lines stored at ancestors of v.
KD Tree - Region of a node
(contd)
A point is stored in the subtree rooted at node

v if and only if it lies in region(v).

KD Trees Range Search
query range
Need only search nodes
whose region intersects
query region.
Report all points in subtrees
whose regions are entirely
contained in query range.
If a region is partially contained
in the query range check
points.
Example Range Search

Query region: gray rectangle

Gray nodes are the nodes visited in this example.

Example Range Search

Node marked with * corresponds to a region that is

entirely inside the query rectangle

Report all leaves in this subtree.

Example Range Search

All other nodes visited (i.e., gray) correspond to regions

that are only partially inside the query rectangle.
- Report points p6 and p11 only
- Do not report points p3, p12 and p13
KD Tree (vs Range tree)
Construction O(dnlogn)
Sort points in each dimension: O(dnlogn)
Determine splitting line (median finding): O(dn)

Space requirements:
KD tree: O(n)
Range tree: O(nlogd-1n)

Query requirements:
KD tree: O(n1-1/d+k) O(n+k) as d increases!
Range tree: O(logdn+k)
Nearest Neighbor (NN)
Search
Given: a set P of n points in Rd
Goal: find the nearest neighbor p of q in P

p = ( x1 , y1 ) q = ( x2 , y2 )
p d = ( x1 - x2 ) 2 + ( y1 - y2 ) 2
q
Euclidean distance
Nearest Neighbor Search
-Variations

r-search: the distance tolerance k-nearest-neighbor-queries: the

is specified. number of close matches is specified.
Nearest Neighbor (NN)
Search
Nave approach
Compute the distance from the query point to
every other point in the database, keeping track of
the "best so far".
Running time is O(n).

p
q
Array (Grid) Structure
(1) Subdivide the plane into a grid of M x N square cells (same size)

(2) Assign each point to the cell that contains it.

(3) Store as a 2-D (or N-D in general) array:

each cell contains a link to a list of points stored in that cell

p1
p1,p2
p2
Array (Grid) Structure
Algorithm
* Look up cell holding query point. p1
q
p2
* First examine the cell containing the query,
then the cells adjacent to the query
(i.e., there could be points in adjacent
cells that are closer).

Comments
* Uniform grid inefficient if points unequally distributed.
- Too close together: long lists in each grid, serial search.
- Too far apart: search large number of neighbors.

* Multiresolution grid can address some of these issues.

Quadtree
N
A tree in which each internal node has up
to four children.

Every node in the quadtree corresponds W E

to a square.

The children of a node v correspond to the S

four quadrants of the square of v.

The children of a node are labelled NE,

NW, SW, and SE to indicate to which
quadrant they correspond.
Quadtree Construction
(data stored at leaves)

400 a
Input: point set P b
while Some cell C contains more than k c
points do d e
Split cell C
Y
g f
end h l
j
i k
X 50, Y 200 0 X 100
SW SE NW NE

X 75, Y 100 c e X 25, Y 300

i h

j k f g l d a b
Query

Partitioning of the plane The quad tree

A(50,50)

D(35,85)

P
B(75,80) SE NE
C(90,65) SW
NW
B(75,80)
A(50,50) E D SE
NE
SW NW

E(25,25) C

To search for P(55, 75):

Since XA< XP and YA < YP go to NE (i.e., B).
Since XB > XP and YB > YP go to SW, which in this case is null.
 Quadtree Nearest Neighbor
Query

SW
X1,Y1
NE
NW SE

X2,Y2
Y

X
 Quadtree Nearest Neighbor
Query

SW X1,Y1 NE
NW SE

X2,Y2
Y NW

X
 Quadtree Nearest Neighbor
Query
SW X1,Y1 NE
NW SE

X2,Y2
SW
Y NW SE NE

X
 Quadtree Nearest Neighbor
Search
Algorithm
Initialize range search with large r
Put the root on a stack
Repeat
Pop the next node T from the stack
q
For each child C of T

if C intersects with a circle (ball) of radius r

around q, add C to the stack
if C is a leaf, examine point(s) in C and
update r

Whenever a point is found, update r (i.e., current minimum)

Only investigate nodes with respect to current r.
Quadtree (contd)

Simple data structure.

Easy to implement.

But, it might not be efficient:

A quadtree could have a lot of empty cells.

If the points form sparse clouds, it takes a while to reach nearest

neighbors.
Nearest Neighbor with KD
Trees

Traverse the tree, looking for the rectangle that contains

the query.
Nearest Neighbor with KD
Trees

Explore the branch of the tree that is closest to the query

point first.
Nearest Neighbor with KD
Trees

Explorethebranchofthetreethatisclosesttothequery
pointfirst.
Nearest Neighbor with KD
Trees

When we reach a leaf, compute the distance to each

point in the node.
Nearest Neighbor with KD
Trees

When we reach a leaf, compute the distance to each

point in the node.
Nearest Neighbor with KD
Trees

Then, backtrack and try the other branch at each node

visited.
Nearest Neighbor with KD
Trees

Each time a new closest node is found, we can update

the distance bounds.
Nearest Neighbor with KD
Trees

Each time a new closest node is found, we can update

the distance bounds.
Nearest Neighbor with KD
Trees

Using the distance bounds and the bounds of the data

below each node, we can prune parts of the tree that
could NOT include the nearest neighbor.
Nearest Neighbor with KD
Trees

Using the distance bounds and the bounds of the data

below each node, we can prune parts of the tree that
could NOT include the nearest neighbor.
Nearest Neighbor with KD
Trees

Using the distance bounds and the bounds of the data

below each node, we can prune parts of the tree that
could NOT include the nearest neighbor.
K-Nearest Neighbor Search

Can find the k-nearest neighbors to a query by

maintaining the k current bests instead of just one.

Branches are only eliminated when they can't have

points closer than any of the k current bests.
NN example using kD trees
d=1 (binary search tree)

5 7 8 10 12 13 15 18
20

7,8,10,12 13,15,18

7,8 10,12 13,15 18

8 ,7 12 ,10 15 ,13 18
NN example using kD trees
(contd)
d=1 (binary search tree)

5 20
7 8 10 12 13 15 18

query
7,8,10,12 13,15,18 17

7,8 10,12 13,15 18

min dist = 1
8 ,7 12 ,10 15 ,13 18
NN example using kD trees
(contd)
d=1 (binary search tree)

5 20
7 8 10 12 13 15 18

query
7,8,10,12 13,15,18 16

7,8 10,12 13,15 18

min dist = 2
min dist = 1
8 ,7 12 ,10 15 ,13 18
KD variations - PCP Trees
Splits can be in directions other than x and y.

Divide points perpendicular

to the axis with widest
spread.

Principal Component
Partitioning (PCP)
KD variations - PCP Trees
Curse of dimensionality
KD-trees are not suitable for efficiently finding the
nearest neighbor in high dimensional spaces.
Query time: O(n1-1/d+k)
Approximate Nearest-Neighbor (ANN)
Examine only the N closest bins of the kD-tree
Use a heap to identify bins in order by their distance
from query.
Return nearest-neighbors with high probability
(e.g., 95%).
J. Beis and D. Lowe, Shape Indexing Using Approximate Nearest-Neighbour Search in
High-Dimensional Spaces, IEEE Computer Vision and Pattern Recognition, 1997.

118
Dimensionality Reduction
Idea: Find a mapping T to reduce the dimensionality
of the data.
Drawback: May not be able to find all similar objects
(i.e., distance relationships might not be preserved)

119