Nonparametric Methods

Jason Corso
SUNY at Bualo

J. Corso (SUNY at Bualo)

Nonparametric Methods

1 / 49

Histograms

Histograms
p(X, Y )

Y =2

Y =1

J. Corso (SUNY at Bualo)

Nonparametric Methods

3 / 49

Histograms

Histograms
p(X)

p(X, Y )

Y =2

Y =1

J. Corso (SUNY at Bualo)

Nonparametric Methods

3 / 49

Histograms

Histograms
p(X)

p(X, Y )

Y =2

Y =1

X
p(Y )

J. Corso (SUNY at Bualo)

Nonparametric Methods

3 / 49

Histograms

Histogram Density Representation

Consider a single continuous variable x and lets say we have a set D
of N of them {x1 , . . . , xN }. Our goal is to model p(x) from D.

J. Corso (SUNY at Bualo)

Nonparametric Methods

4 / 49

Histograms

Histogram Density Representation

Consider a single continuous variable x and lets say we have a set D
of N of them {x1 , . . . , xN }. Our goal is to model p(x) from D.

Standard histograms simply partition x into distinct bins of width i

and then count the number ni of observations x falling into bin i.

J. Corso (SUNY at Bualo)

Nonparametric Methods

4 / 49

Histograms

Histogram Density Representation

Consider a single continuous variable x and lets say we have a set D
of N of them {x1 , . . . , xN }. Our goal is to model p(x) from D.

Standard histograms simply partition x into distinct bins of width i

and then count the number ni of observations x falling into bin i.

To turn this count into a normalized probability density, we simply

divide by the total number of observations N and by the width i of
the bins.
This gives us:
ni
pi =
N i

(1)

Hence the model for the density p(x) is constant over the width of
each bin. (And often the bins are chosen to have the same width
i = .)
J. Corso (SUNY at Bualo)

Nonparametric Methods

4 / 49

Histograms

Bin Number
Bin Count

J. Corso (SUNY at Bualo)

0
3

1
6

2
7

Nonparametric Methods

5 / 49

Histograms

Histogram Density as a Function of Bin Width

5
0
5
0
5
0

= 0.04

0.5

0.5

0.5

= 0.08

0
= 0.25

J. Corso (SUNY at Bualo)

Nonparametric Methods

6 / 49

Histograms

Histogram Density as a Function of Bin Width

The green curve is the underlying true
density from which the samples were
drawn. It is a mixture of two Gaussians.

5
0
5
0
5
0

J. Corso (SUNY at Bualo)

Nonparametric Methods

= 0.04

0.5

0.5

0.5

= 0.08

0
= 0.25

7 / 49

Histograms

Histogram Density as a Function of Bin Width

The green curve is the underlying true
density from which the samples were
drawn. It is a mixture of two Gaussians.
When is very small (top), the
resulting density is quite spiky and
hallucinates a lot of structure not
present in p(x).

J. Corso (SUNY at Bualo)

Nonparametric Methods

5
0
5
0
5
0

= 0.04

0.5

0.5

0.5

= 0.08

0
= 0.25

7 / 49

Histograms

Histogram Density as a Function of Bin Width

The green curve is the underlying true
density from which the samples were
drawn. It is a mixture of two Gaussians.
When is very small (top), the
resulting density is quite spiky and
hallucinates a lot of structure not
present in p(x).

5
0
5
0
5
0

= 0.04

0.5

0.5

0.5

= 0.08

0
= 0.25

When is very big (bottom), the resulting density is quite smooth

and consequently fails to capture the bimodality of p(x).

J. Corso (SUNY at Bualo)

Nonparametric Methods

7 / 49

Histograms

Histogram Density as a Function of Bin Width

The green curve is the underlying true
density from which the samples were
drawn. It is a mixture of two Gaussians.
When is very small (top), the
resulting density is quite spiky and
hallucinates a lot of structure not
present in p(x).

5
0
5
0
5
0

= 0.04

0.5

0.5

0.5

= 0.08

0
= 0.25

When is very big (bottom), the resulting density is quite smooth

and consequently fails to capture the bimodality of p(x).
It appears that the best results are obtained for some intermediate
value of , which is given in the middle figure.
In principle, a histogram density model is also dependent on the
choice of the edge location of each bin.
J. Corso (SUNY at Bualo)

Nonparametric Methods

7 / 49

Histograms

Analyzing the Histogram Density

What are the advantages and disadvantages of the histogram density

estimator?

J. Corso (SUNY at Bualo)

Nonparametric Methods

8 / 49

Histograms

Analyzing the Histogram Density

What are the advantages and disadvantages of the histogram density

estimator?
Advantages:
Simple to evaluate and simple to use.
One can throw away D once the histogram is computed.
Can be computed sequentially if data continues to come in.

J. Corso (SUNY at Bualo)

Nonparametric Methods

8 / 49

Histograms

What can we learn from Histogram Density

Estimation?
Lesson 1: To estimate the probability density at a particular location,
we should consider the data points that lie within some local
neighborhood of that point.
This requires we define some distance measure.
There is a natural smoothness parameter describing the spatial extent
of the regions (this was the bin width for the histograms).

J. Corso (SUNY at Bualo)

Nonparametric Methods

9 / 49

Histograms

What can we learn from Histogram Density

Estimation?
Lesson 1: To estimate the probability density at a particular location,
we should consider the data points that lie within some local
neighborhood of that point.
This requires we define some distance measure.
There is a natural smoothness parameter describing the spatial extent
of the regions (this was the bin width for the histograms).

Lesson 2: The value of the smoothing parameter should neither be

too large or too small in order to obtain good results.

J. Corso (SUNY at Bualo)

Nonparametric Methods

9 / 49

Histograms

What can we learn from Histogram Density

Estimation?
Lesson 1: To estimate the probability density at a particular location,
we should consider the data points that lie within some local
neighborhood of that point.
This requires we define some distance measure.
There is a natural smoothness parameter describing the spatial extent
of the regions (this was the bin width for the histograms).

Lesson 2: The value of the smoothing parameter should neither be

too large or too small in order to obtain good results.
With these two lessons in mind, we proceed to kernel density
estimation and nearest neighbor density estimation, two closely
related methods for density estimation.

J. Corso (SUNY at Bualo)

Nonparametric Methods

9 / 49

Kernel Density Estimation

The Space-Averaged / Smoothed Density

Consider again samples x from underlying density p(x).
Let R denote a small region containing x.

The probability mass associated with R is given by

!
P =
p(x! )dx!

(2)

J. Corso (SUNY at Bualo)

Nonparametric Methods

10 / 49

Kernel Density Estimation

The Space-Averaged / Smoothed Density

The expected value for k is thus
E[k] = nP

J. Corso (SUNY at Bualo)

Nonparametric Methods

(4)

11 / 49

Kernel Density Estimation

The Space-Averaged / Smoothed Density

The expected value for k is thus
E[k] = nP

(4)

The binomial for k peaks very sharply about the mean. So, we expect
k/n to be a very good estimate for the probability P (and thus for
the space-averaged density).
This estimate is increasingly accurate as n increases.
relative
probability
1

0.5

20 50
0

100
P = 0.7

k/n

FIGURE 4.1. The relative probability an estimate given by Eq. 4 will yield a particular

J. Corso (SUNY at Bualo)

Nonparametric Methods

11 / 49

Kernel Density Estimation

Practical Concerns

Practical Concerns
The validity of our estimate depends on two contradictory
assumptions:
1

The region R must be suciently small the the density is

approximately constant over the region.
The region R must be suciently large that the number k of points
falling inside it is sucient to yield a sharply peaked binomial.

J. Corso (SUNY at Bualo)

Nonparametric Methods

14 / 49

Kernel Density Estimation

Practical Concerns

Practical Concerns
The validity of our estimate depends on two contradictory
assumptions:
1

The region R must be suciently small the the density is

approximately constant over the region.
The region R must be suciently large that the number k of points
falling inside it is sucient to yield a sharply peaked binomial.

Another way of looking it is to fix the volume V and increase the

number of training samples. Then, the ratio k/n will converge as
desired. But, this will only yield an estimate of the space-averaged
density (P/V ).

J. Corso (SUNY at Bualo)

Nonparametric Methods

14 / 49

Kernel Density Estimation

Practical Concerns

Practical Concerns
The validity of our estimate depends on two contradictory
assumptions:
1

The region R must be suciently small the the density is

approximately constant over the region.
The region R must be suciently large that the number k of points
falling inside it is sucient to yield a sharply peaked binomial.

Another way of looking it is to fix the volume V and increase the

number of training samples. Then, the ratio k/n will converge as
desired. But, this will only yield an estimate of the space-averaged
density (P/V ).
We want p(x), so we need to let V approach 0. However, with a
fixed n, R will become so small, that no points will fall into it and
our estimate would be useless: p(x) # 0.

J. Corso (SUNY at Bualo)

Nonparametric Methods

14 / 49

Kernel Density Estimation

Practical Concerns

Practical Concerns
The validity of our estimate depends on two contradictory
assumptions:
1

The region R must be suciently small the the density is

approximately constant over the region.
The region R must be suciently large that the number k of points
falling inside it is sucient to yield a sharply peaked binomial.

Another way of looking it is to fix the volume V and increase the

number of training samples. Then, the ratio k/n will converge as
desired. But, this will only yield an estimate of the space-averaged
density (P/V ).
We want p(x), so we need to let V approach 0. However, with a
fixed n, R will become so small, that no points will fall into it and
our estimate would be useless: p(x) # 0.
Note that in practice, we cannot let V to become arbitrarily small
because the number of samples is always limited.
J. Corso (SUNY at Bualo)

Nonparametric Methods

14 / 49

Kernel Density Estimation

Practical Concerns

How can we skirt these limitations when an unlimited number of samples

if available?
To estimate the density at x, form a sequence of regions R1 , R2 , . . .
containing x with the R1 having 1 sample, R2 having 2 samples and
so on.

J. Corso (SUNY at Bualo)

Nonparametric Methods

15 / 49

Kernel Density Estimation

Practical Concerns

limn Vn = 0 ensures that our space-averaged density will converge

to p(x).

J. Corso (SUNY at Bualo)

Nonparametric Methods

16 / 49

Kernel Density Estimation

Practical Concerns

limn Vn = 0 ensures that our space-averaged density will converge

to p(x).
limn kn = basically ensures that the frequency ratio will
converge to the probability P (the binomial will be suciently
peaked).

J. Corso (SUNY at Bualo)

Nonparametric Methods

16 / 49

Kernel Density Estimation

Practical Concerns

limn Vn = 0 ensures that our space-averaged density will converge

to p(x).
limn kn = basically ensures that the frequency ratio will
converge to the probability P (the binomial will be suciently
peaked).
limn kn /n = 0 is required for pn (x) to converge at all. It also says
that although a huge number of samples will fall within the region
Rn , they will form a negligibly small fraction of the total number of
samples.

J. Corso (SUNY at Bualo)

Nonparametric Methods

16 / 49

Kernel Density Estimation

Practical Concerns

limn Vn = 0 ensures that our space-averaged density will converge

to p(x).
limn kn = basically ensures that the frequency ratio will
converge to the probability P (the binomial will be suciently
peaked).
limn kn /n = 0 is required for pn (x) to converge at all. It also says
that although a huge number of samples will fall within the region
Rn , they will form a negligibly small fraction of the total number of
samples.
There are two common ways of obtaining regions that satisfy these
conditions:

J. Corso (SUNY at Bualo)

Nonparametric Methods

16 / 49

Kernel Density Estimation

Practical Concerns

limn Vn = 0 ensures that our space-averaged density will converge

to p(x).
limn kn = basically ensures that the frequency ratio will
converge to the probability P (the binomial will be suciently
peaked).
limn kn /n = 0 is required for pn (x) to converge at all. It also says
that although a huge number of samples will fall within the region
Rn , they will form a negligibly small fraction of the total number of
samples.
There are two common ways of obtaining regions that satisfy these
conditions:
1

Shrink an initial region

by specifying the volume Vn as some function
of n such as Vn = 1/ n. Then, we need to show that pn (x) converges
to p(x). (This is like the Parzen window well talk about next.)

J. Corso (SUNY at Bualo)

Nonparametric Methods

16 / 49

Kernel Density Estimation

Practical Concerns

limn Vn = 0 ensures that our space-averaged density will converge

to p(x).
limn kn = basically ensures that the frequency ratio will
converge to the probability P (the binomial will be suciently
peaked).
limn kn /n = 0 is required for pn (x) to converge at all. It also says
that although a huge number of samples will fall within the region
Rn , they will form a negligibly small fraction of the total number of
samples.
There are two common ways of obtaining regions that satisfy these
conditions:
1

Shrink an initial region

by specifying the volume Vn as some function
of n such as Vn = 1/ n. Then, we need to show that pn (x) converges
to p(x). (This is like the Parzen window well talk
about next.)
Specify kn as some function of n such as kn = n. Then, we grow the
volume Vn until it encloses kn neighbors of x. (This is the
k-nearest-neighbor).

Both of these methods converge...

J. Corso (SUNY at Bualo)

Nonparametric Methods

16 / 49

Kernel Density Estimation

n=1

Vn =1/ n

kn = n

n=4

Practical Concerns

n=9

n = 16

n = 100

...

...

...

...

FIGURE 4.2. There are two leading methods for estimating the density at a point, here
at the center of each square. The one shown in the top row is to start with a large
volume
centered on the test point and shrink it according to a function such as Vn = 1/ n. The
other method, shown in the bottom row, is to decrease the volumein a data-dependent
way, for instance letting the volume enclose some number kn = n of sample points.
The sequences in both cases represent random variables that generally converge and
allow the true density at the test point to be calculated. From: Richard O. Duda, Peter
c 2001 by John Wiley17&
Classification
. Copyright "
E. J.Hart,
David
G. Stork, PatternNonparametric
Corso and
(SUNY
at Bualo)
Methods
/ 49