Scribd
Upload
70 views13 pages

Non-Parametric Estimation: Rajkumar Saha (WS)

This document outlines non-parametric estimation techniques. It begins by defining the empirical CDF and showing graphs of the empirical CDF for different sample sizes. It then discusses useful inequalities like Hoeffding's inequality. It also covers density estimation techniques like histograms and kernel density estimation. It discusses choosing the bandwidth for kernel density estimation using cross-validation and states Stone's theorem about the consistency of cross-validation bandwidth selection.

Uploaded by

Raj Kumar Saha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
70 views13 pages

Non-Parametric Estimation: Rajkumar Saha (WS)

This document outlines non-parametric estimation techniques. It begins by defining the empirical CDF and showing graphs of the empirical CDF for different sample sizes. It then discusses useful inequalities like Hoeffding's inequality. It also covers density estimation techniques like histograms and kernel density estimation. It discusses choosing the bandwidth for kernel density estimation using cross-validation and states Stone's theorem about the consistency of cross-validation bandwidth selection.

Uploaded by

Raj Kumar Saha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Outline

Non-Parametric

Non-Parametric Estimation
Rajkumar Saha(ws)
University of South Carolina

April 30, 2015

1 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

2 / 13

Non-Parametric

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Non-parametric
Empirical CDF:
Definition: The empirical distribution function Fn (x) is the
CDF that puts mass 1/n at each data point Xi . Formally,
n

1X
Fn (x) =
I (Xi x)
n
i=1

where
(
1, if Xi x;
I (Xi x) =
0, if Xi > x;

3 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Graphs of the empirical CDF for different value of n:

Fhat
0

0
t

n=100

n=1000

Fhat

0.4
0.0

0.4
0.0

Fhat

0.8

0.8

4 / 13

0.4
0.0

0.4
0.0

Fhat

0.8

n=50

0.8

n=10

Rajkumar Saha(ws)

0
t

Non-Parametric Estimation

Outline

Non-Parametric

Useful inequalities
Hoeffdings Inequality: Let Y1 , Y2 , ......, Yn be independent
observations such that E (Yi ) = 0 and ai Yi bi . Let
 > 0. Then, for any t > 0,
!
n
n
X
Y
2
2
t
P
Yi  e
e t (bi ai ) /8 .
i=1

i=1

Hoeffdings Inequality for Bernoulli Random Variables.


Let X1 , X2 , ......, Xn Bernoulli(p). Then, for any  > 0,
2

P(kXn pk > ) 2e 2n


where Xn = n1
5 / 13

Pn

i=1 Xi

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Useful inequalities more general version


Hoeffdings Inequality: Let X1 , ......, Xn be independent
real-valued random variables. Let a1 , b1 , ......an , bn R, and
assume thatXi [ai , bi ] with probability one (i = 1, ......, n)
Then, from all  > 0

( n
)
2
1 X

1 Pn 2n
2


bi ai |
|
n
i=1
P
(Xi E (Xi )) >  2e
n

i=1

Hoeffdings Inequality for Bernoulli Random Variables.


LetX1 , X2 , ......, Xn Bernoulli(p). Then, for any  > 0,
2

P(kXn pk > ) 2e 2n


Where Xn = n1
6 / 13

Pn

i=1 Xi

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Density estimation
Underlying idea: Let F be a distribution with probability
density f = F 0 and let
X1 , X2 , ......, Xn F be an IID sample from F . The goal of
the nonparametric density estimation is to estimate f with
as few assumptions about f as possible.
Histograms: Assuming f supported on [0,1]. Let m be an
integer anddefine bins




B1 = 0, m1 , B2 = m1 , m2 , ........., Bm = m1
m , 1 . Thus
binwidth h = 1/m
Let Yj beR the number of observations in Bj , let pj = Yj /n and
let pj = Bj f (u)du
The the histogram estimator is define by
m
X
pj

fn (x) =
I (x Bj ).
h
j=1

7 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Graphs of the the histogram estimator for n=235, Singer


data:

60

Alto 2

65

70

75

60

Alto 1

Soprano 2

65

70

75

Soprano 1
40
30

Percent of Total

20
10
0

Bass 2

Bass 1

Tenor 2

Tenor 1

40
30
20
10
0
60

65

70

75

60

65

70

75

Height (inches)

8 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Kernel density estimation


Definition:Given a kernel K and a positive number h, called
the bandwidth, the Kernel density estimator is defined to
be


n
1X1
x Xi

fn (x) =
K
n
h
h
i=1

For verifying performance:


Theorem. Assume that f is continuous at x and that
p
hn 0 and nhn as n . Then fn (x) f (x).

9 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Graphs of the the Kernel Density:

0.020
0.010

Density function

0.04
0.03
0.00

0.000

0.01

0.02

Density

0.05

0.06

0.030

density.default(x = x)

10

20

30

N = 50 Bandwidth = 2.15

10 / 13

Rajkumar Saha(ws)

20

40

60

80
x

Non-Parametric Estimation

100

120

Outline

Non-Parametric

We have to choose h
Cross validation estimator:
Definition. The cross-validation estimator of risk is
Z 
n
2
2X

f(i) (Xi )
J(h) =
fn (x) dx
n
i=1

We f(i) is the density estimator obtained after removing the


as the cross-validation score
i th observation. We refer to J(h)
or estimated risk.

11 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

Stones Theorem
Theorem: Suppose that f is bounded. Let Jh denote the
kernel estimator with bandwidth h denote the bandwidth
chosen by cross-validation. Then,
2
f (x) fh (x) dx a.s.
1.
2
R
infh f (x) fh (x) dx
R

A very good approximation:


XX
= 1
J(h)
K
hn2

Xi xj
h

2
1
K (0) + O( 2 )
nh
n
i
j
Z
where K (x) = K (2) (x) 2K (x) and K (2) (z) = K (z y )K (y )dy
12 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

Outline

Non-Parametric

R-code:

13 / 13

Rajkumar Saha(ws)

Non-Parametric Estimation

You might also like