Decompose USE into bins and variance

MSE l f ,
f) :
ET If (X0) -
f'(x) ( 2
] = LET Y'CXOC ] -

f(x) 5 + var >

FCXOLC
First , find the of f'nbias (x0)

2 f(x)/8X
F'(x) = Wor t . No Not Xi

i IiE Xi
10
-

=
, K'(

#[f' (xO] =
ii ZE , Elk') *** >]

= Elk ( * X017 ind

= Sk( ** > f(x)dx

=
i SkCusf(un Xo) du + Mo Y (x . -

X0)

Here , we
may not use the mean value theorem to
replace f(uh + Xo) Since

the integration part contains K(u) instead of K(u).

f(uh+) - orderot ,
B
Instead ,
use taylor's expansion to expand .
p

f f(8-1(xo \uhyB-
fi tub)
+

f (uh + xo) =
f(x) +
(an) +
nee
(wh +.

(B- 1) :
contain B-3 terms -
remainder term

where Xo += Uh [xo ,
Xo + Uh]
Plug EIf'(x01]
this into

#[f(xoS] in Sk (n) =
fixo) du + in 9 k(n) fixo) wh du

-
[
in Sicus faces chdu +.+
(
Skcu) frotun cubs da

Contain B-3 terms

in /k' (n) du i f(x)k(u)

The first term is f(xo) = =
0 by 11

second calculated
The term can be
by integration by part

Sk'cn)udu =
uk(u) -

Sklu) do W= W do = dn v k(a) du=

=
k'(r) du

=
UK(u) -
1

↓
f'(x) SK (w)ndn = -

f'(xo)Uk(u) +
f' (x) =
f'(x0) by C

This term cancels out with -f'(x) in bias calculation

The B-3 calculated

remaining terms can be
by integration by part.

Let at [2 , B-2] , GER.

& k'lasnadu = n9km)- aSkcusuda = nakcu) by 22

i Skicusfaxes Cancadu = " SKCuda = 0

by 2

Thus ,
every other terms cancel out except the remainder term in the bias

IEI f(xOC] -
f'(x)

I Skcu) fettu (B d
1We
=

o + tuh) -
f(x)
n
=

<Bhs : Sk(u) fiB (0) \nhB"du

·
Skcus [f- (o +
enh) -

fB-P(xos] Lunc
+
du

"
=
<is SkcnsfP(oteah) f8-P(e) - Cut do
& i als : J1k)) IfBr" (xo + EUS -
-18 (x) & \nhs9-du
= ha : SIkcus) < Inh ChP"du by Holder's
property
LI
bo"supu /K Cas /S us de
(B-) !

=
Che where c = supulk'cus/Sudde so
Next , find the variance of Fr(XO
van)f(x) = var (wh Z k'( *** 1)
= whe var (1 ( *** I ind
= nE(k ( *** )(
= n Sk( *** ) f(x.) dx
.

has SK (n) f(ub + x) do

* Xo
-

=
n= h
I

· supuK (n) Sf(ub + X du

= G was wherecn =
suyuk Las If (un +
Xo) do

We know that

(f f) C: rB Ch
supf USE ,
=
+
-

2
haB wishs
+ 1

Choose h so that ,
then hi n

-18-2)/
RB+
3/28 +
Thus supfMSE (f f) C C wil
= +
, .

- 2(B 1)/(B + 1)
-

= (C , + G) n

-
(B x/kB +
1)
-

0(n
=
① G holds
tricially by the indicator function .

& We know that 90j] is a

complete orthonormal basis on L(E-1 13) , ee
i

Si Uj (a)
&j(u)du
= 1

Since [rj]EL (Ey , 1) ,

then Oj(m) =
[Andlu) where Sass -.. 3 is

square-summable sequence infinite sequence that the of

.

a i e . an sum
,

each elements
squared converge .
For i t 90 , 1. . .
-

B c)
-

↓ i km) du =
J Ijeojcos0j(u) 1914 *
1] de by def of Los
=
Si n jajlos UJ Lus d

=
Injjlos Ep Ulus da

=
I 10) I a S, wicklunde by dominate
convergence theorem

:
=

o by or the normality
9 kusdu
:

When := 0 ,
= 0 =

02
When i so , gui kln)dn = = 0

③ Since K is a polynomial and the domain is on the compact set [ I

-
. 17 .

as' also holds

c) is not pdf -

There is no Kernel
satisfying 4 , Cr, C and is also pdf .

Prove by contradiction .

Assume I
following C .. C, G is pdf
By Lemma 2 2
. . 5 , K( ) > .
0

By C2 , Suklu) du =

uk(n) 0
=
a. .
e

k(u) = 0

Then Siku) duzo this contradicts with C.

First , we want to find the
expression for
the bias and variance .

Here , we consider the case BC2 , then we need to

apply the taylor expansion
bias = ETFn (o)] -

for exe

=
S If(xauh) -

fLxo3] klms de

f" f(B-(xo subs"- f(x1] Klasdn

fil
(X0) + = wb)
=
/[f(x) +
(ab) +
nee
2 ! (wh +.

(B- 1) :
contain B-3 terms -
remainder term

where Xo += Uh [xo ,
Xo + Uh]

=
Sf'(o)whk(n) + & (h) k(a) +

=
JIjf(o) Lub Klus/51 dn +
&
fig-(o +
Enhsh"klws da.
(B - 13 !

= Jac fis cub Klas/jhdeagus18't"Gottuns-f'd" KLusdm (0) .

=
I f(x295 3
+hi gu gcotuh-fookLud
is even
, . .

Next evaluate the second term in the bias .

B gu , gottuk-fokLud . .

% gue glcotuk-fokLud a
>has "
-

Jiah) Kasdn by Holder's condition

-h guSt
kn Enh = uh
Then ,
we can write Bias asf" (0) + 0 (h)
Next , we find the format of the variance
+
Var (In(t) = Vardns[k ( 2**
=
nh var(k( *** () ind

= is [Elk[xix1] -
EEKLY 13]
= ↓ ( f()Skuldn O(1)-Ek( *** <]) plug dias
+
in

- (
=

f(x)(k(u)dn O() [f(x0) -"(x) 0(4452 +

-
+
+

= (In f(x)
·

f(x) + O(k)

= nh f(x) / +
0(t) f(x)S kcustu =I

suppose hn = 15

By Lindberg -
Feller theorem ,

↑ nn (X0) -
Elfhm(x0)] In n ) - Ek(
Var
[frn (XO)J' La [Var(Xi n))
-
Xo

= Ik(Xi]x) -
ETk(x1]
& [ var (E , >
= N(0 1).

Since Var(fucos) = win f(x)/ + Oct .

Then whWar(fn(x) >
-

f(x)/ as new .

By Slutsky's theorem
EnexoS El Enxos]
(fn(x0) ElEnkos]) Var(Enixos)]'k
-

dnd -
=
Inh
Var
(fn(0)1
=> N(0 , f(x) (2)

Since bias = f"(x) +

Olh) ,
taking h = vills,
 - 2/5
n

bias = 6 f" (x0) +

0 (1215s

By Slutsky's theorem ,

dnb [f(0) -
f(x0)] Es N2f"(xo) /6 , f(x) ((

This distribution is not mean-zero .

From the previous part , we know that

↓ n4 (fn(x0) E[Enko(T)
-
N20 f(x) (2) = ,

Sinc bias = f"(x) Olh) by Slutcky's

+
, Lemma

duh (fn(x0) -
f(x)) => N10, f(x) (2)
 By the bing expression
(Eanxo)) (fnkol
Fankos EnKos
-
Bias -
Bias
# y =

-
=

(h(z -
f "(x0) + o(h)

= bf"(x0) + O(b))

= Bias (fn(x)) + 0 (4)

By rearranging ,
we
get Bias (fn(x) 5 (fan()- fnx))
-
=
OCn

Next , evaluate the variance

Var (fu(x0) 5(frn(x0) fn(xo) (

- -

: var > Fr(x)) + ↑ var(Fan(x)) - Cor(Fn(x0) ,

fan(0)
/ wh
=

f(x)k + inf(x)k -

&Cor(fne) · fan(e)) + O 1)
= ni fixol -
En Cor(fuko) · Frn(x)( + 0 (t)

Take a look at the covariance term.

Cor(fn(x) · Frn(xo) ) = [Kn(X 1

-

Xo) Kan (x -
x0)] + O(1

= Skn(x . -0) Kah(x , -xo) f(x1)dx .

+
0(1)
=
zinSkla)k(2) f(xo + Uh) da + Pla
= 2 f(xosSkLas Kluk) da +OL

= an +
P(
1

Var (fn(x0) -
51 fan (x0) -

Eu(x)
=
isin f(x) -
- In f(x) + 0 (i)
= win f(x) + 0 (t)

By Lindeberg -
Feller CLT and Slutsky's Lemma ,

dub [fn(x - 5 (fan (10)

-

Enke)] = N20 , f(x)

 Before need
proceeding , we to preve the
following lemma .

Chemma) 25 X, Y are iid , then var(Y) = El (X-45]

# :
I E[(x -
45] : El Et (x -
%5143] law of total expectation

= E[E[X -
2xy + y 1 % 3]

= I El EIX] -

2YE(X(Y) +
Y]
= ELEIY]-2YE[Y] + /23 by id
= I Elvarly) + (EYS-2YETY) +
Y 2]
= IE[ var() +
(EY -
Y( ]
*

= var (4) I

Denote : (X ., X Xi-1 Xits Xa)

Xi
of
:
·

and Xi' is the

ghost sample
-

Xi
-
=
-

,
,

This construction of %: gives (Y: 1X) IXj :

j+i·
Then ,
we can calculate

var (f(x) (Xj :

j + i) = ET (f(X) -

f(Y)(Xj j + i] :
by independence and the lemma

EEEEIIX-Y /5 1Xj : :
jti] by Lipchitz property
~
[E[/X : -
X: 'R] X and % :
Only have one
different element

= oar (Xi) by lemma

varif(x)) =
I :E
E[(f(x) ETf(x) (Xj jti])) by Efron-Steve inequality
- :

=
IE
El El(f(x) E[f(x) (Xj jti])"(Xj j law of total expectation
- : : + :])

= Zivar (i)

before proceeding . first following Lemma

prove the

Clemma 2) For a
differentiable convex functions , .
OXY in the domain of g
.
g(y)
-

g(x))yg(x) ,
y
-
x]

Of Fix X , y in the domain of g

.

By convexity . EtE Lo , K ,

(k +(x)
g(x +
+(y -
x)) =
g(ty + =
+g(y) + ( -

t)g(x) .

definition of derivative also have

By the ,
he

g(x +
+ (y x)) -
=
g(x) + + >
Og(x) ,
y
-
x3 + O(t)

We
may
conclude that g(y) -g(x)> <
g(x) ,
yex > + "

We know that El f(x) -E(f(x))xj jfi)] : =

infEICE Elf(x)-Zig2
Let Y : =
infxietoy & IX , x2)"Xi-1 , Xis Lit --
Xr)

Since is differentiable , and thus it is continuous .

Since the domain is a compact set , f attains its minimum .

Let Wi =
min
argmitous & (X ,
X ·
Xin Xi , Xit , ... Xn)

By continuity of f , Y := infxEtOBlQf(X ., X-Xiv . Xis Xit ...

Xm) .

Defince Fas r(X ., X

... Xin , Xit ...
Xn)

For x :E to 1]eR .
, fLXX-Xiv is XitssXn) is fil-many
Then Yi is also Fil -
meas
We know &Wi <
w] = Socw3n[f(X ,
:
Xin , W, xiti .... Xn >
Yiz ,
thus W: is also

↓is -
measurable·

By Lemma

# f(x) -
El f(x) /X :
jti]] =
EEfWs-Yi]
=
EEf(x) f(X -

,
...
Xin , Wis Xinc"--Xn)]

>
-
EI (x))(x: -
Wil]
·
El(2
By E from -Stein inequality,
var (f(x))
= [ Et f(x) -

E(f(x) (j :
jei]

= In EI((x()
=
E[l0f(x)12]
 Let f(x) = Zi Xi

Then for X , xEIR,

|f *
. (x) -

f(x))) =
1 [(X :
-

Xic)
= I 1x -X: 1 by Triangle's inequality
=
/IX-XII ,

This shows that fo *

E F,

var (f * (x() =
var (ii) = : Var(x:

Let &* (X) =

c where ceR is constant.

It's clear that fat e F2

E[ fc (x)112]
*
var (f(x)) =
0 = 110