HSP 511: Economics Lab

Lecture 3
Indian Institute of Technology Delhi

Contents
The Bootstrap 1
Why Does the Bootstrap Work? 3

Let X1 , . . . , Xn ∼ F be random variables distributed according to F , and

θ = T (F )
be some
R
object of interest that we want to estimate using the data. For example, if
θ = xdF (x), then its estimator is
n
1X
θ̂ = Xi ,
n i=1

with Var(θ̂) = σ 2 /n, where σ 2 = Var(Xi ).

※ The Bootstrap
The plug-in estimator of θ is defined as
θ̂ = T (F̂ ) = g(X1 , . . . , Xn )
In the Bootstrap world, we simulate X1∗ , . . . , Xn∗ from F̂ and then compute the
estimator over these values, θ̂∗ = g(X1∗ , . . . , Xn∗ ).

Real world F =⇒ X1 , . . . , Xn =⇒ F̂ =⇒ θ̂ = g(X1 , . . . , Xn )

Bootstrap world F̂n =⇒ X1∗ , . . . , Xn∗ =⇒ F̂ ∗ =⇒ θ̂∗ = g(X1∗ , . . . , Xn∗ )

Bootstrap Variance Estimation

1. Draw X1∗ , . . . , Xn∗ ∼ F̂n .
2. Compute θ̂∗ = g(X1∗ , . . . , Xn∗ ).
3. Repeat steps 1 and 2, B times, to get θ̂1∗ , . . . , θ̂B
∗
.
4. Let v
u
B B
!2
1 X 1 X
u
θ̂b∗ − θ̂∗
u
v̂b = .
B r=1 r
t
B b=1

1
Lecture Notes The Bootstrap
 
The next theorem states that v̂b2 approximates Var θb . The are two sources
of error in this approximation. The first is due to the fact that n is finite and the
second is due to the fact that B is finite. However, we can make B as large as we
like. (something like B = 10, 000 suffice in practice.)

Theorem 1.1
v̂b2 P
Under appropriate regularity conditions, Var(θ̂)
→ 1 as n → ∞.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

y = np.random.rand(100)
tmean=np.mean(y)
resample=np.mean(np.random.choice(y,size=(100,10000),replace=True),
axis=0)

print('The variance for tmean is: {}'.format(np.var(resample)))

## The variance for tmean is: 0.0007704722692846294

Now we describe the confidence interval algorithm.

Bootstrap Confidence Interval

1. Draw a bootstrap sample X1∗ , . . . , Xn∗ ∼ F̂ . Compute θ̂∗ =
g(X1∗ , . . . , Xn∗ ).
2. Repeat the previous step, B times, yielding estimators θ̂1∗ , . . . , θ̂B
∗
.
3. Let  
1 XB
θbj∗ − θb
Ĝ(t) = 1 ≤ t .
B j=1 v̂b
4. Let h i
Cn = θb − v̂b t1−α/2 , θb − v̂b tα/2 ,
b −1 (α/2) and t b −1
where tα/2 = G 1−α/2 = G (1 − α/2).
5. Output Cn .

The next states that our Bootstrap confidence has right coverage asymptotically.

Theorem 1.2
Under appropriate regularity conditions,
!
1
Pr(θ ∈ Cn ) = 1 − α + O √ .
n

2
Lecture Notes Why Does the Bootstrap Work?

x_axis = np.arange(-3, 3, 0.01)

plt.hist((resample-tmean)/np.std(resample), bins=50,
density= True)
plt.plot(x_axis, norm.pdf(x_axis, 0, 1))
plt.show()

0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
4 2 0 2 4

※ Why Does the Bootstrap Work?

To explain why the bootstrap works, let us begin with a heuristic. Very often
√ d
n(T (F̂ ) − T (F )) →
− F,

for some distribution F . Main asymptotic justification of the bootstrap is that

conditional on X1 , X2 , . . .
√ d
n(T (F̂ ∗ ) − T (F̂ )) →
− F.

Example 2.1
R
Suppose that θ = xdF (x), then its estimator is
n
1X
θ̂ = Xi .
n i=1

3
Lecture Notes Why Does the Bootstrap Work?

We have asymptotic normality, under the condition that σ = Var(Xi ) < ∞

√ d
− N (0, σ 2 ).
n(θ̂ − θ) →

Lets also define bootstrap estimator as

n
1X
θ̂∗ = X ∗.
n i=1 i

Then for every sequence X1 , X2 , . . .

√ d
n(θ̂∗ − θ̂) →
− N (0, σ 2 ).

Along with asymptotic normality, we also have Edgeworth expansion, for each
t
√  !
n(θ̂ − θ) p 1 (t)ϕ(t) p j (t)ϕ(t) 1
Pr  ≤ t − Φ(t) = √ + ... + + ... = O √ ,
σ̂ n nj/2 n

where ϕ, Φ is pdf, cdf of normal distribution and for κ3 skewness, κ4 kurtosis

1
p1 (t) = − κ3 (t2 − 1)
6
1 1

p2(x) = −t κ4 (t2 − 3) + κ23 (t4 − 10t2 + 15) .
24 72
Similar expansion can be found for bootstrap version as
 
θbj∗ − θb p̂∗1 (t)ϕ(t) p̂∗j (t)ϕ(t)
Pr  ≤ t − Φ(t) =
 √ + ... + + ....
v̂b n nj/2
√ √ ∗
Since in bootstrap we are approximating distribution of n(σ̂θ̂−θ) with n(σ̂θ̂∗ −θ̂) ,
we have
  √ 
θbj∗ − θb n(θ̂ − θ) [p̂∗ (t) − p1 (t)]ϕ(t)
Pr  ≤ t − Pr  ≤ t = 1 √ + ....
v̂b σ̂ n
 
One can show that p̂∗1 (t) − p1 (t) = O √1
n
. With this we have
  √ 
θbj∗ − θb n(θ̂ − θ) 1
 
Pr  ≤ t − Pr
  ≤t =O
 .
v̂b σ̂ n