See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/37434447

Bootstrap Methods and Their Application

Article  in  Technometrics · January 1997

DOI: 10.2307/1271471 · Source: OAI

CITATIONS READS

4,359 21,218

2 authors, including:

D. V. Hinkley
University of California, Santa Barbara
120 PUBLICATIONS   13,450 CITATIONS   

SEE PROFILE

All content following this page was uploaded by D. V. Hinkley on 26 September 2015.

The user has requested enhancement of the downloaded file.

 Bootstrap Methods and their
Application
Anthony Davison

c
2006
A short course based on the book
‘Bootstrap Methods and their Application’,
by A. C. Davison and D. V. Hinkley
c
Cambridge University Press, 1997

Anthony Davison: Bootstrap Methods and their Application, 1

Summary

◮ Bootstrap: simulation methods for frequentist inference.

◮ Useful when
• standard assumptions invalid (n small, data not normal,
. . .);
• standard problem has non-standard twist;
• complex problem has no (reliable) theory;
• or (almost) anywhere else.
◮ Aim to describe
• basic ideas;
• confidence intervals;
• tests;
• some approaches for regression

Anthony Davison: Bootstrap Methods and their Application, 2

Motivation

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 3

Motivation

AIDS data
◮ UK AIDS diagnoses 1988–1992.
◮ Reporting delay up to several years!
◮ Problem: predict state of epidemic at end 1992, with
realistic statement of uncertainty.
◮ Simple model: number of reports in row j and column k
Poisson, mean
µjk = exp(αj + βk ).
◮ Unreported diagnoses in period j Poisson, mean
X X
µjk = exp(αj ) exp(βk ).
k unobs k unobs
◮ Estimate total unreported diagnoses from
period j by replacing αj and βk by MLEs.
• How reliable are these predictions?
• How sensible is the Poisson model?
Anthony Davison: Bootstrap Methods and their Application, 4
Motivation

Diagnosis Reporting-delay interval (quarters): Total

period reports
to end
Year Quarter 0† 1 2 3 4 5 6 ··· ≥14 of 1992
1988 1 31 80 16 9 3 2 8 ··· 6 174
2 26 99 27 9 8 11 3 ··· 3 211
3 31 95 35 13 18 4 6 ··· 3 224
4 36 77 20 26 11 3 8 ··· 2 205
1989 1 32 92 32 10 12 19 12 ··· 2 224
2 15 92 14 27 22 21 12 ··· 1 219
3 34 104 29 31 18 8 6 ··· 253
4 38 101 34 18 9 15 6 ··· 233
1990 1 31 124 47 24 11 15 8 ··· 281
2 32 132 36 10 9 7 6 ··· 245
3 49 107 51 17 15 8 9 ··· 260
4 44 153 41 16 11 6 5 ··· 285
1991 1 41 137 29 33 7 11 6 ··· 271
2 56 124 39 14 12 7 10 263
3 53 175 35 17 13 11 306
4 63 135 24 23 12 258
1992 1 71 161 48 25 310
2 95 178 39 318
3 76 181 273
4 67 133

Anthony Davison: Bootstrap Methods and their Application, 5

Motivation

AIDS data
◮ Data (+), fits of simple model (solid), complex model
(dots)
◮ Variance formulae could be derived — painful! but useful?
◮ Effects of overdispersion, complex model, . . .?

500
400

+ ++
300
Diagnoses

+ ++ +
+ ++ + +
+ + +
+ + +
200

++
+
++ +
+++
100

+
++
+
++ +
++++
0

1984 1986 1988 1990 1992

Time

Anthony Davison: Bootstrap Methods and their Application, 6

Motivation

Goal

Find reliable assessment of uncertainty when

◮ estimator complex
◮ data complex
◮ sample size small
◮ model non-standard

Anthony Davison: Bootstrap Methods and their Application, 7

Basic notions

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 8

Basic notions

Handedness data

Table: Data from a study of handedness; hand is an integer measure

of handedness, and dnan a genetic measure. Data due to Dr Gordon
Claridge, University of Oxford.

dnan hand dnan hand dnan hand dnan hand

1 13 1 11 28 1 21 29 2 31 31 1
2 18 1 12 28 2 22 29 1 32 31 2
3 20 3 13 28 1 23 29 1 33 33 6
4 21 1 14 28 4 24 30 1 34 33 1
5 21 1 15 28 1 25 30 1 35 34 1
6 24 1 16 28 1 26 30 2 36 41 4
7 24 1 17 29 1 27 30 1 37 44 8
8 27 1 18 29 1 28 31 1
9 28 1 19 29 1 29 31 1
10 28 2 20 29 2 30 31 1

Anthony Davison: Bootstrap Methods and their Application, 9

Basic notions

Handedness data

Figure: Scatter plot of handedness data. The numbers show the mul-
tiplicities of the observations.

1
8
7

1
6
5
hand

1 1
4

1
3

2211
2

1 1 2 2 15534 11
1

15 20 25 30 35 40 45
dnan

Anthony Davison: Bootstrap Methods and their Application, 10

Basic notions

Handedness data

◮ Is there dependence between dnan and hand for these

n = 37 individuals?
◮ Sample product-moment correlation coefficient is θb = 0.509.
◮ Standard confidence interval (based on bivariate normal
population) gives 95% CI (0.221, 0.715).
◮ Data not bivariate normal!
◮ What is the status of the interval? Can we do better?

Anthony Davison: Bootstrap Methods and their Application, 11

Basic notions

Frequentist inference

◮ Estimator θb for unknown parameter θ.

iid
◮ Statistical model: data y1 , . . . , yn ∼ F , unknown
◮ Handedness data
• y = (dnan, hand)
• F puts probability mass on subset of R2
• θb is correlation coefficient
◮ Key issue: what is variability of θb when samples are
repeatedly taken from F ?
◮ Imagine F known — could answer question by
• analytical (mathematical) calculation
• simulation

Anthony Davison: Bootstrap Methods and their Application, 12

Basic notions

Simulation with F known

◮ For r = 1, . . . , R:
iid
• generate random sample y1∗ , . . . , yn∗ ∼ F ;
• compute θbr using y1∗ , . . . , yn∗ ;
◮ Output after R iterations:

θb1∗ , θb2∗ , . . . , θbR

∗

◮ Use θb1∗ , θb2∗ , . . . , θbR b

∗ to estimate sampling distribution of θ

(histogram, density estimate, . . .)

◮ If R → ∞, then get perfect match to theoretical calculation
(if available): Monte Carlo error disappears completely
◮ In practice R is finite, so some error remains

Anthony Davison: Bootstrap Methods and their Application, 13

Basic notions

Handedness data: Fitted bivariate normal model

Figure: Contours of bivariate normal distribution fitted to handedness data;

b1 = 28.5, µ
parameter estimates are µ b2 = 1.7, σ
b1 = 5.4, σ
b2 = 1.5, ρb = 0.509.
The data are also shown.

10

0.020
8 1

0.015
6 1
hand

0.010
4 1 1

2 2211 0.005

1 1 2 2 15534 11

0 0.000
10 15 20 25 30 35 40 45
dnan

Anthony Davison: Bootstrap Methods and their Application, 14

Basic notions

Handedness data: Parametric bootstrap samples

Figure: Left: original data, with jittered vertical values. Centre and
right: two samples generated from the fitted bivariate normal distribu-
tion.
10

10

10
8

8
Correlation 0.509 Correlation 0.753 Correlation 0.533
6

6
hand

hand

hand
4

4
2

2
0

0
0 10 20 30 40 50 0 10 20 30 40 50 0 10 20 30 40 50
dnan dnan dnan

Anthony Davison: Bootstrap Methods and their Application, 15

Basic notions

Handedness data: Correlation coefficient

Figure: Bootstrap distributions with R = 10000. Left: simulation from

fitted bivariate normal distribution. Right: simulation from the data by
bootstrap resampling. The lines show the theoretical probability density
function of the correlation coefficient under sampling from a fitted bivariate
normal distribution.
3.5

3.5
3.0

3.0
1.0 1.5 2.0 2.5

1.0 1.5 2.0 2.5

Probability density

Probability density
0.5

0.5
0.0

0.0

−0.5 0.0 0.5 1.0 −0.5 0.0 0.5 1.0

Correlation coefficient Correlation coefficient

Anthony Davison: Bootstrap Methods and their Application, 16

Basic notions

F unknown
◮ Replace unknown F by estimate Fb obtained
• parametrically — e.g. maximum likelihood or robust fit of
distribution F (y) = F (y; ψ) (normal, exponential, bivariate
normal, . . .)
• nonparametrically — using empirical distribution function
(EDF) of original data y1 , . . . , yn , which puts mass 1/n on
each of the yj
◮ Algorithm: For r = 1, . . . , R:
iid
• generate random sample y1∗ , . . . , yn∗ ∼ Fb ;
• compute θbr using y1 , . . . , yn ;
∗ ∗

◮ Output after R iterations:

θb1∗ , θb2∗ , . . . , θbR

∗

Anthony Davison: Bootstrap Methods and their Application, 17

Basic notions

Nonparametric bootstrap

iid
◮ Bootstrap (re)sample y1∗ , . . . , yn∗ ∼ Fb, where Fb is EDF of
y1 , . . . , yn
• Repetitions will occur!
◮ Compute bootstrap θb∗ using y1∗ , . . . , yn∗ .
◮ For handedness data take n = 37 pairs y ∗ = (dnan, hand)∗
with equal probabilities 1/37 and replacement from original
pairs (dnan, hand)
◮ Repeat this R times, to get θb∗ , . . . , θb∗
1 R
◮ See picture
◮ Results quite different from parametric simulation — why?

Anthony Davison: Bootstrap Methods and their Application, 18

Basic notions

Handedness data: Bootstrap samples

Figure: Left: original data, with jittered vertical values. Centre and
right: two bootstrap samples, with jittered vertical values.
8

8
Correlation 0.509 Correlation 0.733 Correlation 0.491
7

7
6

6
5

5
hand

hand

hand
4

4
3

3
2

2
1

1
10 15 20 25 30 35 40 45 10 15 20 25 30 35 40 45 10 15 20 25 30 35 40 45
dnan dnan dnan

Anthony Davison: Bootstrap Methods and their Application, 19

Basic notions

Using the θb∗

◮ b
Bootstrap replicates θbr∗ used to estimate properties of θ.
◮ Write θ = θ(F ) to emphasize dependence on F
◮ Bias of θb as estimator of θ is
iid
β(F ) = E(θb | y1 , . . . , yn ∼ F ) − θ(F )

estimated by replacing unknown F by known estimate Fb:

iid
β(Fb) = E(θb | y1 , . . . , yn ∼ Fb ) − θ(Fb )

◮ Replace theoretical expectation E() by empirical average:

R
X
β(Fb) ≈ b = θb∗ − θb = R−1 θbr∗ − θb
r=1

Anthony Davison: Bootstrap Methods and their Application, 20

Basic notions

◮ Estimate variance ν(F ) = var(θb | F ) by

1 X  b∗ b∗ 2
R
v= θ −θ
R − 1 r=1 r

◮ Estimate quantiles of θb by taking empirical quantiles of

θb1∗ , . . . , θbR
∗

◮ For handedness data, 10,000 replicates shown earlier give

b = −0.046, v = 0.043 = 0.2052

Anthony Davison: Bootstrap Methods and their Application, 21

Basic notions

Handedness data

Figure: Summaries of the θb∗ . Left: histogram, with vertical line showing θ.
b
Right: normal Q–Q plot of θb∗ .

Histogram of t
2.5
2.0

0.5
1.0 1.5
Density

t*
0.0
0.5

−0.5
0.0

−0.5 0.0 0.5 1.0 −4 −2 0 2 4

t* Quantiles of Standard Normal

Anthony Davison: Bootstrap Methods and their Application, 22

Basic notions

How many bootstraps?

◮ Must estimate moments and quantiles of θb and derived
quantities. Nowadays often feasible to take R ≥ 5000
◮ Need R ≥ 100 to estimate bias, variance, etc.
◮ Need R ≫ 100, prefer R ≥ 1000 to estimate quantiles
needed for 95% confidence intervals
R=199 R=999
4

4
2

2
Z*

Z*
0

0
−2

−2
−4

−4

−4 −2 0 2 4 −4 −2 0 2 4
Theoretical Quantiles Theoretical Quantiles

Anthony Davison: Bootstrap Methods and their Application, 23

Basic notions

Key points
◮ Estimator is algorithm
• applied to original data y1 , . . . , yn gives original θb
• applied to simulated data y1∗ , . . . , yn∗ gives θb∗
• θb can be of (almost) any complexity
• for more sophisticated ideas (later) to work, θb must often be
smooth function of data
◮ Sample is used to estimate F
• Fb ≈ F — heroic assumption
◮ Simulation replaces theoretical calculation
• removes need for mathematical skill
• does not remove need for thought
• check code very carefully — garbage in, garbage out!
◮ Two sources of error
• statistical (Fb 6= F ) — reduce by thought
• simulation (R 6= ∞) — reduce by taking R large (enough)

Anthony Davison: Bootstrap Methods and their Application, 24

Confidence intervals

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 25

Confidence intervals

Normal confidence intervals

.
◮ If θb approximately normal, then θb ∼ N (θ + β, ν), where θb
has bias β = β(F ) and variance ν = ν(F )
◮ If β, ν known, (1 − 2α) confidence interval for θ would be
(D1)
θb − β ± zα ν 1/2 ,
where Φ(zα ) = α.
◮ Replace β, ν by estimates:
. .
β(F ) = β(Fb) = b = θb∗ − θb
. . X
ν(F ) = ν(Fb) = v = (R − 1)−1 (θbr∗ − θb∗ )2 ,
r

giving (1 − 2α) interval θb − b ± zα v 1/2 .

◮ Handedness data: R = 10, 000, b = −0.046, v = 0.2052 ,
α = 0.025, zα = −1.96, so 95% CI is (0.147, 0.963)
Anthony Davison: Bootstrap Methods and their Application, 26
Confidence intervals

Normal confidence intervals

◮ Normal approximation reliable? Transformation needed?
◮ b
Here are plots for ψb = 12 log{(1 + θ)/(1 b
− θ)}:

1.5
Transformed correlation coefficient
1.5

1.0
1.0
Density

0.0 0.5
0.5

−0.5
0.0

−0.5 0.0 0.5 1.0 1.5 −4 −2 0 2 4

Transformed correlation coefficient Quantiles of Standard Normal

Anthony Davison: Bootstrap Methods and their Application, 27

Confidence intervals

Normal confidence intervals

◮ Correlation coefficient: try Fisher’s z transformation:

ψb = ψ(θ)
b = 1
2
b
log{(1 + θ)/(1 b
− θ)}

for which compute

1 X  b∗ b∗ 2
R
X R
bψ = R−1 ψbr∗ − ψ,
b vψ = ψ −ψ ,
r=1
R − 1 r=1 r

◮ (1 − 2α) confidence interval for θ is

n o n o
1/2 1/2
ψ −1 ψb − bψ − z1−α vψ , ψ −1 ψb − bψ − zα vψ

◮ For handedness data, get (0.074, 0.804)

◮ But how do we choose a transformation in general?

Anthony Davison: Bootstrap Methods and their Application, 28

Confidence intervals

Pivots
◮ Hope properties of θb1∗ , . . . , θbR
∗ mimic effect of sampling from

original model.
◮ Amounts to faith in ‘substitution principle’: may replace
unknown F with known Fb — false in general, but often
more nearly true for pivots.
◮ Pivot is combination of data and parameter whose
distribution is independent of underlying model.
iid
◮ Canonical example: Y1 , . . . , Yn ∼ N (µ, σ 2 ). Then
Y −µ
Z= ∼ tn−1 ,
(S 2 /n)1/2
for all µ, σ 2 — so independent of the underlying
distribution, provided this is normal
◮ Exact pivot generally unavailable in nonparametric case.

Anthony Davison: Bootstrap Methods and their Application, 29

Confidence intervals

Studentized statistic
◮ Idea: generalize Student t statistic to bootstrap setting
◮ Requires variance V for θb computed from y1 , . . . , yn
◮ Analogue of Student t statistic:
θb − θ
Z=
V 1/2
◮ If the quantiles zα of Z known, then
!
θb − θ
Pr (zα ≤ Z ≤ z1−α ) = Pr zα ≤ 1/2 ≤ z1−α = 1 − 2α
V
(zα no longer denotes a normal quantile!) implies that
 
Pr θb − V 1/2 z1−α ≤ θ ≤ θb − V 1/2 zα = 1 − 2α

so (1 − 2α) confidence interval is (θb − V 1/2 z1−α , θb − V 1/2 zα )

Anthony Davison: Bootstrap Methods and their Application, 30

Confidence intervals

◮ Bootstrap sample gives (θb∗ , V ∗ ) and hence

θb∗ − θb
Z∗ =
V ∗1/2
◮ b V ):
R bootstrap copies of (θ,
(θb1∗ , V1∗ ), (θb2∗ , V2∗ ), ..., (θbR
∗
, VR∗ )
and corresponding
θb1∗ − θb θb2∗ − θb θbR
∗ −θ b
z1∗ = ∗1/2
, z2∗ = ∗1/2
, ..., ∗
zR = ∗1/2
.
V1 V2 VR
◮ Use ∗ ∗
z1 , . . . , zR
to estimate distribution of Z — for example,
∗ < · · · < z∗
order statistics z(1) (R) used to estimate quantiles
◮ Get (1 − 2α) confidence interval
θb − V 1/2 z((1−α)(R+1))
∗
, θb − V 1/2 z(α(R+1))
∗

Anthony Davison: Bootstrap Methods and their Application, 31

Confidence intervals

Why Studentize?

D
◮ Studentize, so Z −→ N (0, 1) as n → ∞. Edgeworth series:

Pr(Z ≤ z | F ) = Φ(z) + n−1/2 a(z)φ(z) + O(n−1 );

a(·) even quadratic polynomial.

◮ Corresponding expansion for Z ∗ is

Pr(Z ∗ ≤ z | Fb) = Φ(z) + n−1/2 b

a(z)φ(z) + Op (n−1 ).

a(z) = a(z) + Op (n−1/2 ), so

Typically b

Pr(Z ∗ ≤ z | Fb) − Pr(Z ≤ z | F ) = Op (n−1 ).

Anthony Davison: Bootstrap Methods and their Application, 32

Confidence intervals

D
◮ If don’t studentize, Z = (θb − θ) −→ N (0, ν). Then
 z   z   z 
Pr(Z ≤ z | F ) = Φ 1/2 +n−1/2 a′ 1/2 φ 1/2 +O(n−1 )
ν ν ν
and
 z   z   z 
Pr(Z ∗ ≤ z | Fb) = Φ +n −1/2 ′
a
b φ 1/2 +Op (n−1 ).
νb1/2 νb1/2 νb

Typically νb = ν + Op (n−1/2 ), giving

Pr(Z ∗ ≤ z | Fb ) − Pr(Z ≤ z | F ) = Op (n−1/2 ).

◮ Thus use of Studentized Z reduces error from Op (n−1/2 ) to

Op (n−1 ) — better than using large-sample asymptotics, for
which error is usually Op (n−1/2 ).

Anthony Davison: Bootstrap Methods and their Application, 33

Confidence intervals

Other confidence intervals

◮ Problem for studentized intervals: must obtain V , intervals
not scale-invariant
◮ Simpler approaches:
• Basic bootstrap interval: treat θb − θ as pivot, get
θb − (θb((R+1)(1−α))
∗ b
− θ), θb − (θb((R+1)α)
∗ b
− θ).

• Percentile interval: use empirical quantiles of θb1∗ , . . . , θbR

∗
:

θb((R+1)α)
∗
, θb((R+1)(1−α))
∗
.
◮ Improved percentile intervals (BCa , ABC, . . .)
• Replace percentile interval with
θb((R+1)α
∗
′) , θb((R+1)(1−α
∗
′′ )) ,

where α′ , α′′ chosen to improve properties.

• Scale-invariant.

Anthony Davison: Bootstrap Methods and their Application, 34

Confidence intervals

Handedness data

◮ 95% confidence intervals for correlation coefficient θ,

R = 10, 000:

Normal 0.147 0.963

Percentile −0.047 0.758
Basic 0.262 1.043
′ ′′
BCa (α = 0.0485, α = 0.0085) 0.053 0.792
Student 0.030 1.206

Basic (transformed) 0.131 0.824

Student (transformed) −0.277 0.868

◮ Transformation is essential here!

Anthony Davison: Bootstrap Methods and their Application, 35

Confidence intervals

General comparison

◮ Bootstrap confidence intervals usually too short — leads to

under-coverage
◮ Normal and basic intervals depend on scale.
◮ Percentile interval often too short but is scale-invariant.
◮ Studentized intervals give best coverage overall, but
• depend on scale, can be sensitive to V
• length can be very variable
• best on transformed scale, where V is approximately
constant
◮ Improved percentile intervals have same error in principle
as studentized intervals, but often shorter — so lower
coverage

Anthony Davison: Bootstrap Methods and their Application, 36

Confidence intervals

Caution
◮ Edgeworth theory OK for smooth statistics — beware
rough statistics: must check output.
◮ Bootstrap of median theoretically OK, but very sensitive to
sample values in practice.
◮ Role for smoothing?

.. .
10

.
..
....
........
.........
5

.............
T*-t for medians

.
.............................
..........
0

................
............
..............
........
-5

.
.........
. ......
-10

. .

-2 0 2
Quantiles of Standard Normal

Anthony Davison: Bootstrap Methods and their Application, 37

Confidence intervals

Key points

◮ Numerous procedures available for ‘automatic’ construction

of confidence intervals
◮ Computer does the work
◮ Need R ≥ 1000 in most cases
◮ Generally such intervals are a bit too short
◮ Must examine output to check if assumptions (e.g.
smoothness of statistic) satisfied
◮ May need variance estimate V — see later

Anthony Davison: Bootstrap Methods and their Application, 38

Several samples

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 39

Several samples

Gravity data

Table: Measurements of the acceleration due to gravity, g, given as

deviations from 980,000 ×10−3 cms−2 , in units of cms−2 × 10−3 .

Series
1 2 3 4 5 6 7 8
76 87 105 95 76 78 82 84
82 95 83 90 76 78 79 86
83 98 76 76 78 78 81 85
54 100 75 76 79 86 79 82
35 109 51 87 72 87 77 77
46 109 76 79 68 81 79 76
87 100 93 77 75 73 79 77
68 81 75 71 78 67 78 80
75 62 75 79 83
68 82 82 81
67 83 76 78
73 78
64 78

Anthony Davison: Bootstrap Methods and their Application, 40

Several samples

Gravity data

Figure: Gravity series boxplots, showing a reduction in variance, a shift

in location, and possible outliers.
100
80
g
60
40

1 2 3 4 5 6 7 8

series

Anthony Davison: Bootstrap Methods and their Application, 41

Several samples

Gravity data
◮ Eight series of measurements of gravitational acceleration g
made May 1934 – July 1935 in Washington DC
◮ Data are deviations from 9.8 m/s2 in units of 10−3 cm/s2
◮ Goal: Estimate g and provide confidence interval
◮ Weighted combination of series averages and its variance
estimate
P8 8
!−1
y × n i /s 2 X
i=1 i
θb = P 8 2
i
, V = ni /s2i ,
i=1 n i /s i i=1

giving
θb = 78.54, V = 0.592
and 95% confidence interval of θb ± 1.96V 1/2 = (77.5, 79.8)

Anthony Davison: Bootstrap Methods and their Application, 42

Several samples

Gravity data: Bootstrap

◮ Apply stratified (re)sampling to series, taking each series as
a separate stratum. Compute θb∗ , V ∗ for simulated data
◮ Confidence interval based on
θb∗ − θb
Z ∗ = ∗1/2 ,
V
whose distribution is approximated by simulations
θb1∗ − θb θbR
∗ −θ b
z1∗ = 1/2
, . . . , zR
∗
= 1/2
,
V1 VR
giving
(θb − V 1/2 z((R+1)(1−α))
∗
, θb − V 1/2 z((R+1)α)
∗
)
◮ For 95% limits set α = 0.025, so with R = 999 use
∗ , z∗
z(25) (975) , giving interval (77.1, 80.3).

Anthony Davison: Bootstrap Methods and their Application, 43

Several samples

Figure: Summary plots for 999 nonparametric bootstrap simulations.

Top: normal probability plots of t∗ and z ∗ = (t∗ − t)/v ∗1/2 . Line on
the top left has intercept t and slope v 1/2 , line on the top right has
intercept zero and unit slope. Bottom: the smallest t∗r also has the
smallest v ∗ , leading to an outlying value of z ∗ .

. .
.. .. .

5
81 ....
.. ............
.... ...........
.... ...
......
........ .....................
.........
..... .............
..........
80

0
....
.......
....
..
....... ...
... .............
....
.
.... .....
......... ........
............

-5
79

.
........

z*
t*

...........
.........
......

-10
....
78

..
...........
.
.
....
.....

-15
....
77

...
.. .

-2 0 2 -2 0 2
Quantiles of Standard Normal Quantiles of Standard Normal

... ..
0.1 0.2 0.3 0.4 0.5 0.6 0.7

0.1 0.2 0.3 0.4 0.5 0.6 0.7

. .. .. . ...
.. ................... . .....................
. . .................................................................. . . .. .. ................... .... ..
. .... . .. .. ... ... .. . ........................................ .
.. ....................................................................................... . . ..........................
....................
. . ................................................................. .............. ....... .
... . . . . . ......................................................... ..
. . . . .. . ................................................................ ....
. . .. .
. . . .............................................. . .
. . . . ..
sqrt(v*)

sqrt(v*)

.. ...... ..... ..... ........... . . . .. ...................................... . .

. ..... . .. ........................ . . . . ................................. ....
. . ............................. ........ . . . . . . . ...
... . . ........... ..... .................. .. .. .. . ... .. .................................... .
. . . .. . . ..... ...... .... . .. . . . ...... ..... .... .
. . . . . ..
. . . .. . . . . . . ... ... .....
.
. .
. . . .
. .
. .

77 78 79 80 81 -15 -10 -5 0 5
t* z*

Anthony Davison: Bootstrap Methods and their Application, 44

Several samples

Key points

◮ For several independent samples, implement bootstrap by

stratified sampling independently from each
◮ Same basic ideas apply for confidence intervals

Anthony Davison: Bootstrap Methods and their Application, 45

Variance estimation

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 46

Variance estimation

Variance estimation

◮ Variance estimate V needed for certain types of confidence

interval (esp. studentized bootstrap)
◮ Ways to compute this:
• double bootstrap
• delta method
• nonparametric delta method
• jackknife

Anthony Davison: Bootstrap Methods and their Application, 47

Variance estimation

Double bootstrap

◮ Bootstrap sample y1∗ , . . . , yn∗ and corresponding estimate θb∗

◮ Take Q second-level bootstrap samples y1∗∗ , . . . , yn∗∗ from
y1∗ , . . . , yn∗ , giving corresponding bootstrap estimates
θb1∗∗ , . . . , θbQ
∗∗ ,

◮ Compute variance estimate V as sample variance of

θb1∗∗ , . . . , θbQ
∗∗

◮ Requires total R(Q + 1) resamples, so could be expensive

.
◮ Often reasonable to take Q = 50 for variance estimation, so
need O(50 × 1000) resamples — nowadays not infeasible

Anthony Davison: Bootstrap Methods and their Application, 48

Variance estimation

Delta method
◮ Computation of variance formulae for functions of averages
and other estimators
◮ b estimates ψ = g(θ), and θb ∼. N (θ, σ 2 /n)
Suppose ψb = g(θ)
◮ Then provided g′ (θ) 6= 0, have (D2)
b = g(θ) + O(n−1 )
E(ψ)
b = σ 2 g′ (θ)2 /n + O(n−3/2 )
var(ψ)
b = . 2 ′ b2
◮ Then var(ψ) σ
b g (θ) /n = V
◮ Example (D3): θb = Y , ψb = log θb
.
b =
◮ Variance stabilisation (D4): if var(θ) S(θ)2 /n, find
.
b =constant
transformation h such that var{h(θ)}
◮ Extends to multivariate estimators, and to
ψb = g(θb1 , . . . , θbd )

Anthony Davison: Bootstrap Methods and their Application, 49

Variance estimation

Anthony Davison: Bootstrap Methods and their Application, 50

Variance estimation

Nonparametric delta method

◮ Write parameter θ = t(F ) as functional of distribution F
◮ General approximation:
n
. 1 X
V = VL = 2 L(Yj ; F )2 .
n
j=1

◮ L(y; F ) is influence function value for θ for observation at y

when distribution is F :
t {(1 − ε)F + εHy } − t(F )
L(y; F ) = lim ,
ε→0 ε
where Hy puts unit mass at y. Close link to robustness.
◮ Empirical versions of L(y; F ) and VL are
X
lj = L(yj ; Fb), vL = n−2 lj2 ,
usually obtained by analytic/numerical differentation.
Anthony Davison: Bootstrap Methods and their Application, 51
Variance estimation

Computation of lj
◮ Write θb in weighted form, differentiate with respect to ε
◮ Sample average:
1X X 

θb = y = yj = wj yj 
n wj ≡1/n

Change weights:
wj 7→ ε + (1 − ε) n1 , wi 7→ (1 − ε) n1 , i 6= j
so (D5)
y 7→ y ε = εyj + (1 − ε)y = ε(yj − y) + y,
P
giving lj = yj − y and vL = n12 (yj − y)2 = n−1 n n s
−1 2

◮ Interpretation: lj is standardized change in y when increase

mass on yj

Anthony Davison: Bootstrap Methods and their Application, 52

Variance estimation

Nonparametric delta method: Ratio

◮ Population F (u, x) with y = (u, x) and
Z Z
θ = t(F ) = x dF (u, x)/ u dF (u, x),

sample version is
Z Z
θb = t(Fb) = x dFb(u, x)/ u dFb(u, x) = x/u

◮ Then using chain rule of differentiation (D6),

b j )/u,
lj = (xj − θu
giving
!2
1 X b j
xj − θu
vL = 2
n u

Anthony Davison: Bootstrap Methods and their Application, 53

Variance estimation

Handedness data: Correlation coefficient

◮ Correlation coefficient
P may be written as a function of
averages xu = n −1 xj uj etc.:

xu − x u
θb = n o1/2 ,
(x2 − x2 )(u2 − u2 )

from which empirical influence values lj can be derived

◮ In this example (and for others involving only averages),
nonparametric delta method is equivalent to delta method
◮ Get
vL = 0.029
for comparison with v = 0.043 obtained by bootstrapping.
◮ b — as here!
vL typically underestimates var(θ)

Anthony Davison: Bootstrap Methods and their Application, 54

Variance estimation

Delta methods: Comments

◮ Can be applied to many complex statistics
◮ Delta method variances often underestimate true variances:
b
vL < var(θ)

◮ Can be applied automatically (numerical differentation) if

algorithm for θb written in weighted form, e.g.
X
xw = wj xj , wj ≡ 1/n for x

and vary weights successively for j = 1, . . . , n, setting

X
wj = wi + ε, i 6= j, wi = 1

for ε = 1/(100n) and using the definition as derivative

Anthony Davison: Bootstrap Methods and their Application, 55

Variance estimation

Jackknife
◮ Approximation to empirical influence values given by

lj ≈ ljack,j = (n − 1)(θb − θb−j ),

where θb−j is value of θb computed from sample

y1 , . . . , yj−1 , yj+1 , . . . , yn

◮ Jackknife bias and variance estimates are

1X 1 X 
2
bjack = − ljack,j , vjack = ljack,j − nb2jack
n n(n − 1)

◮ Requires n + 1 calculations of θb
◮ b with
Corresponds to numerical differentiation of θ,
ε = −1/(n − 1)

Anthony Davison: Bootstrap Methods and their Application, 56

Variance estimation

Key points

◮ Several methods available for estimation of variances

◮ Needed for some types of confidence interval
◮ Most general method is double bootstrap: can be expensive
◮ Delta methods rely on linear expansion, can be applied
numerically or analytically
◮ Jackknife gives approximation to delta method, can fail for
rough statistics

Anthony Davison: Bootstrap Methods and their Application, 57

Tests

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 58

Tests

Ingredients

◮ Ingredients for testing problems:

• data y1 , . . . , yn ;
• model M0 to be tested;
• test statistic t = t(y1 , . . . , yn ), with large values giving
evidence against M0 , and observed value tobs
◮ P-value, pobs = Pr(T ≥ tobs | M0 ) measures evidence
against M0 — small pobs indicates evidence against M0 .
◮ Difficulties:
• pobs may depend upon ‘nuisance’ parameters, those of M0 ;
• pobs often hard to calculate.

Anthony Davison: Bootstrap Methods and their Application, 59

Tests

Examples
◮ Balsam-fir seedlings in 5 × 5 quadrats — Poisson sample?
0 1 2 3 4 3 4 2 2 1
0 2 0 2 4 2 3 3 4 2
1 1 1 1 4 1 5 2 2 3
4 1 2 5 2 0 3 2 1 1
3 1 4 3 1 0 0 2 7 0
◮ Two-way layout: row-column independence?
1 2 2 1 1 0 1
2 0 0 2 3 0 0
0 1 1 1 2 7 3
1 1 2 0 0 0 1
0 1 1 1 1 0 0

Anthony Davison: Bootstrap Methods and their Application, 60

Tests

Estimation of pobs
◮ Estimate pobs by simulation from fitted null hypothesis
model Mc0 .
◮ Algorithm: for r = 1, . . . , R:
c0 ;
• simulate data set y1∗ , . . . , yn∗ from M
• calculate test statistic t∗r from y1∗ , . . . , yn∗ .
◮ Calculate simulation estimate
1 + #{t∗r ≥ tobs }
pb =
1+R
of
c0 ).
pbobs = Pr(T ≥ tobs | M
◮ Simulation and statstical errors:

pb ≈ pbobs ≈ pobs

Anthony Davison: Bootstrap Methods and their Application, 61

Tests

Handedness data: Test of independence

◮ Are dnan and hand positively associated?
◮ Take T = θb (correlation coefficient), with tobs = 0.509; this
is large in case of positive association (one-sided test)
◮ Null hypothesis of independence: F (u, x) = F1 (u)F2 (x)
◮ Take bootstrap samples independently from
Fb1 ≡ (dnan1 , . . . , dnann ) and from Fb2 ≡ (hand1 , . . . , handn ),
then put them together to get bootstrap data
(dnan∗1 , hand∗1 ), . . . , (dnan∗n , hand∗n ).
◮ With R = 9, 999 get 18 values of θb∗ ≥ θ, b so
1 + 18
pb = = 0.0019 :
1 + 9999
hand and dnan seem to be positively associated
◮ To test positive or negative association (two-sided test),
b gives pb = 0.004.
take T = |θ|:
Anthony Davison: Bootstrap Methods and their Application, 62
Tests

c0
Handedness data: Bootstrap from M

1
8

2.5
7

1.0 1.5 2.0

Probability density
2 1 1
65
hand
4

1
3

0.5
2 2
2

2 1 1102 3 4 3 1 0.0
1

15 20 25 30 35 40 45 −0.6 −0.2 0.2 0.6

dnan Test statistic

Anthony Davison: Bootstrap Methods and their Application, 63

Tests

Choice of R

◮ Take R big enough to get small standard error for pb,

typically ≥ 100, using binomial calculation:
 
1 + #{t∗r ≥ tobs }
var(b
p) = var
1+R
. 1 pobs (1 − pobs )
= 2
Rpobs (1 − pobs ) =
R R
.
so if pobs = 0.05 need R ≥ 1900 for 10% relative error (D7)
.
◮ Can choose R sequentially: e.g. if pb = 0.06 and R = 99, can
augment R enough to diminish standard error.
◮ Taking R too small lowers power of test.

Anthony Davison: Bootstrap Methods and their Application, 64

Tests

Duality with confidence interval

◮ Often unclear how to impose null hypothesis on sampling

scheme
◮ General approach based on duality between confidence
interval I1−α = (θα , ∞) and test of null hypothesis θ = θ0
◮ Reject null hypothesis at level α in favour of alternative
θ > θ0 , if θ0 < θα
◮ Handedness data: θ0 = 0 6∈ I0.95 , but θ0 = 0 ∈ I0.99 , so
estimated significance level 0.01 < pb < 0.05: weaker
evidence than before
◮ Extends to tests of θ = θ0 against other alternatives:
6 I 1−α = (−∞, θα ), have evidence that θ < θ0
• if θ0 ∈
• if θ0 ∈6 I1−2α = (θα , θα ), have evidence that θ 6= θ0

Anthony Davison: Bootstrap Methods and their Application, 65

Tests

Pivot tests
◮ Equivalent to use of confidence intervals
◮ Idea: use (approximate) pivot such as Z = (θb − θ)/V 1/2 as
statistic to test θ = θ0
◮ Observed value of pivot is zobs = (θb − θ0 )/V 1/2
◮ Significance level is
!
θb − θ
Pr ≥ zobs | M0 = Pr(Z ≥ zobs | M0 )
V 1/2
= Pr(Z ≥ zobs | F )
.
= Pr(Z ≥ zobs | Fb)
◮ Compare observed zobs with simulated distribution of
Z ∗ = (θb∗ − θ)/V
b ∗1/2 , without needing to construct null

hypothesis model M c0
◮ Use of (approximate) pivot is essential for success
Anthony Davison: Bootstrap Methods and their Application, 66
Tests

Example: Handedness data

◮ Test zero correlation (θ0 = 0), not independence; θb = 0.509,
V = 0.1702 :
θb − θ0 0.509 − 0
zobs = 1/2 = = 2.99
V 0.170
◮ Observed significance level is
1 + #{zr∗ ≥ zobs } 1 + 215
pb = = = 0.0216
1+R 1 + 99990.3
Probability density
0.1 0.20.0

−6 −4 −2 0 2 4 6
Test statistic

Anthony Davison: Bootstrap Methods and their Application, 67

Tests

Exact tests

◮ Problem: bootstrap estimate is

c0 ) 6= Pr(T ≥ t | M0 ) = pobs ,
pbobs = Pr(T ≥ tobs | M

so estimate the wrong thing

◮ In some cases can eliminate parameters from null
hypothesis distribution by conditioning on sufficient
statistic
◮ Then simulate from conditional distribution
◮ More generally, can use Metropolis–Hastings algorithm to
simulate from conditional distribution (below)

Anthony Davison: Bootstrap Methods and their Application, 68

Tests

Example: Fir data

iid
◮ Data Y1 , . . . , Yn ∼ Pois(λ), with λ unknown
◮ Poisson model has E(Y ) = var(Y ) = λ: base test of
overdispersion on
X .
T = (Yj − Y )2 /Y ∼ χ2n−1 ;
observed value is tobs = 55.15
◮ Unconditional significance level:
c0 , λ)
Pr(T ≥ tobs | M
P
◮ Condition on value w of sufficient statistic W = Yj :
c0 , W = w),
pobs = Pr(T ≥ tobs | M
independent of λ, owing to sufficiency of W
◮ Exact test: simulate from P multinomial distribution of
Y1 , . . . , Yn given W = Yj = 107.
Anthony Davison: Bootstrap Methods and their Application, 69
Tests

Example: Fir data

Figure: Simulation results for dispersion test. Left panel: R = 999 val-
ues of the dispersion statistic t∗ obtained under multinomial sampling:
the data value is tobs = 55.15 and pb = 0.25. Right panel: chi-squared
plot of ordered values of t∗ , dotted line shows χ249 approximation to
null conditional distribution.

.
0.04

80
...

dispersion statistic t*
.
....
0.03

......
..........
.
.....
......
60
......
.....
0.02

..
.........
.
......
......
.....
40

......
0.01

.
........
.
..
.....
...
.
..
0.0

20

20 40 60 80 30 40 50 60 70 80
t* chi-squared quantiles

Anthony Davison: Bootstrap Methods and their Application, 70

Tests

Handedness data: Permutation test

◮ Are dnan and hand related?
◮ Take T = θb (correlation coefficient) again
◮ Impose null hypothesis of independence:
F (u, x) = F1 (u)F2 (x), but condition so that marginal
distributions Fb1 and Fb2 are held fixed under resampling
plan — permutation test
◮ Take resamples of form
(dnan1 , hand1∗ ), . . . , (dnann , handn∗ )
where (1∗ , . . . , n∗ ) is random permutation of (1, . . . , n)
◮ Doing this with R = 9, 999 gives one- and two-sided
significance probabilities of 0.002, 0.003
◮ Typically values of pb very similar to those for
corresponding bootstrap test

Anthony Davison: Bootstrap Methods and their Application, 71

Tests

Handedness data: Permutation resample

3.0
8

1.0 1.5 2.0 2.5

7

Probability density
65
hand
4 3

0.5
2

0.0
1

15 20 25 30 35 40 45 −0.6 −0.2 0.2 0.6

dnan Test statistic

Anthony Davison: Bootstrap Methods and their Application, 72

Tests

Contingency table
1 2 2 1 1 0 1
2 0 0 2 3 0 0
0 1 1 1 2 7 3
1 1 2 0 0 0 1
0 1 1 1 1 0 0
◮ Are row and column classifications independent:
Pr(row i, column j) = Pr(row i) × Pr(column j)?
◮ Standard test statistic for independence is
X (yij − ybij )2 yi· y·j
T = , ybij =
ybij y··
i,j
.
◮ Get Pr(χ224 ≥ 38.53) = 0.048, but is T ∼ χ224 ?

Anthony Davison: Bootstrap Methods and their Application, 73

Tests

Exact tests: Contingency table

◮ For exact test, need to simulate distribution of T
conditional on sufficient statistics — row and column totals
◮ Algorithm (D8) for conditional simulation:
1. choose two rows j1 < j2 and two columns k1 < k2 at
random
2. generate new values from hypergeometric distribution
of yj1 k1 conditional on margins of 2 × 2 table

y j 1 k1 y j 1 k2
y j 2 k1 y j 2 k2

3. compute test statistic T ∗ every I = 100 iterations, say

◮ Compare observed value tobs = 38.53 with simulated T ∗ —
.
get pb = 0.08

Anthony Davison: Bootstrap Methods and their Application, 74

Tests

Key points

◮ Tests can be performed using resampling/simulation

◮ Must take account of null hypothesis, by
• modifying sampling scheme to satisfy null hypothesis
• inverting confidence interval (pivot test)
◮ Can use Monte Carlo simulation to get approximations to
exact tests — simulate from null distribution of data,
conditional on observed value of sufficient statistic
◮ Sometimes obtain permutation tests — very similar to
bootstrap tests

Anthony Davison: Bootstrap Methods and their Application, 75

Regression

 Motivation

 Basic notions

 Confidence intervals

 Several samples

 Variance estimation

 Tests

 Regression

Anthony Davison: Bootstrap Methods and their Application, 76

Regression

Linear regression
◮ Independent data (x1 , y1 ), . . ., (xn , yn ) with

yj = xTj β + εj , εj ∼ (0, σ 2 )

◮ b leverages hj , residuals
Least squares estimates β,

yj − xTj βb .
ej = ∼ (0, σ 2 )
(1 − hj )1/2
◮ Design matrix X is experimental ancillary — should be
held fixed if possible, as
b = σ 2 (X T X)−1
var(β)

if model y = Xβ + ε correct

Anthony Davison: Bootstrap Methods and their Application, 77

Regression

Linear regression: Resampling schemes

◮ Two main resampling schemes
◮ Model-based resampling:

yj∗ = xTj βb + ε∗j , ε∗j ∼ EDF(e1 − e, . . . , en − e)

• Fixes design but not robust to model failure

• Assumes εj sampled from population
◮ Case resampling:

(xj , yj )∗ ∼ EDF{(x1 , y1 ), . . . , (xn , yn )}

• Varying design X but robust

• Assumes (xj , yj ) sampled from population
• Usually design variation no problem; can prove awkward in
designed experiments and when design sensitive.

Anthony Davison: Bootstrap Methods and their Application, 78

Regression

Cement data

Table: Cement data: y is the heat (calories per gram of cement) evolved
while samples of cement set. The covariates are percentages by weight
of four constituents, tricalciumaluminate x1 , tricalcium silicate x2 , tet-
racalcium alumino ferrite x3 and dicalcium silicate x4 .

x1 x2 x3 x4 y
1 7 26 6 60 78.5
2 1 29 15 52 74.3
3 11 56 8 20 104.3
4 11 31 8 47 87.6
5 7 52 6 33 95.9
6 11 55 9 22 109.2
7 3 71 17 6 102.7
8 1 31 22 44 72.5
9 2 54 18 22 93.1
10 21 47 4 26 115.9
11 1 40 23 34 83.8
12 11 66 9 12 113.3
13 10 68 8 12 109.4

Anthony Davison: Bootstrap Methods and their Application, 79

Regression

Cement data

◮ Fit linear model

y = β0 + β1 x1 + β2 x2 + β3 x3 + β4 x4 + ε

and apply case resampling

.
◮ Covariates compositional: x1 + · · · + x4 = 100% so X
almost collinear — smallest eigenvalue of X T X is
l5 = 0.0012
◮ Plot of βb1∗ against smallest eigenvalue of X ∗T X ∗ reveals
that var∗ (βb∗ ) strongly variable
1
◮ Relevant subset for case resampling — post-stratification of
output based on l5∗ ?

Anthony Davison: Bootstrap Methods and their Application, 80

Regression

Cement data

10

10
.
. .
. ..
5

5
. . . . ..... .
. . .
. . . .. ..... .................................................................... . . . .
...... .. ........ ..
beta1hat*

beta2hat*
. . .... . .
.. .. .............................................................. . .. .................................. .............................. .
........
. . .. ... .. ................................................................. . . ................
.. . .. ... ... ................................................................................
.. . .... ................ .. . . ... ..................... . ... .
0

0
. . ... ..... . . . . .......... ............... ..
.. .. ... ..
. .. . .
-5

-5
. .
.
.
. .
-10

-10
1 5 10 50 500 1 5 10 50 500
smallest eigenvalue smallest eigenvalue

Anthony Davison: Bootstrap Methods and their Application, 81

Regression

Cement data

Table: Standard errors of linear regression coefficients for cement data.

Theoretical and error resampling assume homoscedasticity. Resampling
results use R = 999 samples, but last two rows are based only on those
samples with the middle 500 and the largest 800 values of ℓ∗1 .

βb0 βb1 βb2

Normal theory 70.1 0.74 0.72
Model-based resampling, R = 999 66.3 0.70 0.69
Case resampling, all R = 999 108.5 1.13 1.12
Case resampling, largest 800 67.3 0.77 0.69

Anthony Davison: Bootstrap Methods and their Application, 82

Regression

Survival data
dose x 117.5 235.0 470.0 705.0 940.0 1410
survival % y 44.000 16.000 4.000 0.500 0.110 0.700
55.000 13.000 1.960 0.320 0.015 0.006
6.120 0.019

◮ Data on survival % for rats at different doses

◮ Linear model:
log(survival) = β0 + β1 dose

• ••

4
50

••
•

2
•
40

•
log survival %
survival %

•
30

0 • •
•
20

-2

• •
•
10

-4

•
• ••
• • • • •
0

200 600 1000 1400 200 600 1000 1400

dose dose

Anthony Davison: Bootstrap Methods and their Application, 83

Regression

Survival data
◮ Case resampling
◮ Replication of outlier: none (0), once (1), two or more (•).
◮ Model-based sampling including residual would lead to
change in intercept but not slope.

0.0

• •
• •• •
-0.004

•
estimated slope

• •
1 • • •• •
• ••
1 ••• • 1 •••••• ••• •••
111 1 11 1 ••• • •••••••• •••••
1111 1
1
11
11111111 11
1
11111
1
1 1 11
11 1111111111 1 1111
•
0 1
11 11
1 1 11 1 1 11 1 1 1
0 0 01000 00 0
-0.008

0 0 0 0
0 000 0 00000 0 0 0 0 0
0 0 00 0 000
0
0 0000
0 00 000 00
0000
00 0
-0.012

0.5 1.0 1.5 2.0 2.5 3.0

sum of squares

Anthony Davison: Bootstrap Methods and their Application, 84

Regression

Generalized linear model

◮ Response may be binomial, Poisson, gamma, normal, . . .
yj ∼ mean µj , variance φV (µj ),
where g(µj ) = xTj β is linear predictor; g(·) is link function.
◮ b fitted values µ
MLE β, bj , Pearson residuals
yj − µ
bj .
rP j = 1/2
∼ (0, φ).
{V (b
µj )(1 − hj )}
◮ Bootstrapped responses
yj∗ = µ µj )1/2 ε∗j
bj + V (b
where ε∗j ∼ EDF(rP 1 − rP , . . . , rP n − rP ). However
• possible that yj∗ 6∈ {0, 1, 2, . . . , }
• rP j not exchangeable, so may need stratified resampling

Anthony Davison: Bootstrap Methods and their Application, 85

Regression

AIDS data
◮ Log-linear model: number of reports in row j and column k
follows Poisson distribution with mean
µjk = exp(αj + βk )
◮ Log link function
g(µjk ) = log µjk = αj + βk
and variance function
var(Yjk ) = φ × V (µjk ) = 1 × µjk
◮ Pearson residuals:
Yjk − µbjk
rjk =
µjk (1 − hjk )}1/2
{b
◮ Model-based simulation:
∗ 1/2
Yjk =µ bjk ε∗jk
bjk + µ
Anthony Davison: Bootstrap Methods and their Application, 86
Regression

Diagnosis Reporting-delay interval (quarters): Total

period reports
to end
Year Quarter 0† 1 2 3 4 5 6 ··· ≥14 of 1992
1988 1 31 80 16 9 3 2 8 ··· 6 174
2 26 99 27 9 8 11 3 ··· 3 211
3 31 95 35 13 18 4 6 ··· 3 224
4 36 77 20 26 11 3 8 ··· 2 205
1989 1 32 92 32 10 12 19 12 ··· 2 224
2 15 92 14 27 22 21 12 ··· 1 219
3 34 104 29 31 18 8 6 ··· 253
4 38 101 34 18 9 15 6 ··· 233
1990 1 31 124 47 24 11 15 8 ··· 281
2 32 132 36 10 9 7 6 ··· 245
3 49 107 51 17 15 8 9 ··· 260
4 44 153 41 16 11 6 5 ··· 285
1991 1 41 137 29 33 7 11 6 ··· 271
2 56 124 39 14 12 7 10 263
3 53 175 35 17 13 11 306
4 63 135 24 23 12 258
1992 1 71 161 48 25 310
2 95 178 39 318
3 76 181 273
4 67 133

Anthony Davison: Bootstrap Methods and their Application, 87

Regression

AIDS data
◮ Poisson two-way model deviance 716.5 on 413 df —
indicates strong overdispersion: φ > 1, so Poisson model
implausible
◮ Residuals highly inhomogeneous — exchangeability
doubtful
500

6
400

4
2
Diagnoses

+ ++
300

+ + +
+ ++ ++ + rP
0
+ + +
+++
200

++ -2
++
++++ +
100

+
-4

++
+++
+++++
-6
0

1984 1986 1988 1990 1992 0 1 2 3 4 5

skewness

Anthony Davison: Bootstrap Methods and their Application, 88

Regression

AIDS data: Prediction intervals

◮ To estimate prediction error:

∗
• simulate complete table yjk ;
∗
• estimate parameters from incomplete yjk
• get estimated row totals and ‘truth’
∗ X b∗ X
b∗+,j = eαbj
µ eβk , y+,j
∗
= ∗
yjk .
k unobs k unobs

• Prediction error ∗
y+,j b∗+,j
−µ
∗1/2
µ
b+,j
studentized so more nearly pivotal.
◮ Form prediction intervals from R replicates.

Anthony Davison: Bootstrap Methods and their Application, 89

Regression

AIDS data: Resampling plans

◮ Resampling schemes:
• parametric simulation, fitted Poisson model
• parametric simulation, fitted negative binomial model
• nonparametric resampling of rP
• stratified nonparametric resampling of rP
◮ Stratification based on skewness of residuals, equivalent to
stratifying original data by values of fitted means
◮ Take strata for which

µ
bjk < 1, 1≤µ
bjk < 2, µ
bjk ≥ 2

Anthony Davison: Bootstrap Methods and their Application, 90

Regression

AIDS data: Results

◮ Deviance/df ratios for the sampling schemes, R = 999.
◮ Poisson variation inadequate.
◮ 95% prediction limits.
poisson negative binomial
6

6
5

600
4

4
3

3
2

500
1

1
0

Diagnoses
0.0 0.5 1.0 1.5 2.0 2.5 0.0 0.5 1.0 1.5 2.0 2.5

400
deviance/df deviance/df

nonparametric stratified nonparametric

+ ++
6

300
+ +
5

+ ++ + +
+
4

+
+ ++ +
3

200

+
2

2
1

1989 1990 1991 1992

0

0.0 0.5 1.0 1.5 2.0 2.5 0.0 0.5 1.0 1.5 2.0 2.5
deviance/df deviance/df

Anthony Davison: Bootstrap Methods and their Application, 91

Regression

AIDS data: Semiparametric model

◮ More realistic: generalized additive model
µjk = exp {α(j) + βk } ,
where α(j) is locally-fitted smooth.
◮ Same resampling plans as before
◮ 95% intervals now generally narrower and shifted upwards
100 200 300 400 500 600

600
500
Diagnoses

Diagnoses
+ ++ 400
+ +
+ ++ ++ + +
+++++ +
+++ + ++
300

+++++ + + + +
+ + + ++ +
++ +
++++ + ++ +
++++
200

+
0

1984 1986 1988 1990 1992 1989 1990 1991 1992

Anthony Davison: Bootstrap Methods and their Application, 92

Regression

Key points

◮ Key assumption: independence of cases

◮ Two main resampling schemes for regression settings:
• Model-based
• Case resampling
◮ Intermediate schemes possible
◮ Can help to reduce dependence on assumptions needed for
regression model
◮ These two basic approaches also used for more complex
settings (time series, . . .), where data are dependent

Anthony Davison: Bootstrap Methods and their Application, 93

Regression

Summary

◮ Bootstrap: simulation methods for frequentist inference.

◮ Useful when
• standard assumptions invalid (n small, data not normal,
. . .);
• standard problem has non-standard twist;
• complex problem has no (reliable) theory;
• or (almost) anywhere else.
◮ Have described
• basic ideas
• confidence intervals
• tests
• some approaches for regression

Anthony Davison: Bootstrap Methods and their Application, 94

Regression

Books
◮ Chernick (1999) Bootstrap Methods: A Practicioner’s
Guide. Wiley
◮ Davison and Hinkley (1997) Bootstrap Methods and their
Application. Cambridge University Press
◮ Efron and Tibshirani (1993) An Introduction to the
Bootstrap. Chapman & Hall
◮ Hall (1992) The Bootstrap and Edgeworth Expansion.
Springer
◮ Lunneborg (2000) Data Analysis by Resampling: Concepts
and Applications. Duxbury Press
◮ Manly (1997) Randomisation, Bootstrap and Monte Carlo
Methods in Biology. Chapman & Hall
◮ Shao and Tu (1995) The Jackknife and Bootstrap. Springer

View publication stats

Anthony Davison: Bootstrap Methods and their Application, 95