STAT 801: Mathematical Statistics

Course outline:

• Distribution Theory.

– Basic concepts of probability.

– Distributions

– Expectation and moments

– Transforms (such as characteristic func-

tions, moment generating functions)

– Distribution of transformations

1
• Point estimation

– Maximum likelihood estimation.

– Method of moments.

– Optimality Theory.

– Bias, mean squared error.

– Sufficiency.

– Uniform Minimum Variance (UMV) Un-

biased Estimators.

2
• Hypothesis Testing

– Neyman Pearson optimality theory.

– Most Powerful, Uniformly Most Power-

ful, Unbiased tests.

• Confidence sets

– Pivots

– Associated Hypothesis Tests

– Inversion of hypothesis tests to get con-

fidence sets.

• Decision Theory.

3
 Statistics versus Probability

Standard view of scientific inference has a set

of theories which make predictions about the
outcomes of an experiment:

Theory Prediction
A 1
B 2
C 3

Conduct experiment, see outcome 2: infer B

is correct (or at least A and C are wrong).

Add Randomness

Theory Prediction
A Usually 1 sometimes 2 never 3
B Usually 2 sometimes 1 never 3
C Usually 3 sometimes 1 never 2

See outcome 2: infer Theory B probably cor-

rect, Theory A probably not correct, Theory C
is wrong.
4
Probability Theory: construct table: com-
pute likely outcomes of experiments.

Statistics: inverse process. Use table to draw

inferences from outcome of experiment. How
should we do it and how wrong are our infer-
ences likely to be? Notice: hopeless task unless
different theories make different predictions.

Start with Probability; switch after about 5

weeks to statistics.

5
 Probability Definitions

Probability Space (or Sample Space): or-

dered triple (Ω, F , P ).

• Ω is a set (possible outcomes); elements

are ω called elementary outcomes.

• F is a family of subsets (events) of Ω with

the property that F is a σ-field (or Borel
field or σ-algebra):

1. Empty set ∅ and Ω are members of F .

2. A ∈ F implies Ac = {ω ∈ Ω : ω 6∈ A} ∈ F .

3. A1, A2, · · · in F implies A = ∪∞

i=1 Ai ∈ F .

6
 • P a function, domain F , range a subset of
[0, 1] satisfying:

1. P (∅) = 0 and P (Ω) = 1.

2. Countable additivity: A1, A2, · · · pair-

wise disjoint (j 6= k Aj ∩ Ak = ∅)
∞
P (∪∞
X
i=1 Ai) = P (Ai )
i=1

Axioms guarantee can compute probabilities by

usual rules, including approximation.

Consequences of axioms:

Ai ∈ F ; i = 1, 2, · · · implies ∩i Ai ∈ F

A1 ⊂ A2 ⊂ · · · implies P (∪Ai) = lim P (An)

n→∞

A1 ⊃ A2 ⊃ · · · implies P (∩Ai) = lim P (An)

n→∞

7
Vector valued random variable: function X :
Ω 7→ Rp such that, writing X = (X1 , . . . , Xp ),

P (X1 ≤ x1, . . . , Xp ≤ xp)

is defined for any constants (x1, . . . , xp). For-
mally the notation

X1 ≤ x 1 , . . . , X p ≤ x p
is a subset of Ω or event:

{ω ∈ Ω : X1 (ω) ≤ x1, . . . , Xp(ω) ≤ xp} .

Remember X is a function on Ω so X1 is also
a function on Ω.

In almost all of probability and statistics the

dependence of a random variable on a point
in the probability space is hidden! You almost
always see X not X(ω).

8
Borel σ-field in Rp: smallest σ-field in Rp con-
taining every open ball.

Every common set is a Borel set, that is, in

the Borel σ-field.

An Rp valued random variable is a map X :

Ω 7→ Rp such that when A is Borel then {ω ∈
Ω : X(ω) ∈ A} ∈ F .

Fact: this is equivalent to

{ω ∈ Ω : X1 (ω) ≤ x1, . . . , Xp(ω) ≤ xp} ∈ F

for all (x1, . . . , xp) ∈ Rp.

Jargon and notation: we write P (X ∈ A) for

P ({ω ∈ Ω : X(ω) ∈ A}) and define the distri-
bution of X to be the map

A 7→ P (X ∈ A)
which is a probability on the set Rp with the
Borel σ-field rather than the original Ω and F .
9
Cumulative Distribution Function (CDF)
of X: function FX on Rp defined by

FX (x1, . . . , xp) = P (X1 ≤ x1, . . . , Xp ≤ xp) .

Properties of FX (usually just F ) for p = 1:

1. 0 ≤ F (x) ≤ 1.

2. x > y ⇒ F (x) ≥ F (y) (monotone non-

decreasing).

3. limx→−∞ F (x) = 0 and limx→∞ F (x) = 1.

4. limx&y F (x) = F (y) (right continuous).

5. limx%y F (x) ≡ F (y−) exists.

6. F (x) − F (x−) = P (X = x).

7. FX (t) = FY (t) for all t implies that X and Y

have the same distribution, that is, P (X ∈
A) = P (Y ∈ A) for any (Borel) set A.

10
Defn: Distribution of rv X is discrete (also
call X discrete) if ∃ countable set x1, x2, · · ·
such that
X
P (X ∈ {x1, x2 · · · }) = 1 = P (X = xi) .
i
In this case the discrete density or probability
mass function of X is
fX (x) = P (X = x) .

Defn: Distribution of rv X is absolutely con-

tinuous if there is a function f such that
Z
P (X ∈ A) = f (x)dx (1)
A
for any (Borel) set A. This is a p dimensional
integral in general. Equivalently
Z x
F (x) = f (y) dy .
−∞

Defn: Any f satisfying (1) is a density of X.

For most x F is differentiable at x and

F 0 (x) = f (x) .
11
Example: X is Uniform[0,1].

 0 x≤0

F (x) = x 0<x<1
 1 x ≥ 1.



 1
 0<x<1
f (x) = undefined x ∈ {0, 1}
 0

otherwise .

Example: X is exponential.
1 − e−x x > 0
(
F (x) =
0 x ≤ 0.

−x

 e
 x>0
f (x) = undefined x = 0
 0

x < 0.

12
 Distribution Theory

Basic Problem:

Start with assumptions about f or CDF of ran-

dom vector X = (X1 , . . . , Xp ).

Define Y = g(X1, . . . , Xp ) to be some function

of X (usually some statistic of interest).

Compute distribution or CDF or density of Y ?

Univariate Techniques

Method 1: compute the CDF by integration

and differentiate to find fY .

Example: U ∼ Uniform[0, 1] and Y = − log U .

FY (y) = P (Y ≤ y) = P (− log U ≤ y)
−y
= P (log U ≥ −y) = P (U ≥ e )
1 − e−y y > 0
(
=
0 y≤0
so Y has standard exponential distribution.
13
Example: Z ∼ N (0, 1), i.e.
1 −z 2/2
fZ (z) = √ e
2π
and Y = Z 2. Then

FY (y) = P (Z 2 ≤ y)
(
0 y<0
= √ √
P (− y ≤ Z ≤ y) y ≥ 0 .
Now differentiate
√ √ √ √
P (− y ≤ Z ≤ y) = FZ ( y) − FZ (− y)
to get

 0 h y<0

d F (√y) − F (−√y) y > 0
 i
fY (y) = dy Z Z

undefined y = 0.



14
Then
d √ √ d√
FZ ( y) = fZ ( y) y
dy dy
1  √ 2  1 −1/2
= √ exp − ( y ) /2 y
2π 2
1
= √ e−y/2 .
2 2πy
(Similar formula for other derivative.) Thus

1 −y/2 y > 0
 √2πy e


fY (y) = 0 y<0

 undefined

y = 0.

We will find indicator notation useful:

(
1 y>0
1(y > 0) =
0 y≤0
which we use to write
1
fY (y) = √ e−y/2 1(y > 0)
2πy
(changing definition unimportantly at y = 0).

15
Notice: I never evaluated FY before differen-
tiating it. In fact FY and FZ are integrals I
can’t do but I can differentiate them anyway.
Remember fundamental theorem of calculus:
d x
Z
f (y) dy = f (x)
dx a
at any x where f is continuous.

Summary: for Y = g(X) with X and Y each

real valued

P (Y ≤ y) = P (g(X) ≤ y)
= P (X ∈ g −1(−∞, y]) .
Take d/dy to compute the density
d
Z
fY (y) = fX (x) dx .
dy {x:g(x)≤y}
Often can differentiate without doing integral.

16
Method 2: Change of variables.

Assume g is one to one. I do: g is increas-

ing and differentiable. Interpretation of density
(based on density = F 0 ):
P (y ≤ Y ≤ y + δy)
fY (y) = lim
δy→0 δy
F (y + δy) − FY (y)
= lim Y
δy→0 δy
and
P (x ≤ X ≤ x + δx)
fX (x) = lim .
δx→0 δx
Assume y = g(x). Define δy by y + δy = g(x +
δx). Then
P (y ≤ Y ≤ y + δy) = P (x ≤ X ≤ x + δx) .
Get
P (y ≤ Y ≤ y + δy)) P (x ≤ X ≤ x + δx)/δx
= .
δy {g(x + δx) − y}/δx
Take limit to get
fY (y) = fX (x)/g 0(x)
or
fY (g(x))g 0(x) = fX (x) .
17
Alternative view:

Each probability is integral of a density:

First is integral of fY over the small interval

from y = g(x) to y = g(x + δx). The interval
is narrow so fY is nearly constant and

P (y ≤ Y ≤ g(x + δx)) ≈ fY (y)(g(x + δx) − g(x)) .

Since g has a derivative the difference

g(x + δx) − g(x) ≈ δxg 0(x)

and we get

P (y ≤ Y ≤ g(x + δx)) ≈ fY (y)g 0(x)δx .

Same idea applied to P (x ≤ X ≤ x + δx) gives

P (x ≤ X ≤ x + δx) ≈ fX (x)δx
so that

fY (y)g 0(x)δx ≈ fX (x)δx

or, cancelling the δx in the limit

fY (y)g 0(x) = fX (x) .

18
If you remember y = g(x) then you get

fX (x) = fY (g(x))g 0(x) .

Or solve y = g(x) to get x in terms of y, that
is, x = g −1(y) and then

fY (y) = fX (g −1(y))/g 0(g −1(y))

This is just the change of variables formula
for doing integrals.

Remark: For g decreasing g 0 < 0 but then

the interval (g(x), g(x + δx)) is really (g(x +
δx), g(x)) so that g(x) − g(x + δx) ≈ −g 0(x)δx.
In both cases this amounts to the formula

fX (x) = fY (g(x))|g 0(x)| .

Mnemonic:

fY (y)dy = fX (x)dx .

19
Example: X ∼ Weibull(shape α, scale β) or
!α−1
α x
fX (x) = exp {−(x/β)α} 1(x > 0) .
β β

Let Y = log X or g(x) = log(x).

Solve y = log x: x = exp(y) or g −1(y) = ey .

Then g 0(x) = 1/x and

1/g 0(g −1(y)) = 1/(1/ey ) = ey .
Hence
!α−1
α e y
fY (y) = exp {−(ey /β)α } 1(ey > 0)ey .
β β
For any y, ey > 0 so indicator = 1. So
α
fY (y) = α exp {αy − eαy /β α} .
β
Define φ = log β and θ = 1/α; then,
1 y−φ y−φ
  
fY (y) = exp − exp .
θ θ θ
Extreme Value density with location param-
eter φ and scale parameter θ. (Note: several
distributions are called Extreme Value.)
20
 Marginalization

Simplest multivariate problem:

X = (X1 , . . . , Xp), Y = X1
(or in general Y is any Xj ).

Theorem 1 If X has density f (x1 , . . . , xp) and

q < p then Y = (X1 , . . . , Xq ) has density

fY (x1, . . . , xq ) =
Z ∞ Z ∞
··· f (x1 , . . . , xp) dxq+1 . . . dxp
−∞ −∞

fX1 ,...,Xq is the marginal density of X1 , . . . , Xq

and fX the joint density of X but they are both
just densities. “Marginal” just to distinguish
from the joint density of X.

21
Example: The function

f (x1 , x2) = Kx1x21(x1 > 0, x2 > 0, x1 + x2 < 1)

is a density provided
Z ∞ Z ∞
P (X ∈ R2 ) = f (x1 , x2) dx1 dx2 = 1 .
−∞ −∞
The integral is
Z 1 Z 1−x
1
K x1x2 dx2 dx1
0 0
Z 1
=K x1 (1 − x1)2 dx1/2
0
= K(1/2 − 2/3 + 1/4)/2
= K/24
so K = 24. The marginal density of x1 is
Z ∞
fX1 (x1 ) = 24x1 x2
−∞
× 1(x1 > 0, x2 > 0, x1 + x2 < 1) dx2
Z 1−x
1
=24 x1 x21(0 < x1 < 1)dx2
0
=12x1(1 − x1)21(0 < x1 < 1) .
This is a Beta(2, 3) density.

22
General case: Y = (Y1, . . . , Yq ) with compo-
nents Yi = gi(X1 , . . . , Xp).

Case 1: q > p.

Y won’t have density for “smooth” g.

Y will have a singular or discrete distribution.

Problem rarely of real interest.

But, e.g., residuals in regression problems have

singular distribution.

Case 2: q = p.

Use change of variables formula which gen-

eralizes the one derived above for the case
p = q = 1. (See below.)

23
Case 3: q < p.

Pad out Y : add on p − q more variables (care-

fully chosen) say Yq+1, . . . , Yp.

Find functions gq+1, . . . , gp. Define for q < i ≤

p, Yi = gi(X1 , . . . , Xp ) and Z = (Y1, . . . , Yp) .

Choose gi so that we can use change of vari-

ables on g = (g1, . . . , gp) to compute fZ .

Find fY by integration:

fY (y1,...,yq )=
R∞ R∞
−∞ ··· −∞ fZ (y1 ,...,yq ,zq+1 ,...,zp )dzq+1 ...dzp

24
 Change of Variables

Suppose Y = g(X) ∈ Rp with X ∈ Rp having

density fX . Assume g is a one to one (“in-
jective”) map, i.e., g(x1) = g(x2) if and only
if x1 = x2. Find fY :

Step 1: Solve for x in terms of y: x = g −1(y).

Step 2: Use basic equation:

fY (y)dy = fX (x)dx
and rewrite it in the form
dx
fY (y) = fX (g −1(y))
dy
Interpretation of derivative dx
dy when p > 1:
 !
dx  ∂xi 
= det


dy  ∂yj 
which is the so called Jacobian.

25
Equivalent formula inverts the matrix:
fX (g −1(y))
fY (y) =  dy 
  .
 dx 

This notation means

 
∂y1 ∂y1 ∂y1 

···

∂x1 ∂x2 ∂xp 
 
 dy 
 
..
 
  = det 




dx
    ∂yp ∂yp ∂yp 

 ∂x1 ∂x2 ··· ∂xp 
but with x replaced by the corresponding value
of y, that is, replace x by g −1(y).

Example: The density

x2 2
( )
1 + x
fX (x1 , x2) = exp − 1 2
2π 2
is the standard bivariate normal
q density. Let
Y = (Y1 , Y2) where Y1 = X12 + X22 and 0 ≤
Y2 < 2π is angle from the positive x axis to the
ray from the origin to the point (X1 , X2 ). I.e.,
Y is X in polar co-ordinates.

26
Solve for x in terms of y:

X1 = Y1 cos(Y2)
X2 = Y1 sin(Y2)
so that
g(x1, x2) = (g1(x1, x2), g2(x1 , x2))
q
= ( x2
1 + x 2
2, argument(x1, x2))

g −1(y1 , y2) = (g1−1(y1 , y2), g2−1(y1, y2))

= (y1 cos(y2 ), y1 sin(y2))

  
 
 dx 

 cos(y2) −y1 sin(y2 ) 
  = det 
    

 dy 
sin(y2) y1 cos(y2) 
 


= y1 .
It follows that
y12
( )
1
fY (y1, y2) = exp − y1 ×
2π 2

1(0 ≤ y1 < ∞)1(0 ≤ y2 < 2π) .

27
Next: marginal densities of Y1, Y2?

Factor fY as fY (y1 , y2) = h1(y1)h2 (y2) where

−y12 /2
h1(y1 ) = y1e 1(0 ≤ y1 < ∞)
and

h2(y2) = 1(0 ≤ y2 < 2π)/(2π) .

Then
Z ∞
fY1 (y1) = h1 (y1)h2 (y2) dy2
−∞ Z ∞
= h1(y1 ) h2(y2) dy2
−∞
so marginal density of Y1 is a multiple of h1 .
R
Multiplier makes fY1 = 1 but in this case
Z ∞ Z 2π
h2 (y2) dy2 = (2π)−1 dy2 = 1
−∞ 0
so that
−y12/2
fY1 (y1) = y1e 1(0 ≤ y1 < ∞) .
(Special Weibull or Rayleigh distribution.)

28
Similarly

fY2 (y2) = 1(0 ≤ y2 < 2π)/(2π)

which is the Uniform((0, 2π) density. Exercise:
W = Y12/2 has standard exponential distribu-
tion. Recall: by definition U = Y12 has a χ2
distribution on 2 degrees of freedom. Exer-
cise: find χ2
2 density.

Note: We show below factorization of density

is equivalent to independence.

29
 Independence, conditional distributions

So far density of X specified explicitly. Often

modelling leads to a specification in terms of
marginal and conditional distributions.
Def’n: Events A and B are independent if

P (AB) = P (A)P (B) .

(Notation: AB is the event that both A and B
happen, also written A ∩ B.)
Def’n: Ai, i = 1, . . . , p are independent if
r
Y
P (Ai1 · · · Air ) = P (Aij )
j=1
for any 1 ≤ i1 < · · · < ir ≤ p.
Example: p = 3
P (A1 A2A3) = P (A1 )P (A2 )P (A3 )
P (A1 A2) = P (A1 )P (A2 )
P (A1 A3) = P (A1 )P (A3 )
P (A2 A3) = P (A2 )P (A3 ) .
All these equations needed for independence!

30
Example: Toss a coin twice.

A1 = {first toss is a Head}

A2 = {second toss is a Head}
A3 = {first toss and second toss different}

Then P (Ai ) = 1/2 for each i and for i 6= j

1
P (Ai ∩ Aj ) =
4
but

P (A1 ∩ A2 ∩ A3) = 0 6= P (A1)P (A2 )P (A3 ) .

Def’n: X and Y are independent if

P (X ∈ A; Y ∈ B) = P (X ∈ A)P (Y ∈ B)
for all A and B.

Def’n: Rvs X1 , . . . , Xp independent:

Y
P (X1 ∈ A1, · · · , Xp ∈ Ap) = P (Xi ∈ Ai)
for any A1, . . . , Ap.
31
Theorem:
1. If X and Y are independent then for all x, y
FX,Y (x, y) = FX (x)FY (y) .
2. If X and Y are independent with joint den-
sity fX,Y (x, y) then X and Y have densities
fX and fY , and
fX,Y (x, y) = fX (x)fY (y) .
3. If X and Y independent with marginal den-
sities fX and fY then (X, Y ) has joint den-
sity
fX,Y (x, y) = fX (x)fY (y) .

4. If FX,Y (x, y) = FX (x)FY (y) for all x, y then

X and Y are independent.
5. If (X, Y ) has density f (x, y) and there ex-
ist g(x) and h(y) st f (x, y) = g(x)h(y) for
(almost) all (x, y) then X and Y are inde-
pendent with densities given by
Z ∞
fX (x) = g(x)/ g(u)du
−∞
Z ∞
fY (y) = h(y)/ h(u)du .
−∞

32
Proof:

1: Since X and Y are independent so are the

events X ≤ x and Y ≤ y; hence

P (X ≤ x, Y ≤ y) = P (X ≤ x)P (Y ≤ y) .

2: Suppose X and Y real valued.

Asst 2: existence of fX,Y implies that of fX

and fY (marginal density formula). Then for
any sets A and B
Z Z
P (X ∈ A, Y ∈ B) = fX,Y (x, y)dydx
ZA B Z
P (X ∈ A)P (Y ∈ B) = fX (x)dx fY (y)dy
ZA Z B
= fX (x)fY (y)dydx
A B

Since P (X ∈ A, Y ∈ B) = P (X ∈ A)P (Y ∈ B)
Z Z
[fX,Y (x, y) − fX (x)fY (y)]dydx = 0
A B
It follows (measure theory) that the quantity
in [] is 0 (almost every pair (x, y)).
33
3: For any A and B we have

P (X ∈ A, Y ∈B)
= P (X ∈ A)P (Y ∈ B)
Z Z
= fX (x)dx fY (y)dy
ZA Z B
= fX (x)fY (y)dydx
A B
Define g(x, y) = fX (x)fY (y); we have proved
that for C = A × B
Z
P ((X, Y ) ∈ C) = g(x, y)dydx
C
To prove that g is fX,Y we need only prove that
this integral formula is valid for an arbitrary
Borel set C, not just a rectangle A × B.

Use monotone class argument: collection C

of sets C for which identity holds has closure
properties which guarantee that C includes the
Borel sets.

4: Another monotone class argument.

34
5: We are given
Z Z
P (X ∈ A, Y ∈ B) = g(x)h(y)dydx
ZA B Z
= g(x)dx h(y)dy
A B
Take B = R1 to see that
Z
P (X ∈ A) = c1 g(x)dx
A
R
where c1 = h(y)dy. So c1g is the density of
RR
X. Since fX,Y (xy)dxdy = 1 we see that
R R R
g(x)dx h(y)dy = 1 so that c1 = 1/ g(x)dx.
Similar argument for Y .

Theorem: If X1 , . . . , Xp are independent and

Yi = gi(Xi ) then Y1, . . . , Yp are independent.
Moreover, (X1 , . . . , Xq ) and (Xq+1 , . . . , Xp ) are
independent.

35
 Conditional probability

Def’n: P (A|B) = P (AB)/P (B) if P (B) 6= 0.

Def’n: For discrete X and Y the conditional

probability mass function of Y given X is
fY |X (y|x) = P (Y = y|X = x)
= fX,Y (x, y)/fX (x)
X
= fX,Y (x, y)/ fX,Y (x, t)
t

For absolutely continuous X P (X = x) = 0

for all x. What is P (A|X = x) or fY |X (y|x)?
Solution: use limit
P (A|X = x) = lim P (A|x ≤ X ≤ x + δx)
δx→0
If, e.g., X, Y have joint density fX,Y then with
A = {Y ≤ y} we have
P (A|x ≤ X ≤ x + δx)
P (A ∩ {x ≤ X ≤ x + δx})
=
P (x ≤ X ≤ x + δx)
R y R x+δx
−∞ x fX,Y (u, v)dudv
= R x+δx
x fX (u)du
36
Divide top, bottom by δx; let δx → 0. Denom
converges to fX (x); numerator converges to
Z y
fX,Y (x, v)dv
−∞
Define conditional cdf of Y given X = x:
Ry
−∞ fX,Y (x, v)dv
P (Y ≤ y|X = x) =
fX (x)
Differentiate wrt y to get def’n of conditional
density of Y given X = x:

fY |X (y|x) = fX,Y (x, y)/fX (x) ;

in words “conditional = joint/marginal”.

37
 The Multivariate Normal Distribution

Defn: Z ∈ R1 ∼ N (0, 1) iff

1 −z 2/2
fZ (z) = √ e .
2π

Defn: Z ∈ Rp ∼ M V N (0, I) if and only if Z =

(Z1 , . . . , Zp)t with the Zi independent and each
Zi ∼ N (0, 1).

In this case according to our theorem

Y 1 −z 2/2
fZ (z1 , . . . , zp) = √ e i
2π
= (2π)−p/2 exp{−z tz/2} ;
superscript t denotes matrix transpose.

Defn: X ∈ Rp has a multivariate normal distri-

bution if it has the same distribution as AZ + µ
for some µ ∈ Rp, some p×p matrix of constants
A and Z ∼ M V N (0, I).

38
Matrix A singular: X does not have a density.

A invertible: derive multivariate normal density

by change of variables:
X = AZ + µ ⇔ Z = A−1(X − µ)
∂X ∂Z
=A = A−1 .
∂Z ∂X
So
fX (x) = fZ (A−1(x − µ))| det(A−1)|
exp{−(x − µ)t(A−1)tA−1(x − µ)/2}
= .
(2π)p/2 | det A|
Now define Σ = AAt and notice that
Σ−1 = (At)−1A−1 = (A−1)tA−1
and
det Σ = det A det At = (det A)2 .
Thus fX is
exp{−(x − µ)tΣ−1(x − µ)/2}
;
(2π)p/2(det Σ)1/2
the M V N (µ, Σ) density. Note density is the
same for all A such that AAt = Σ. This justi-
fies the notation M V N (µ, Σ).
39
For which µ, Σ is this a density?

Any µ but if x ∈ Rp then

xtΣx = xtAAtx
= (Atx)t (Atx)
p
yi2 ≥ 0
X
=
1
where y = Atx. Inequality strict except for
y = 0 which is equivalent to x = 0. Thus Σ is
a positive definite symmetric matrix.

Conversely, if Σ is a positive definite symmet-

ric matrix then there is a square invertible ma-
trix A such that AAt = Σ so that there is a
M V N (µ, Σ) distribution. (A can be found via
the Cholesky decomposition, e.g.)

When A is singular X will not have a density:

∃a such that P (at X = atµ) = 1; X is confined
to a hyperplane.

Still true: distribution of X depends only on

Σ = AAt: if AAt = BB t then AZ + µ and
BZ + µ have the same distribution.
40
Properties of the M V N distribution

1: All margins are multivariate normal: if

" #
X1
X=
X2
" #
µ1
µ=
µ2
and
" #
Σ11 Σ12
Σ=
Σ21 Σ22
then X ∼ M V N (µ, Σ) ⇒ X1 ∼ M V N (µ1, Σ11 ).

2: All conditionals are normal: the conditional

distribution of X1 given X2 = x2 is M V N (µ1 +
Σ12 Σ−1 −1
22 (x2 − µ2), Σ11 − Σ12 Σ22 Σ21 )

3: M X + ν ∼ M V N (M µ + ν, M ΣM t): affine
transformation of MVN is normal.

41
 Normal samples: Distribution Theory

Theorem: Suppose X1 , . . . , Xn are indepen-

dent N (µ, σ 2) random variables. Then

1. X̄ (sample mean)and s2 (sample variance)

independent.

2. n1/2 (X̄ − µ)/σ ∼ N (0, 1).

3. (n − 1)s2 /σ 2 ∼ χ2
n−1.

4. n1/2 (X̄ − µ)/s ∼ tn−1.

42
Proof: Let Zi = (Xi − µ)/σ.

Then Z1 , . . . , Zp are independent N (0, 1).

So Z = (Z1, . . . , Zp)t is multivariate standard

normal.

Note that X̄ = σ Z̄+µ and s2 = (Xi −X̄)2/(n−

P

1) = σ 2 (Zi − Z̄)2/(n − 1) Thus

P

n1/2(X̄ − µ)
= n1/2Z̄
σ

(n − 1)s2 X
2
= (Z i − Z̄)
σ2
and
n1/2(X̄ − µ) n1/2Z̄
T = =
s sZ
where (n − 1)sZ = (Zi − Z̄)2 .
2 P

So: reduced to µ = 0 and σ = 1.

43
Step 1: Define
√
Y = ( nZ̄, Z1 − Z̄, . . . , Zn−1 − Z̄)t .
(So Y has same dimension as Z.) Now
 
√1 √1 ··· √1 
Z1

 n n n 
 1−1 ··· −n1 −n1 Z2
  
Y = n ..
 
 −1 1 ··· 1
 
n 1 − n −n  
.. .. .. ..
 
Zn

or letting M denote the matrix

Y = MZ .
It follows that Y ∼ M V N (0, M M t) so we need
to compute M M t:
 
1 0 0 ··· 0

0 1−n 1 −1 · · · − 1 
t 
MM =  n n

.. − 1 ... · · · 1


 n − n


0 .
. ··· 1−n 1
 
1 0
=  .
 
0 Q

44
Solve for Z from Y : Zi = n−1/2Y1 + Yi+1 for
1 ≤ i ≤ n − 1. Use the identity
n
X
(Zi − Z̄) = 0
i=1
to get Zn = − n
P
Y + n −1/2Y . So M invert-
i=2 i 1
ible:
 
1 0
Σ−1 ≡ (M M t)−1 =   .
 

0 Q−1

Use change of variables to find fY . Let y2 de-

note vector whose entries are y2, . . . , yn. Note
that
y tΣ−1 y = y12 + y2
t −1
Q y2 .
Then

fY (y) =(2π)−n/2 exp[−y tΣ−1y/2]/| det M |

1 −y 2 /2
=√ e 1 ×
2π
(2π)−(n−1)/2 exp[−y2 t Q−1 y /2]
2
.
| det M |

45
Note: fY is ftn of y1 times a ftn of y2, . . . , yn.

√
Thus nZ̄ is independent of Z1 − Z̄, . . . , Zn−1 −
Z̄.

Since s2
Z is a√function of Z1 − Z̄, . . . , Zn−1 − Z̄
we see that nZ̄ and s2 Z are independent.

Also, density of Y1 is a multiple of the function

of y1 in the factorization above. But factor is
√
standard normal density so nZ̄ ∼ N (0, 1).

First 2 parts done. Third part is a homework

exercise.

46
Derivation of the χ2 density:

Suppose Z1, . . . , Zn are independent N (0, 1). De-

fine χ2 2
n distribution to be that of U = Z1 +· · ·+
Zn2. Define angles θ1, . . . , θn−1 by

Z1 = U 1/2 cos θ1
Z2 = U 1/2 sin θ1 cos θ2
.. = ..
Zn−1 = U 1/2 sin θ1 · · · sin θn−2 cos θn−1
Zn = U 1/2 sin θ1 · · · sin θn−1 .
(Spherical co-ordinates in n dimensions. The
θ values run from 0 to π except last θ from 0
to 2π.) Derivative formulas:
∂Zi 1
= Zi
∂U 2U
and

∂Zi  0
 j>i
= −Zi tan θi j = i
∂θj  Z cot θ

j < i.
i j

47
Fix n = 3 to clarify the formulas. Use short-
√
hand R = U

Matrix of partial derivatives is

cos θ1
 
 2R −R sin θ1 0 
 
 
 sin θ1 cos θ2
R cos θ1 cos θ2 −R sin θ1 sin θ2  .

2R

 
 
 
sin θ1 sin θ2
2R R cos θ1 sin θ2 R sin θ1 cos θ2
Find determinant by adding 2U 1/2 cos θj / sin θj
times col 1 to col j + 1 (no change in determi-
nant).

Resulting matrix lower triangular; diagonal en-

tries:
cos θ1 R cos θ2 R sin θ1
, ,
R cos θ1 cos θ2

Multiply these together to get

U 1/2 sin(θ1)/2
(non-negative for all U and θ1).
48
General n: every term in the first column con-
tains a factor U −1/2/2 while every other entry
has a factor U 1/2.

FACT: multiplying a column in a matrix by c

multiplies the determinant by c.

SO: Jacobian of transformation is

u(n−1)/2u−1/2/2 × h(θ1, θn−1)

for some function, h, which depends only on
the angles.

Thus joint density of U, θ1, . . . θn−1 is

(2π)−n/2 exp(−u/2)u(n−2)/2h(θ1, · · · , θn−1)/2 .

To compute the density of U we must do an
n−1 dimensional multiple integral dθn−1 · · · dθ1 .

Answer has the form

cu(n−2)/2 exp(−u/2)
for some c.
49
Evaluate c by making
Z Z ∞
fU (u)du = c u(n−2)/2 exp(−u/2)du
0
= 1.
Substitute y = u/2, du = 2dy to see that
Z ∞
c2n/2 y (n−2)/2e−y dy = c2n/2Γ(n/2)
0
= 1.
CONCLUSION: the χ2
n density is
 (n−2)/2
1 u
e−u/21(u > 0) .
2Γ(n/2) 2

50
Fourth part: consequence of first 3 parts and
def’n of tν distribution.

Defn: T ∼ tν if T has same distribution as

q
Z/ U/ν
for Z ∼ N (0, 1), U ∼ χ2
ν and Z, U independent.

Derive density of T in this definition:

q
P (T ≤ t) = P (Z ≤ t U/ν)
Z ∞Z √ t u/ν
= fZ (z)fU (u)dzdu
0 −∞
Differentiate wrt t by differentiating inner in-
tegral:
∂ bt
Z
f (x)dx = bf (bt) − af (at)
∂t at
by fundamental thm of calculus. Hence
 1/2 2
Z ∞ !
d fU (u) u t u
P (T ≤ t) = √ exp − du .
dt 0 2π ν 2ν

51
Plug in
1
fU (u) = (u/2)(ν−2)/2 e−u/2
2Γ(ν/2)
to get
R∞
(u/2) (ν−1)/2 e−u(1+t2/ν)/2 du
fT (t) = 0 √ .
2 πνΓ(ν/2)
Substitute y = u(1 + t2 /ν)/2, to get

dy = (1 + t2/ν)du/2

(u/2)(ν−1)/2 = [y/(1 + t2/ν)](ν−1)/2

leading to
(1 + t2 /ν)−(ν+1)/2 ∞ (ν−1)/2 −y
Z
fT (t) = √ y e dy
πνΓ(ν/2) 0
or
Γ((ν + 1)/2) 1
fT (t) = √ .
πνΓ(ν/2) (1 + t2 /ν)(ν+1)/2

52
 Expectation, moments

Two elementary definitions of expected values:

Defn: If X has density f then

Z
E{g(X)} = g(x)f (x) dx .

Defn: If X has discrete density f then

X
E{g(X)} = g(x)f (x) .
x

FACT: if Y = g(X) for a smooth g

Z
E(Y ) = yfY (y) dy
Z
= g(x)fY (g(x))g 0(x) dx
= E{g(X)}
by change of variables formula for integration.
This is good because otherwise we might have
two different values for E(eX ).

53
In general, there are random variables which
are neither absolutely continuous nor discrete.
Here’s how probabilists define E in general.

Defn: RV X is simple if we can write

n
X
X(ω) = ai1(ω ∈ Ai)
1
for some constants a1, . . . , an and events Ai.

Defn: For a simple rv X define

X
E(X) = aiP (Ai ) .

For positive random variables which are not

simple extend definition by approximation:

Defn: If X ≥ 0 then

E(X) = sup{E(Y ) : 0 ≤ Y ≤ X, Y simple} .

54
Defn: X is integrable if

E(|X|) < ∞ .
In this case we define

E(X) = E{max(X, 0)} − E{max(−X, 0)} .

Facts: E is a linear, monotone, positive oper-

ator:

1. Linear: E(aX + bY ) = aE(X) + bE(Y ) pro-

vided X and Y are integrable.

2. Positive: P (X ≥ 0) = 1 implies E(X) ≥ 0.

3. Monotone: P (X ≥ Y ) = 1 and X, Y inte-

grable implies E(X) ≥ E(Y ).

55
Major technical theorems:

Monotone Convergence: If 0 ≤ X1 ≤ X2 ≤
· · · and X = lim Xn (which has to exist) then

E(X) = lim E(Xn ) .

n→∞

Dominated Convergence: If |Xn | ≤ Yn and

∃ rv X such that Xn → X (technical details
of this convergence later in the course) and
a random variable Y such that Yn → Y with
E(Yn) → E(Y ) < ∞ then

E(Xn ) → E(X) .
Often used with all Yn the same rv Y .

Fatou’s Lemma: If Xn ≥ 0 then

E(lim sup Xn ) ≤ lim sup E(Xn ) .

56
Theorem: With this definition of E if X has
density f (x) (even in Rp say) and Y = g(X)
then
Z
E(Y ) = g(x)f (x)dx .

(Could be a multiple integral.) If X has pmf f

then
X
E(Y ) = g(x)f (x) .
x

First conclusion works, e.g., even if X has a

density but Y doesn’t.

57
Defn: The r th moment (about the origin) of
a real rv X is µ0r = E(X r ) (provided it exists).
We generally use µ for E(X).

Defn: The r th central moment is

µr = E[(X − µ)r ]
We call σ 2 = µ2 the variance.

Defn: For an Rp valued random vector X

µX = E(X)
is the vector whose ith entry is E(Xi ) (provided
all entries exist).

Defn: The (p × p) variance covariance matrix

of X is
t
h i
Var(X) = E (X − µ)(X − µ)
which exists provided each component Xi has
a finite second moment.

58
Moments and probabilities of rare events are
closely connected as will be seen in a number
of important probability theorems.

Example: Markov’s inequality

P (|X − µ| ≥ t) = E[1(|X − µ| ≥ t)]

r
" #
|X − µ|
≤E r
1(|X − µ| ≥ t)
t
E[|X − µ|r ]
≤
tr
Intuition: if moments are small then large de-
viations from average are unlikely.

Special case is Chebyshev’s inequality:

Var(X)
P (|X − µ| ≥ t) ≤ 2
.
t

59
Example moments: If Z ∼ N (0, 1) then
Z ∞
−z 2/2
√
E(Z) = ze dz/ 2π
−∞ ∞
2
−e−z /2 

= √
2π 

−∞
=0
and (integrating by parts)
r
Z ∞
r −z 2/2
√
E(Z ) = z e dz/ 2π
−∞
2 /2 ∞

−z r−1 e −z 
= √ 
2π

Z ∞−∞


r−2 −z 2/2
√
+ (r − 1) z e dz/ 2π
−∞
so that
µr = (r − 1)µr−2
for r ≥ 2. Remembering that µ1 = 0 and
Z ∞
0 −z 2/2
√
µ0 = z e dz/ 2π = 1
−∞
we find that
(
0 r odd
µr =
(r − 1)(r − 3) · · · 1 r even .

60
If now X ∼ N (µ, σ 2 ), that is, X ∼ σZ + µ, then
E(X) = σE(Z) + µ = µ and

µr (X) = E[(X − µ)r ] = σ r E(Z r )

In particular, we see that our choice of nota-
tion N (µ, σ 2) for the distribution of σZ + µ is
justified; σ is indeed the variance.

Similarly for X ∼ M V N (µ, Σ) we have X =

AZ + µ with Z ∼ M V N (0, I) and

E(X) = µ
and

Var(X) = E (X − µ)(X − µ)t

n o

t
n o
= E AZ(AZ)
= AE(ZZ t)At
= AIAt = Σ .
Note use of easy calculation: E(Z) = 0 and

Var(Z) = E(ZZ t) = I .

61
 Moments and independence

Theorem: If X1 , . . . , Xp are independent and

each Xi is integrable then X = X1 · · · Xp is in-
tegrable and

E(X1 · · · Xp) = E(X1 ) · · · E(Xp ) .

Proof: Suppose each Xi is simple:

X
Xi = xij 1(Xi = xij )
j
where the xij are the possible values of Xi .
Then

62
E(X1 · · · Xp)
X
= x1j1 · · · xpjp
j1...jp
× E(1(X1 = x1j1 ) · · · 1(Xp = xpjp ))
X
= x1j1 · · · xpjp
j1...jp
× P (X1 = x1j1 · · · Xp = xpjp )
X
= x1j1 · · · xpjp
j1...jp
× P (X1 = x1j1 ) · · · P (Xp = xpjp )
X
= x1j1 P (X1 = x1j1 ) × · · ·
j1
X
× xpjp P (Xp = xpjp )
jp
Y
= E(Xi) .

63
General Xi ≥ 0:

Let Xin be Xi rounded down to nearest multi-

ple of 2−n (to maximum of n).

That is: if
k k+1
≤ X i <
2n 2n
then Xin = k/2n for k = 0, . . . , n2n. For Xi > n
putXin = n.

Apply case just done:

Y Y
E( Xin ) = E(Xin) .
Monotone convergence applies to both sides.

For general case write each Xi as difference of

positive and negative parts:

Xi = max(Xi , 0) − max(−Xi , 0) .
Apply positive case.

64
 Moment Generating Functions

Defn: The moment generating function of a

real valued X is
MX (t) = E(etX )
defined for those real t for which the expected
value is finite.

Defn: The moment generating function of

X ∈ Rp is
ut X
MX (u) = E[e ]
defined for those vectors u for which the ex-
pected value is finite.

Formal connection to moments:

∞
E[(tX)k ]/k!
X
MX (t) =
k=0
∞
µ0k tk /k! .
X
=
k=0
Sometimes can find power series expansion of
MX and read off the moments of X from the
coefficients of tk /k!.
65
Theorem: If M is finite for all t ∈ [−, ] for
some  > 0 then

1. Every moment of X is finite.

2. M is C ∞ (in fact M is analytic).

d k
0
3. µk = k MX (0).
dt

Note: C ∞ means has continuous derivatives

of all orders. Analytic means has convergent
power series expansion in neighbourhood of
each t ∈ (−, ).

The proof, and many other facts about mgfs,

rely on techniques of complex variables.

66
 MGFs and Sums

If X1 , . . . , Xp are independent and Y = Xi

P

then the moment generating function of Y is

the product of those of the individual Xi :
tY
E(etXi )
Y
E(e )=
i
or MY = M Xi .
Q

Note: also true for multivariate Xi.

Problem: power series expansion of MY not

nice function of expansions of individual MXi .

Related fact: first 3 moments (meaning µ, σ 2

and µ3) of Y are sums of those of the Xi:
X
E(Y ) = E(Xi )
X
Var(Y ) = Var(Xi )
3 E[(Xi − E(Xi))3 ]
X
E[(Y − E(Y )) ] =
but
E[(Y − E(Y ))4 ] =
{E[(Xi − E(Xi ))4 ] − 3E 2[(Xi − E(Xi ))2 ]}
X

o2
2
nX
+3 E[(Xi − E(Xi )) ]

67
Related quantities: cumulants add up prop-
erly.

Note: log of the mgf of Y is sum of logs of

mgfs of the Xi.

Defn: the cumulant generating function of a

variable X by
KX (t) = log(MX (t)) .
Then
X
KY (t) = KXi (t) .
Note: mgfs are all positive so that the cumula-
tive generating functions are defined wherever
the mgfs are.

SO: KY has power series expansion:

∞
κr tr /r! .
X
KY (t) =
r=1
Defn: the κr are the cumulants of Y .

Observe
X
κr (Y ) = κr (Xi ) .

68
Relation between cumulants and moments:

Cumulant generating function is

K(t) = log(M (t))

= log(1 + [µ1t + µ02t2/2 + µ03t3 /3! + · · · ])
Call quantity in [. . .] x; expand

log(1 + x) = x − x2/2 + x3/3 − x4/4 · · · .

Stick in the power series

x = µt + µ02t2 /2 + µ03t3/3! + · · · ;
Expand out powers of x; collect together like
terms. For instance,

x2 = µ2t2 + µµ02t3
+ [2µ03 µ/3! + (µ02)2/4]t4 + · · ·
x3 = µ3t3 + 3µ02µ2t4/2 + · · ·
x4 = µ 4 t 4 + · · · .
Now gather up the terms. The power t1 oc-
curs only in x with coefficient µ. The power t2
occurs in x and in x2 and so on.
69
Putting these together gives

K(t) =
µt + [µ02 − µ2]t2 /2
+ [µ03 − 3µµ02 + 2µ3]t3 /3!
+ [µ04 − 4µ03µ − 3(µ02)2 + 12µ02 µ2 − 6µ4]t4 /4! · · ·
Comparing coefficients to tr /r! we see that

κ1 = µ
κ2 = µ02 − µ2 = σ 2
κ3 = µ03 − 3µµ02 + 2µ3 = E[(X − µ)3 ]
κ4 = µ04 − 4µ03µ − 3(µ02 )2 + 12µ02 µ2 − 6µ4
= E[(X − µ)4] − 3σ 4 .

Check the book by Kendall and Stuart (or the

new version called Kendall’s Theory of Ad-
vanced Statistics by Stuart and Ord) for for-
mulas for larger orders r.

70
Example: If X1 , . . . , Xp are independent and
Xi has a N (µi, σi2) distribution then
Z ∞
tx −1 (x−µi)2/σi2
√
MXi (t) = e e 2 dx/( 2πσi )
Z−∞∞ √
t(σ z+µ ) −z 2 /2
= e i i e dz/ 2π
−∞Z
tµ
∞
−(z−tσ ) 2 /2+t2σ 2 /2 √
=e i e i i dz/ 2π
−∞
σi2t2/2+tµi
=e .

So cumulant generating function is:

KXi (t) = log(MXi (t)) = σi2t2 /2 + µit .

Cumulants are κ1 = µi, κ2 = σi2 and every
other cumulant is 0.

Cumulant generating function for Y = Xi is

P

σi2t2 /2 + t
X X
KY (t) = µi
which is the cumulant generating function of
N ( µi, σi2 ).
P P

71
Example: Homework: derive moment and cu-
mulant generating function and moments of a
Gamma rv.

Now suppose Z1, . . . , Zν independent N (0, 1)

rvs.

By definition: Sν = 1 Zi has χ2
Pν 2
ν distribution.
It is easy to check S1 = Z12 has density
−1/2 −u/2 √
(u/2) e /(2 π)
and then the mgf of S1 is

(1 − 2t)−1/2 .
It follows that

MSν (t) = (1 − 2t)−ν/2

which is (homework) moment generating func-
tion of a Gamma(ν/2, 2) rv.

SO: χ2
ν dstbn has Gamma(ν/2, 2) density:

(u/2)(ν−2)/2e−u/2 /(2Γ(ν/2)) .

72
Example: The Cauchy density is
1
2
;
π(1 + x )
corresponding moment generating function is
etx
Z ∞
M (t) = dx
−∞ π(1 + x2)
which is +∞ except for t = 0 where we get 1.

Every t distribution has exactly same mgf. So:

can’t use mgf to distinguish such distributions.

Problem: these distributions do not have in-

finitely many finite moments.

So: develop substitute for mgf which is defined

for every distribution, namely, the characteris-
tic function.

Characteristic Functions

Definition: The characteristic function of a

real rv X is
φX (t) = E(eitX )
√
where i = −1 is the imaginary unit.
73
Aside on complex arithmetic.

√
Complex numbers: add i = −1 to the real
numbers.

Require all the usual rules of algebra to work.

So: if i and any real numbers a and b are to

be complex numbers then so must be a + bi.

Multiplication: If we multiply a complex num-

ber a + bi with a and b real by another such
number, say c+di then the usual rules of arith-
metic (associative, commutative and distribu-
tive laws) require

(a + bi)(c + di) =ac + adi + bci + bdi2

=ac + bd(−1) + (ad + bc)i
=(ac − bd) + (ad + bc)i
so this is precisely how we define multiplication.

74
Addition: follow usual rules to get

(a + bi) + (c + di) = (a + c) + (b + d)i .

Additive inverses: −(a + bi) = −a + (−b)i.

Multiplicative inverses:
1 1 a − bi
=
a + bi a + bi a − bi
a − bi
= 2
a − abi + abi − b2i2
a − bi
= 2 .
a + b2
Division:
a + bi (a + bi) (c − di)
=
c + di (c + di) (c − di)
ac − bd + (bc + ad)i
= 2 2
.
c +d
Notice: usual rules of arithmetic don’t require
any more numbers than

x + yi
where x and y are real.
75
Transcendental functions: For real x have
ex = xk /k! so
P

ex+iy = ex eiy .
How to compute eiy ?

Remember i2 = −1 so i3 = −i, i4 = 1 i5 =
i1 = i and so on. Then
∞
(iy)k
eiy =
X

0 k!
=1 + iy + (iy)2/2 + (iy)3 /6 + · · ·
=1 − y 2/2 + y 4/4! − y 6/6! + · · ·
+ iy − iy 3/3! + iy 5/5! + · · ·
= cos(y) + i sin(y)
We can thus write

ex+iy = ex (cos(y) + i sin(y))

76
Identify x + yi with the corresponding point
(x, y) in the plane. Picture the complex num-
bers as forming a plane.

Now every point in the plane can be written in

polar co-ordinates as (r cos θ, r sin θ) and com-
paring this with our formula for the exponential
we see we can write
q
x + iy = x2 + y 2 eiθ = reiθ
for an angle θ ∈ [0, 2π).

Multiplication revisited: x + iy = reiθ , x0 + iy 0 =

0 iθ 0
re .
0 0
(x + iy)(x0 + iy 0) = reiθ r 0eiθ = rr 0ei(θ+θ ) .

77
We will need from time to time a couple of
other definitions:

Definition: The modulus of x + iy is

q
|x + iy| = x2 + y 2 .

Definition: The complex conjugate of x + iy

is x + iy = x − iy.

Some identities: z = x + iy = reiθ and z 0 =

0
x0 + iy 0 = r 0eiθ . Then

zz = x2 + y 2 = r 2 = |z|2

z0 z 0z 0 i(θ 0 −θ)
= 2 = rr e
z |z|

reiθ = re−iθ .

78
Notes on calculus with complex variables.

Essentially usual rules apply so, for example,

d it
e = ieit .
dt
We will (mostly) be doing only integrals over
the real line; the theory of integrals along paths
in the complex plane is a very important part
of mathematics, however.

FACT: (not used explicitly in course). If f :

C 7→ C is differentiable then f is analytic (has
power series expansion).

End of Aside

79
 Characteristic Functions

Definition: The characteristic function of a

real rv X is

φX (t) = E(eitX )
√
where i = −1 is the imaginary unit.

Since
eitX = cos(tX) + i sin(tX)
we find that

φX (t) = E(cos(tX)) + iE(sin(tX)) .

Since the trigonometric functions are bounded
by 1 the expected values must be finite for all
t.

This is precisely the reason for using character-

istic rather than moment generating functions
in probability theory courses.

80
Theorem 2 For any two real rvs X and Y the
following are equivalent:

1. X and Y have the same distribution, that

is, for any (Borel) set A we have

P (X ∈ A) = P (Y ∈ A) .

2. FX (t) = FY (t) for all t.

3. φX (t) = E(eitX ) = E(eitY ) = φY (t) for all

real t.

Moreover, all of these are implied if there is a

positive  such that for all |t| ≤ 

MX (t) = MY (t) < ∞ .

81
 Inversion

Previous theorem is non-constructive charac-

terization.

Can get from φX to FX or fX by inversion.

See homework for basic inversion formula:

If X is a random variable taking only integer

values then for each integer k
1 2π
Z
P (X = k) = φX (t)e−itk dt
2π Z0
1 π
= φX (t)e−itk dt .
2π −π
The proof proceeds from the formula

eiktP (X = k) .
X
φX (t) =
k

82
Now suppose X has continuous bounded den-
sity f . Define

Xn = [nX]/n
where [a] denotes the integer part (rounding
down to the next smallest integer). We have

P (k/n ≤ X < (k + 1)/n)

=P ([nX] = k)
1 π
Z
= φ[nX](t) × e−itk dt .
2π −π
Make the substitution t = u/n, and get
1
nP (k/n ≤ X < (k + 1)/n) =
Z nπ 2π
× φ[nX](u/n)e−iuk/ndu .
−nπ

83
Now, as n → ∞ we have

φ[nX](u/n) = E(eiu[nX]/n) → E(eiuX ) .

(Dominated convergence: |eiu| ≤ 1.)

Range of integration converges to the whole

real line.

If k/n → x left hand side converges to density

f (x) while right hand side converges to
1 ∞
Z
φX (u)e−iuxdu
2π −∞
which gives the inversion formula
1 ∞
Z
fX (x) = φX (u)e−iux du .
2π −∞
Many other such formulas are available to com-
pute things like F (b) − F (a) and so on.

All such formulas called Fourier inversion for-

mulas.

Characteristic ftn also called Fourier trans-

form of f or F .
84
 Inversion of the Moment Generating
Function

MGF and characteristic function related for-

mally:
MX (it) = φX (t) .
When MX exists this relationship is not merely
formal; the methods of complex variables mean
there is a “nice” (analytic) function which is
E(ezX ) for any complex z = x + iy for which
MX (x) is finite.

SO: there is an inversion formula for MX using

a complex contour integral:

If z1 and z2 are two points in the complex plane

and C a path between these two points we can
define the path integral
Z
f (z)dz
C
by the methods of line integration.

Do algebra with such integrals via usual theo-

rems of calculus.
85
The Fourier inversion formula was
Z ∞
2πf (x) = φ(t)e−itx dt
−∞
so replacing φ by M we get
Z ∞
2πf (x) = M (it)e−itxdt .
−∞
If we just substitute z = it then we find
Z
2πif (x) = M (z)e−zxdz
C
where the path C is the imaginary axis.

Complex contour integration: replace C by any

other path which starts and ends at the same
place.

Sometimes can choose path to make it easy

to do the integral approximately; this is what
saddlepoint approximations are.

Inversion formula is called the inverse Laplace

transform; the mgf is also called the Laplace
transform of f or F .
86
 Applications of Inversion

1): Numerical calculations

Example: Many statistics have a distribution

which is approximately that of

λj Zj2
X
T =
where the Zj are iid N (0, 1). In this case

itT
Y itλj Zj2
E(e )= E(e )
(1 − 2itλj )−1/2 .
Y
=
Imhof (Biometrika, 1961) gives a simplification
of the Fourier inversion formula for

FT (x) − FT (0)
which can be evaluated numerically:

FT (x) − FT (0)
Z x
= fT (y)dy
Z0x
1 ∞ Y
Z
= (1 − 2itλj )−1/2e−ity dtdy .
0 2π −∞

87
Multiply
" #1/2
1
φ(t) = Q
(1 − 2itλj )
top and bottom by the complex conjugate of
the denominator:
Q 1/2
(1 + 2itλj )
φ(t) = Q

2 2
 .
(1 + 4t λj )

The complex number 1 + 2itλj is rj eiθj where

r
rj = 1 + 4t4 λ2
j
and
tan(θj ) = 2tλj .

88
This allows us to rewrite
Q P 1/2
rj e i θj
φ(t) =  Q 2 
rj
or
tan−1(2tλj )/2
P
ei
φ(t) = Q .
(1 + 4t2λ2
j)
1/4

Assemble this to give

1 ∞ eiθ(t) x −iyt
Z Z
FT (x) − FT (0) = e dydt
2π −∞ ρ(t) 0
where
tan−1(2tλj )/2
X
θ(t) =

and ρ(t) = (1 + 4t2 λ2

j)
1/4 . But
Q

e−ixt − 1
Z x
−iyt
e dy = .
0 −it
We can now collect up the real part of the
resulting integral to derive the formula given
by Imhof. I don’t produce the details here.

89
2): The central limit theorem (in some ver-
sions) can be deduced from the Fourier inver-
sion formula: if X1 , . . . , Xn are iid with mean
0 and variance 1 and T = n1/2X̄ then with φ
denoting the characteristic function of a single
X we have
in−1/2 t
P
itT Xj
E(e ) = E(e )
in
= φ(n−1/2 t)
h

#n
tφ0 (0) t2 φ00(0)
"
≈ φ(0) + √ + + o(n−1)
n 2n
But now φ(0) = 1 and
d
0
φ (t) = E(eitX1 ) = iE(X1 eitX1 ) .
dt
So φ0(0) = E(X1 ) = 0. Similarly
φ00(t) = i2E(X12 eitX1 )
so that
φ00(0) = −E(X12) = −1 .
It now follows that
E(eitT ) ≈ [1 − t2/(2n) + o(1/n)]n
−t 2 /2
→e .

90
With care we can then apply the Fourier inver-
sion formula and get
1 ∞ −itx
Z
fT (x) = e [φ(tn−1/2 )]ndt
2π Z−∞
1 ∞ −itx −t2/2
→ e e dt
2π −∞
1
= √ φZ (−x)
2π
where φZ is the characteristic function of a
standard normal variable Z. Doing the integral
we find
2
φZ (x) = φZ (−x) = e−x /2
so that
1 2
fT (x) → √ e−x /2
2π
which is a standard normal density.

91
Proof of the central limit theorem not general:
requires T to have bounded continuous density.

Central limit theorem: statement about cdfs

not densities:

P (T ≤ t) → P (Z ≤ t) .

3) Saddlepoint approximation from MGF in-

version formula
Z i∞
2πif (x) = M (z)e−zx dz
−i∞
(limits of integration indicate contour integral
running up imaginary axis.)

Replace contour (using complex variables) with

line Re(z) = c. (Re(Z) denotes the real part
of z, that is, x when z = x + iy with x and
y real.) Must choose c so that M (c) < ∞.
Rewrite inversion formula using cumulant gen-
erating function K(t) = log(M (t)):
Z c+i∞
2πif (x) = exp(K(z) − zx)dz .
c−i∞

92
Along the contour in question we have z =
c + iy so we can think of the integral as being
Z ∞
i exp(K(c + iy) − (c + iy)x)dy .
−∞
Now do a Taylor expansion of the exponent:

K(c + iy) − (c + iy)x =

K(c) − cx + iy(K 0(c) − x) − y 2K 00(c)/2 + · · · .
Ignore the higher order terms and select a c so
that the first derivative
K 0(c) − x
vanishes. Such a c is a saddlepoint. We get
the formula

2πf (x) ≈ exp(K(c) − cx)

Z ∞
× exp(−y 2K 00(c)/2)dy .
−∞
Integral is normal density calculation; gives
q
2π/K 00(c) .
Saddlepoint approximation is
exp(K(c) − cx)
f (x) = q .
2πK 00(c)

93
Essentially same idea: Laplace’s approxima-
tion.

Example: Sterling’s approximation to facto-

rial:
Z ∞
n! = exp(n log(x) − x)dx .
0
Exponent maximized when x = n.

For n large approximate f (x) = n log(x) − x by

f (x) ≈ f (x0 )+(x−x0)f 0 (x0)+(x−x0)2 f 00(x0 )/2
and choose x0 = n to make f 0(x0) = 0. Then
Z ∞
n! ≈ exp[n log(n) − n − (x − n)2/(2n)]dx .
0
√
Substitute y = (x − n)/ n; get approximation
Z ∞
2
n! ≈ n1/2nne−n e−y /2dy
−∞
or
√
n! ≈ 2πnn+1/2e−n .
Note: sloppy about limits of integration.

Real proof must show integral over x not near

n is negligible.
94
 Convergence in Distribution

Undergraduate version of central limit theo-

rem: if X1 , . . . , Xn are iid from a population
with mean µ and standard deviation σ then
n1/2(X̄ − µ)/σ has approximately a normal dis-
tribution.

Also Binomial(n, p) random variable has ap-

proximately a N (np, np(1 − p)) distribution.

Precise meaning of statements like “X and Y

have approximately the same distribution”?

Desired meaning: X and Y have nearly the

same cdf.

But care needed.

95
Q1) If n is a large number is the N (0, 1/n)
distribution close to the distribution of X ≡ 0?

Q2) Is N (0, 1/n) close to the N (1/n, 1/n) dis-

tribution?

√
Q3) Is N (0, 1/n) close to N (1/ n, 1/n) distri-
bution?

Q4) If Xn ≡ 2−n is the distribution of Xn close

to that of X ≡ 0?

96
Answers depend on how close close needs to
be so it’s a matter of definition.

In practice the usual sort of approximation we

want to make is to say that some random vari-
able X, say, has nearly some continuous distri-
bution, like N (0, 1).

So: want to know probabilities like P (X > x)

are nearly P (N (0, 1) > x).

Real difficulty: case of discrete random vari-

ables or infinite dimensions: not done in this
course.

Mathematicians’ meaning of close:

Either they can provide an upper bound on the

distance between the two things or they are
talking about taking a limit.

In this course we take limits.

97
Definition: A sequence of random variables
Xn converges in distribution to a random vari-
able X if
E(g(Xn )) → E(g(X))
for every bounded continuous function g.

Theorem 3 The following are equivalent:

1. Xn converges in distribution to X.

2. P (Xn ≤ x) → P (X ≤ x) for each x such

that P (X = x) = 0.

3. The limit of the characteristic functions of

Xn is the characteristic function of X:
E(eitXn ) → E(eitX )
for every real t.

These are all implied by

MXn (t) → MX (t) < ∞
for all |t| ≤  for some positive .
98
Now let’s go back to the questions I asked:

• Xn ∼ N (0, 1/n) and X = 0. Then


 1
x>0

P (Xn ≤ x) → 0 x<0
 1/2 x = 0


Now the limit is the cdf of X = 0 except for

x = 0 and the cdf of X is not continuous
at x = 0 so yes, Xn converges to X in
distribution.

• I asked if Xn ∼ N (1/n, 1/n) had a distribu-

tion close to that of Yn ∼ N (0, 1/n). The
definition I gave really requires me to an-
swer by finding a limit X and proving that
both Xn and Yn converge to X in distribu-
tion. Take X = 0. Then
tXn t/n+t2/(2n)
E(e )=e → 1 = E(etX )
and
2
E(etYn ) = et /(2n) → 1
so that both Xn and Yn have the same limit
in distribution.

99
 N(0,1/n) vs X=0; n=10000
1.0

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
X=0
0.8

N(0,1/n)
0.6
0.4
0.2
0.0

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

-3 -2 -1 0 1 2 3

N(0,1/n) vs X=0; n=10000

1.0

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
X=0
0.8

N(0,1/n)
0.6
0.4
0.2
0.0

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

-0.03 -0.02 -0.01 0.0 0.01 0.02 0.03

100
 N(1/n,1/n) vs N(0,1/n); n=10000
1.0

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
N(0,1/n) •••
••
••
0.8

N(1/n,1/n)
•
•
•
•
0.6

•
•
•
•
•
0.4

•
•
•
•
0.2

•
••
•••
••
••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
0.0

-3 -2 -1 0 1 2 3

N(1/n,1/n) vs N(0,1/n); n=10000

1.0

•••••••••••••••••••••••••••••••••••••••••••••••••••••
•••••••••••••••••••••••••••••
N(0,1/n) ••
••
•
••
••••••••••••••••
•
•
••••••••
••••••••
0.8

N(1/n,1/n)
••
••
•••••••••
•
•••••
••••••
•
••
••••••
0.6

•
••••••
•••••
••
••••••
•
••••
•••••
0.4

••
•••••••
•
•••••
•••••
•
••
•
•••••••
0.2

•
•••••••
•••••••••
••
••
•
••
••••••••••
•
•
•••••
•••••••••••••••••••
•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
0.0

-0.03 -0.02 -0.01 0.0 0.01 0.02 0.03

101
• Multiply both Xn and Yn by n1/2 and let
√
X ∼ N (0, 1). Then nXn ∼ N (n−1/2, 1)
√
and nYn ∼ N (0, 1). Use characteristic
√
functions to prove that both nXn and
√
nYn converge to N (0, 1) in distribution.

• If you now let Xn ∼ N (n−1/2, 1/n) and Yn ∼

N (0, 1/n) then again both Xn and Yn con-
verge to 0 in distribution.

• If you multiply Xn and Yn in the previ-

ous point by n1/2 then n1/2Xn ∼ N (1, 1)
and n1/2Yn ∼ N (0, 1) so that n1/2Xn and
n1/2 Yn are not close together in distribu-
tion.

• You can check that 2−n → 0 in distribution.

102
 N(1/sqrt(n),1/n) vs N(0,1/n); n=10000
1.0

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
N(0,1/n) •••
••
••
0.8

N(1/sqrt(n),1/n)
•
•
•
•
0.6

•
•
•
•
•
0.4

•
•
•
•
0.2

•
••
•••
••
••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
0.0

-3 -2 -1 0 1 2 3

N(1/sqrt(n),1/n) vs N(0,1/n); n=10000

1.0

•••••••••••••••••••••••••••••••••••••••••••••••••••••
•••••••••••••••••••••••••••••
N(0,1/n) ••
••
•
••
••••••••••••••••
•
•
••••••••
••••••••
0.8

N(1/sqrt(n),1/n)
••
••
•••••••••
•
•••••
••••••
•
••
••••••
0.6

•
••••••
•••••
••
••••••
•
••••
•••••
0.4

••
•••••••
•
•••••
•••••
•
••
•
•••••••
0.2

•
•••••••
•••••••••
••
••
•
••
••••••••••
•
•
•••••
•••••••••••••••••••
•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
0.0

-0.03 -0.02 -0.01 0.0 0.01 0.02 0.03

103
Summary: to derive approximate distributions:

Show sequence of rvs Xn converges to some

X.

The limit distribution (i.e. dstbon of X) should

be non-trivial, like say N (0, 1).

Don’t say: Xn is approximately N (1/n, 1/n).

Do say: n1/2(Xn − 1/n) converges to N (0, 1)

in distribution.

The Central Limit Theorem

If X1, X2, · · · are iid with mean 0 and vari-

ance 1 then n1/2X̄ converges in distribution
to N (0, 1). That is,
1 x Z
1/2 √ −y 2/2
P (n X̄ ≤ x) → e dy .
2π −∞

Proof: As before
itn1/2 X̄ −t2/2
E(e )→e .
This is the characteristic function of N (0, 1)
so we are done by our theorem.
104
Edgeworth expansions

In fact if γ = E(X 3 ) then

φ(t) ≈ 1 − t2/2 − iγt3 /6 + · · ·

keeping one more term. Then

log(φ(t)) = log(1 + u)
where

u = −t2/2 − iγt3 /6 + · · · .
Use log(1 + u) = u − u2/2 + · · · to get

log(φ(t)) ≈
[−t2/2 − iγt3 /6 + · · · ]
− [· · · ]2/2 + · · ·
which rearranged is

log(φ(t)) ≈ −t2/2 − iγt3 /6 + · · · .

105
Now apply this calculation to

log(φT (t)) ≈ −t2 /2 − iE(T 3 )t3 /6 + · · · .

3 √
Remember E(T ) = γ/ n and exponentiate to
get
−t 2 /2 3 √
φT (t) ≈ e exp{−iγt /(6 n) + · · · } .
You can do a Taylor expansion of the second
exponential around 0 because of the square
root of n and get
−t2/2 3 √
φT (t) ≈ e (1 − iγt /(6 n))
neglecting higher order terms. This approxi-
mation to the characteristic function of T can
be inverted to get an Edgeworth approxima-
tion to the density (or distribution) of T which
looks like
1 −x2 /2 3 √
fT (x) ≈ √ e [1−γ(x −3x)/(6 n)+· · · ] .
2π

106
Remarks:

1. The error using the central limit theorem

to approximate a density or a probability is
proportional to n−1/2.

2. This is improved to n−1 for symmetric den-

sities for which γ = 0.

3. These expansions are asymptotic. This

means that the series indicated by · · · usu-
ally does not converge. When n = 25 it
may help to take the second term but get
worse if you include the third or fourth or
more.

4. You can integrate the expansion above for

the density to get an approximation for the
cdf.

107
Multivariate convergence in distribution

Definition: Xn ∈ Rp converges in distribution

to X ∈ Rp if

E(g(Xn )) → E(g(X))
for each bounded continuous real valued func-
tion g on Rp.

This is equivalent to either of

Cramér Wold Device: atXn converges in dis-

tribution to atX for each a ∈ Rp.

or

Convergence of characteristic functions:

t t
E(eia Xn ) → E(eia X )
for each a ∈ Rp.

108
Extensions of the CLT

1. Y1, Y2, · · · iid in Rp, mean µ, variance co-

variance Σ then n1/2(Ȳ − µ) converges in
distribution to M V N (0, Σ).

2. Lyapunov CLT: for each n Xn1, . . . , Xnn in-

dependent rvs with
E(Xni ) = 0
X
Var( Xni) = 1
i
E(|Xni|3) → 0
X

then i Xni converges to N (0, 1).

P

3. Lindeberg CLT: 1st two conds of Lyapunov

and
2
X
E(Xni 1(|Xni | > )) → 0
each  > 0. Then i Xni converges in dis-
P

tribution to N (0, 1). (Lyapunov’s condition

implies Lindeberg’s.)

4. Non-independent rvs: m-dependent CLT,

martingale CLT, CLT for mixing processes.

5. Not sums: Slutsky’s theorem, δ method.

109
Slutsky’s Theorem: If Xn converges in dis-
tribution to X and Yn converges in distribu-
tion (or in probability) to c, a constant, then
Xn + Yn converges in distribution to X + c.
More generally, if f (x, y) is continuous then
f (Xn , Yn) ⇒ f (X, c).

Warning: the hypothesis that the limit of Yn

be constant is essential.

Definition: We say Yn converges to Y in prob-

ability if ∀ > 0:
P (|Yn − Y | > ) → 0 .
Fact: for Y constant convergence in distribu-
tion and in probability are the same.

Always convergence in probability implies con-

vergence in distribution.

Both are weaker than almost sure convergence:

Definition: We say Yn converges to Y almost

surely if
P ({ω ∈ Ω : lim Yn(ω) = Y (ω)}) = 1 .
n→∞

110
The delta method: Suppose:

• Sequence Yn of rvs converges to some y, a

constant.

• Xn = an (Yn − y) then Xn converges in dis-

tribution to some random variable X.

• f is differentiable ftn on range of Yn.

Then an(f (Yn )−f (y)) converges in distribution

to f 0(y)X.

If Xn ∈ Rp and f : Rp 7→ Rq then f 0 is q × p
matrix of first derivatives of components of f .

Example: Suppose X1 , . . . , Xn are a sample

from a population with mean µ, variance σ 2 ,
and third and fourth central moments µ3 and
µ4. Then
n1/2(s2 − σ 2 ) ⇒ N (0, µ4 − σ 4 )
where ⇒ is notation for convergence in distri-
bution. For simplicity I define s2 = X 2 − X̄ 2 .
111
How to apply δ method:

1) Write statistic as a function of averages:

Define
Xi2
" #
Wi = .
Xi
See that
" #
X2
W̄n =
X
Define
f (x1 , x2) = x1 − x2
2
See that s2 = f (W̄n).

2) Compute mean of your averages:

E(Xi2 ) µ2 + σ 2
" # " #
µW ≡ E(W̄n) = = .
E(Xi) µ

3) In δ method theorem take Yn = W̄n and

y = E(Yn).
112
4) Take an = n1/2.

5) Use central limit theorem:

n1/2(Yn − y) ⇒ M V N (0, Σ)
where Σ = Var(Wi).

6) To compute Σ take expected value of

(W − µW )(W − µW )t
There are 4 entries in this matrix. Top left
entry is
(X 2 − µ2 − σ 2)2
This has expectation:

E (X 2 − µ2 − σ 2 )2 = E(X 4) − (µ2 + σ 2 )2 .
n o

113
Using binomial expansion:

E(X 4 ) = E{(X − µ + µ)4 }

= µ4 + 4µµ3 + 6µ2 σ 2
+ 4µ3E(X − µ) + µ4 .
So
Σ11 = µ4 − σ 4 + 4µµ3 + 4µ2σ 2

Top right entry is expectation of

(X 2 − µ2 − σ 2)(X − µ)
which is
E(X 3 ) − µE(X 2 )
Similar to 4th moment get

µ3 + 2µσ 2

Lower right entry is σ 2 .

So
µ4 − σ 4 + 4µµ3 + 4µ2σ 2 µ3 + 2µσ 2
" #
Σ=
µ3 + 2µσ 2 σ2

114
7) Compute derivative (gradient) of f : has
components (1, −2x2). Evaluate at y = (µ2 +
σ 2 , µ) to get

at = (1, −2µ) .

This leads to

n1/2(s2 − σ 2 ) ≈
" #
X 2 − (µ2 + σ 2 )
n1/2[1, −2µ]
X̄ − µ
which converges in distribution to

(1, −2µ)M V N (0, Σ) .

This rv is N (0, at Σa) = N (0, µ4 − σ 4).

115
Alternative approach worth pursuing. Suppose
c is constant.

Define Xi∗ = Xi − c.

Then: sample variance of Xi∗ is same as sample

variance of Xi.

Notice all central moments of Xi∗ same as for

Xi. Conclusion: no loss in µ = 0. In this case:

at = (1, 0)
and
µ4 − σ 4
" #
µ3
Σ= .
µ3 σ2

Notice that

at Σ = [µ4 − σ 4, µ3]
and
atΣa = µ4 − σ 4 .

116
Special case: if population is N (µ, σ 2) then
µ3 = 0 and µ4 = 3σ 4 . Our calculation has

n1/2(s2 − σ 2) ⇒ N (0, 2σ 4 )
You can divide through by σ 2 and get
s 2
n1/2( 2 − 1) ⇒ N (0, 2)
σ
In fact ns2/σ 2 has a χ2n−1 distribution and so
the usual central limit theorem shows that

(n − 1)−1/2 [ns2/σ 2 − (n − 1)] ⇒ N (0, 2)

(using mean of χ2
1 is 1 and variance is 2).

Factor out n to get

s
n
n1/2(s2 /σ 2 − 1) + (n − 1)−1/2 ⇒ N (0, 2)
n−1
which is δ method calculation except for some
constants.

Difference is unimportant: Slutsky’s theorem.

117
 Monte Carlo

Given rvs X1 , . . . , Xn; distbn specified.

Statistic T (X1 , . . . , Xn) whose dstbn wanted.

To compute P (T > t):

1. Generate X1 , . . . , Xn from the density f .

2. Compute T1 = T (X1 , . . . , Xn).

3. Repeat N times getting T1, . . . , TN .

4. Estimate p = P (T > t) as p̂ = M/N where

M is number of repetitions where Ti > t.
q
5. Estimate accuracy of p̂ using p̂(1 − p̂)/N .

√
Note: accuracy inversely proportional to N.

Next: tricks to make method more accurate.

Warning: tricks only change √constant — SE
still inversely proportional to N .
118
 Generating the Sample

Transformation

Basic computing tool: pseudo uniform random

numbers — variables U which have (approxi-
mately) a Uniform[0, 1] distribution.

Other dstbns generated by transformation:

Exponential: X = − log U has an exponential

distribution:

P (X > x) = P (− log(U ) > x)

= P (U ≤ e−x) = e−x

119
Pitfall: Random uniforms generated on com-
puter sometimes have only 6 or 7 digits.

Consequence: tail of generated dstbn grainy.

Explanation: suppose U multiple of 10−6 .

Largest possible value of X is 6 log(10).

Improved algorithm:

• Generate U a Uniform[0,1] variable.

• Pick a small  like 10−3 say. If U >  take

Y = − log(U ).

• If U ≤ : conditional dstbn of Y − y given

Y > y is exponential. Generate new U 0.
Compute Y 0 = − log(U 0). Take Y = Y 0 −
log().

Resulting Y has exponential distribution.

Exercise: check by computing P (Y > y).

120
General technique: inverse probability integral
transform.

If Y is to have cdf F :

Generate U ∼ U nif orm[0, 1].

Take Y = F −1 (U ):
P (Y ≤ y) = P (F −1 (U ) ≤ y)
= P (U ≤ F (y)) = F (y)

Example: X exponential. F (x) = 1 − e−x and

F −1 (u) = − log(1 − u).

Compare to previous method. (Use U instead

of 1 − U .)

Normal: F = Φ (common notation for stan-

dard normal cdf).

No closed form for F −1 .

One way: use numerical algorithm to compute

F −1 .
121
Alternative: Box Müller

Generate U1, U2 two independent Uniform[0,1]

variables.

Define
q
Y1 = −2 log(U1 ) cos(2πU2 )
and
q
Y2 = −2 log(U1) sin(2πU2 ) .

Exercise: (use change of variables) Y1 and Y2

are independent N (0, 1) variables.

122
Acceptance Rejection

Suppose: can’t calculate F −1 but know f .

Find density g and constant c such that

1) f (x) ≤ cg(x) for each x and

2) G−1 is computable or can generate obser-

vations W1, W2, . . . independently from g.

Algorithm:

1) Generate W1.

2) Compute p = f (W1 )/(cg(W1)) ≤ 1.

3) Generate uniform[0,1] random variable U1

independent of all W s.

4) Let Y = W1 if U1 ≤ p.

5) Otherwise get new W, U ; repeat until you

find Ui ≤ f (Wi )/(cg(Wi)).

6) Make Y be last W generated.

This Y has density f .

123
Markov Chain Monte Carlo

Recently popular tactic, particularly for gener-

ating multivariate observations.

Theorem Suppose W1, W2, . . . is an (ergodic)

Markov chain with stationary transitions and
the stationary initial distribution of W has den-
sity f . Then starting the chain with any initial
distribution
n
1 X
Z
g(Wi) → g(x)f (x)dx .
n i=1

R
Estimate things like A f (x)dx by computing
the fraction of the Wi which land in A.

Many versions of this technique including Gibbs

Sampling and Metropolis-Hastings algorithm.

Technique invented in 1950s: Metropolis et al.

One of the authors was Edward Teller “father

of the hydrogen bomb”.
124
Importance Sampling

If you want to compute

Z
θ ≡ E(T (X)) = T (x)f (x)dx

you can generate observations from a different

density g and then compute
−1
X
θ̂ = n T (Xi )f (Xi )/g(Xi)
Then
−1
X
E(θ̂) = n E {T (Xi )f (Xi )/g(Xi)}
Z
= {T (x)f (x)/g(x)}g(x)dx
Z
= T (x)f (x)dx
=θ

125
Variance reduction

Example: estimate dstbn of sample mean for

a Cauchy random variable.

Cauchy density is
1
f (x) =
π(1 + x2)
Generate U1, . . . , Un uniforms.

Define Xi = tan−1(π(Ui − 1/2)).

Compute T = X̄.

To estimate p = P (T > t) use

N
X
p̂ = 1(Ti > t)/N
i=1
after generating N samples of size n.

Estimate is unbiased.
q
Standard error is p(1 − p)/N .
126
Improvement: −Xi also has Cauchy dstbn.

Take Si = −Ti.

Remember that Si has same dstbn as Ti.

Try (for t > 0)

N
X N
X
p̃ = [ 1(Ti > t) + 1(Si > t)]/(2N )
i=1 i=1
which is the average of two estimates like p̂.

The variance of p̃ is

(4N )−1 Var(1(Ti > t) + 1(Si > t))

= (4N )−1 Var(1(|T | > t))
which is
2p(1 − 2p) p(1 − 2p)
=
4N 2N

Variance has extra 2 in denominator and nu-

merator is also smaller – particularly for p near
1/2.

So need only half the sample size to get the

same accuracy.
127
Regression estimates

Suppose Z ∼ N (0, 1). Compute

θ = E(|Z|) .
Generate N iid N (0, 1) variables Z1 , . . . , ZN .

Compute θ̂ = |Zi|/N .
P

But know E(Zi2) = 1.

Also: θ̂ is positively correlated with Zi2/N .

P

So we try
Zi2/N − 1)
X
θ̃ = θ̂ − c(
Notice that E(θ̃) = θ and

Var(θ̃) =
Zi2/N )
X
Var(θ̂) − 2cCov(θ̂,
+ c2Var( Zi2/N )
X

The value of c which minimizes this is

Cov(θ̂, Zi2/N )
P
c=
Var( Zi2/N )
P

Estimate c by regressing |Zi| on Zi2!

128
 Statistical Inference

Definition: A model is a family {Pθ ; θ ∈ Θ} of

possible distributions for some random variable
X.

WARNING: Data set is X, so X will generally

be a big vector or matrix or even more compli-
cated object.)

Assumption in this course: true distribution P

of X is Pθ0 for some θ0 ∈ Θ.

JARGON: θ0 is true value of the parameter.

Notice: this assumption is wrong; we hope it

is not wrong in an important way.

If it’s wrong: enlarge model, put in more dis-

tributions, make Θ bigger.

Goal: observe value of X, guess θ0 or some

property of θ0.
129
Classic mathematical versions of guessing:

1. Point estimation: compute estimate θ̂ =

θ̂(X) which lies in Θ (or something close
to Θ).

2. Point estimation of ftn of θ: compute es-

timate φ̂ = φ̂(X) of φ = g(θ).

3. Interval (or set) estimation: compute set

C = C(X) in Θ which we think will contain
θ0 .

4. Hypothesis testing: choose between θ0 ∈

Θ0 and θ0 6∈ Θ0 where Θ0 ⊂ Θ.

5. Prediction: guess value of an observable

random variable Y whose distribution de-
pends on θ0. Typically Y is the value of
the variable X in a repetition of the exper-
iment.

130
Several schools of statistical thinking. Main
schools of thought summarized roughly as fol-
lows:

• Neyman Pearson: A statistical procedure

is evaluated by its long run frequency per-
formance. Imagine repeating the data col-
lection exercise many times, independently.
Quality of procedure measured by its aver-
age performance when true distribution of
X values is Pθ0 .

• Bayes: Treat θ as random just like X.

Compute conditional law of unknown quan-
tities given knowns. In particular ask how
procedure will work on the data we actu-
ally got – no averaging over data we might
have got.

• Likelihood: Try to combine previous 2 by

looking only at actual data while trying to
avoid treating θ as random.

We use Neyman Pearson approach to evaluate

quality of likelihood and other methods.
131
 Likelihood Methods of Inference

Toss coin 6 times and get Heads twice.

p is probability of getting H.

Probability of getting exactly 2 heads is

15p2 (1 − p)4
This function of p, is likelihood function.

Definition: The likelihood function is map L:

domain Θ, values given by
L(θ) = fθ (X)

Key Point: think about how the density de-

pends on θ not about how it depends on X.

Notice: X, observed value of the data, has

been plugged into the formula for density.

Notice: coin tossing example uses the discrete

density for f .

We use likelihood for most inference problems:

132
1. Point estimation: we must compute an es-
timate θ̂ = θ̂(X) which lies in Θ. The max-
imum likelihood estimate (MLE) of θ
is the value θ̂ which maximizes L(θ) over
θ ∈ Θ if such a θ̂ exists.

2. Point estimation of a function of θ: we

must compute an estimate φ̂ = φ̂(X) of
φ = g(θ). We use φ̂ = g(θ̂) where θ̂ is the
MLE of θ.

3. Interval (or set) estimation. We must com-

pute a set C = C(X) in Θ which we think
will contain θ0 . We will use
{θ ∈ Θ : L(θ) > c}
for a suitable c.

4. Hypothesis testing: decide whether or not

θ0 ∈ Θ0 where Θ0 ⊂ Θ. We base our deci-
sion on the likelihood ratio
sup{L(θ); θ ∈ Θ \ Θ0}
.
sup{L(θ); θ ∈ Θ0}

133
Maximum Likelihood Estimation

To find MLE maximize L.

Typical function maximization problem:

Set gradient of L equal to 0

Check root is maximum, not minimum or sad-

dle point.

Examine some likelihood plots in examples:

Cauchy Data

IID sample X1 , . . . , Xn from Cauchy(θ) density

1
f (x; θ) =
π(1 + (x − θ)2)
The likelihood function is
n
Y 1
L(θ) = 2
i=1 π(1 + (Xi − θ) )

[Examine likelihood plots.]

134
 Likelihood Function: Cauchy, n=5 Likelihood Function: Cauchy, n=5
1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0
-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Function: Cauchy, n=5 Likelihood Function: Cauchy, n=5

1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Function: Cauchy, n=5 Likelihood Function: Cauchy, n=5

1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

135
 Likelihood Function: Cauchy, n=5 Likelihood Function: Cauchy, n=5
1.0

1.0
0.8
0.8

0.6
Likelihood

Likelihood
0.6

0.4
0.4

0.2
0.2

0.0
-2 -1 0 1 2 -2 -1 0 1 2

Theta Theta

Likelihood Function: Cauchy, n=5 Likelihood Function: Cauchy, n=5

1.0

1.0
0.8

0.8
0.6
0.6
Likelihood

Likelihood

0.4
0.4

0.2
0.2

0.0
0.0

-2 -1 0 1 2 -2 -1 0 1 2

Theta Theta

Likelihood Function: Cauchy, n=5 Likelihood Function: Cauchy, n=5

1.0

1.0
0.8
0.8

0.6
0.6
Likelihood

Likelihood

0.4
0.4

0.2
0.2

0.0
0.0

-2 -1 0 1 2 -2 -1 0 1 2

Theta Theta

136
 Likelihood Function: Cauchy, n=25 Likelihood Function: Cauchy, n=25
1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0
-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Function: Cauchy, n=25 Likelihood Function: Cauchy, n=25

1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Function: Cauchy, n=25 Likelihood Function: Cauchy, n=25

1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

137
 Likelihood Function: Cauchy, n=25 Likelihood Function: Cauchy, n=25
1.0

1.0
0.8
0.8

0.6
0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0
-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Theta Theta

Likelihood Function: Cauchy, n=25 Likelihood Function: Cauchy, n=25

1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0

-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Theta Theta

Likelihood Function: Cauchy, n=25 Likelihood Function: Cauchy, n=25

1.0

1.0
0.8

0.8
0.6

0.6
Likelihood

Likelihood
0.4

0.4
0.2

0.2
0.0

0.0

-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Theta Theta

138
I want you to notice the following points:

• The likelihood functions have peaks near

the true value of θ (which is 0 for the data
sets I generated).

• The peaks are narrower for the larger sam-

ple size.

• The peaks have a more regular shape for

the larger value of n.

• I actually plotted L(θ)/L(θ̂) which has ex-

actly the same shape as L but runs from 0
to 1 on the vertical scale.

139
To maximize this likelihood: differentiate L,
set result equal to 0.

Notice L is product of n terms; derivative is

n Y
X 1 2(Xi − θ)
2 2 2
i=1 j6=i π(1 + (Xj − θ) ) π(1 + (Xi − θ) )
which is quite unpleasant.

Much easier to work with logarithm of L: log

of product is sum and logarithm is monotone
increasing.

Definition: The Log Likelihood function is

`(θ) = log{L(θ)} .

For the Cauchy problem we have

log(1 + (Xi − θ)2) − n log(π)

X
`(θ) = −
[Examine log likelihood plots.]

140
 Likelihood Ratio Intervals: Cauchy, n=5 Likelihood Ratio Intervals: Cauchy, n=5
-12

-10
-14
Log Likelihood

Log Likelihood

-15
-16
-18

-20
-20
-22

-25
• • • • • •• •

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=5 Likelihood Ratio Intervals: Cauchy, n=5

-5
-10

-10
Log Likelihood

Log Likelihood
-15

-15
-20

-20

• • •• • •• • • •

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=5 Likelihood Ratio Intervals: Cauchy, n=5
-10
-12

-15
-14
Log Likelihood

Log Likelihood
-16
-18

-20
-20
-22

-25
-24

• • • •• • • • ••

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

141
 Likelihood Ratio Intervals: Cauchy, n=5 Likelihood Ratio Intervals: Cauchy, n=5
-11.0

-8
-11.5
-12.0
Log Likelihood

Log Likelihood

-10
-12.5

-12
-13.0
-13.5

-14
-2 -1 0 1 2 -2 -1 0 1 2

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=5 Likelihood Ratio Intervals: Cauchy, n=5
-6

-2
-7

-4
-8
Log Likelihood

Log Likelihood
-9

-6
-10

-8
-11
-12

-2 -1 0 1 2 -2 -1 0 1 2

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=5 Likelihood Ratio Intervals: Cauchy, n=5
-10

-12
-11

-13
Log Likelihood

Log Likelihood

-14
-12

-15
-13

-16
-14

-17

-2 -1 0 1 2 -2 -1 0 1 2

Theta Theta

142
 Likelihood Ratio Intervals: Cauchy, n=25 Likelihood Ratio Intervals: Cauchy, n=25

-20
-40

-40
Log Likelihood

Log Likelihood
-60

-60
-80

-80
-100

-100
••• • •• ••••• ••• ••• • • • • • • •• • •••••••• •• • • •

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=25 Likelihood Ratio Intervals: Cauchy, n=25
-20

-40
-40

-60
Log Likelihood

Log Likelihood
-60

-80
-80

-100
-100

• •• ••• •••• •• • • •• • • • • • • • • • ••••• ••• • • • • • • •

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=25 Likelihood Ratio Intervals: Cauchy, n=25
-40
-60

-60
Log Likelihood

Log Likelihood
-80

-80
-100

-100
-120

• • •• • • •• ••• •• ••
• •• • • • • •• ••• • • ••• ••• •• • • • •• ••

-10 -5 0 5 10 -10 -5 0 5 10

Theta Theta

143
 Likelihood Ratio Intervals: Cauchy, n=25 Likelihood Ratio Intervals: Cauchy, n=25

-22
-24

-24
-26
Log Likelihood

Log Likelihood

-26
-28

-28
-30

-30
-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=25 Likelihood Ratio Intervals: Cauchy, n=25

-36
-22

-38
Log Likelihood

Log Likelihood
-24

-40
-26

-42
-44
-28

-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Theta Theta

Likelihood Ratio Intervals: Cauchy, n=25 Likelihood Ratio Intervals: Cauchy, n=25
-43
-46

-44
-48

-45
Log Likelihood

Log Likelihood
-50

-46
-52

-47
-54

-48
-56

-49

-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Theta Theta

144
Notice the following points:

• Plots of ` for n = 25 quite smooth, rather

parabolic.

• For n = 5 many local maxima and minima

of `.

Likelihood tends to 0 as |θ| → ∞ so max of `

occurs at a root of `0, derivative of ` wrt θ.

Def’n: Score Function is gradient of `

∂`
U (θ) =
∂θ

MLE θ̂ usually root of Likelihood Equations

U (θ) = 0
In our Cauchy example we find
X 2(Xi − θ)
U (θ) =
1 + (Xi − θ)2

[Examine plots of score functions.]

Notice: often multiple roots of likelihood equa-

tions.
145
 Score Log Likelihood Score Log Likelihood

-4 -2 0 2 -25 -20 -15 -10 -2 -1 0 1 2 -22 -18 -14

-10

-10

-10

-10
-5

-5

-5

-5
Theta

Theta

Theta

Theta
0

0
5

5
10

10

10

10
Score Log Likelihood Score Log Likelihood

-4 -2 0 2 4 -20 -15 -10 -5 -3 -2 -1 0 1 2 3 -20 -15 -10

-10

-10

-10

-10
-5

-5

-5

-5
Theta

Theta

Theta

Theta
0

0
5

5
10

10

10

10

Score Log Likelihood Score Log Likelihood

-2 -1 0 1 2 3 -25 -20 -15 -2 -1 0 1 2 -24 -20 -16 -12

-10

-10

-10

-10
-5

-5

-5

-5
Theta

Theta

Theta

Theta
0

0
5

5
10

10

10

10

146
 Score Log Likelihood Score Log Likelihood

-15 -5 0 5 10 15 -100 -80 -60 -40 -20 -15 -10 -5 0 5 10 -100 -80 -60 -40
-10

-10

-10

-10
-5

-5

-5

-5
Theta

Theta

Theta

Theta
0

0
5

5
10

10

10

10
Score Log Likelihood Score Log Likelihood

-10 -5 0 5 10 -100 -80 -60 -40 -15 -5 0 5 10 15 -100 -80 -60 -40 -20
-10

-10

-10

-10
-5

-5

-5

-5
Theta

Theta

Theta

Theta
0

0
5

5
10

10

10

10

Score Log Likelihood Score Log Likelihood

-10 -5 0 5 10 -100 -80 -60 -40 -15 -10 -5 0 5 10 -120 -100 -80 -60
-10

-10

-10

-10
-5

-5

-5

-5
Theta

Theta

Theta

Theta
0

0
5

5
10

10

10

10

147
Example : X ∼ Binomial(n, θ)
!
n X
L(θ) = θ (1 − θ)n−X
X
!
n
`(θ) = log + X log(θ) + (n − X) log(1 − θ)
X
X n−X
U (θ) = −
θ 1−θ
The function L is 0 at θ = 0 and at θ = 1
unless X = 0 or X = n so for 1 ≤ X ≤ n
the MLE must be found by setting U = 0 and
getting
X
θ̂ =
n
For X = n the log-likelihood has derivative
n
U (θ) = > 0
θ
for all θ so that the likelihood is an increasing
function of θ which is maximized at θ̂ = 1 =
X/n. Similarly when X = 0 the maximum is at
θ̂ = 0 = X/n.

148
The Normal Distribution

Now we have X1 , . . . , Xn iid N (µ, σ 2). There

are two parameters θ = (µ, σ). We find
(Xi−µ)2/(2σ 2)
P
e−
L(µ, σ) =
(2π)n/2 σ n
n (Xi − µ)2
P
`(µ, σ) = − log(2π) − 2
− n log(σ)
2 2σ
and that U is
 P 
(Xi−µ)
 P σ2 
(Xi−µ) 2
n
 
σ3
−σ
Notice that U is a function with two compo-
nents because θ has two components.

Setting the likelihood equal to 0 and solving

gives
µ̂ = X̄
and
sP
(Xi − X̄)2
σ̂ =
n
149
Check this is maximum by computing one more
derivative. Matrix H of second derivatives of `
is
 P 
−n −2 (Xi−µ)
2 σ3
 −2 Pσ(X −µ)
 
2
P
i −3 (Xi−µ) n

3 + 2
σ σ4 σ
Plugging in the mle gives

−n

2 0
H(θ̂) =  σ̂ −2n

0 σ̂ 2
which is negative definite. Both its eigenvalues
are negative. So θ̂ must be a local maximum.

[Examine contour and perspective plots of `.]

150
 n=10

0 0.2 0.4 0.6 0.8 1

Z

40
30
40
20 30
Y 10 20
10 X

n=100
1
0 0.2 0.4 0.6 0.8
Z

40
30
40
20 30
Y 10 20
10 X

151
 2.0 n=10
Sigma

1.5
1.0

-1.0 -0.5 0.0 0.5 1.0

Mu

n=100
1.2
1.1
Sigma

1.0
0.9

-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2

Mu

152
Notice that the contours are quite ellipsoidal
for the larger sample size.

For X1 , . . . , Xn iid log likelihood is

X
`(θ) = log(f (Xi , θ)) .
The score function is
X ∂ log f
U (θ) = (Xi , θ) .
∂θ
MLE θ̂ maximizes `. If maximum occurs in
interior of parameter space and the log likeli-
hood continuously differentiable then θ̂ solves
the likelihood equations

U (θ) = 0 .

Some examples concerning existence of roots:

153
 Solving U (θ) = 0: Examples

N(µ, σ 2)

Unique root of likelihood equations is a global

maximum.

[Remark: Suppose we called τ = σ 2 the pa-

rameter. Score function still has two compo-
nents: first component same as before but sec-
ond component is
∂ (Xi − µ)2 n
P
`= 2
−
∂τ 2τ 2τ
Setting the new likelihood equations equal to
0 still gives
τ̂ = σ̂ 2
General invariance (or equivariance) princi-
pal: If φ = g(θ) is some reparametrization of a
model (a one to one relabelling of the param-
eter values) then φ̂ = g(θ̂). Does not apply to
other estimators.]
154
Cauchy: location θ

At least 1 root of likelihood equations but often

several more. One root is a global maximum;
others, if they exist may be local minima or
maxima.

Binomial(n, θ)

If X = 0 or X = n: no root of likelihood equa-

tions; likelihood is monotone. Other values of
X: unique root, a global maximum. Global
maximum at θ̂ = X/n even if X = 0 or n.

155
The 2 parameter exponential

The density is
1 −(x−α)/β
f (x; α, β) = e 1(x > α)
β
Log-likelihood is −∞ for α > min{X1 , . . . , Xn}
and otherwise is
X
`(α, β) = −n log(β) − (Xi − α)/β
Increasing function of α till α reaches

α̂ = X(1) = min{X1, . . . , Xn}

which gives mle of α. Now plug in α̂ for α; get
so-called profile likelihood for β:
X
`profile(β) = −n log(β) − (Xi − X(1) )/β
Set β derivative equal to 0 to get
X
β̂ = (Xi − X(1) )/n
Notice mle θ̂ = (α̂, β̂) does not solve likelihood
equations; we had to look at the edge of the
possible parameter space. α is called a support
or truncation parameter. ML methods behave
oddly in problems with such parameters.
156
Three parameter Weibull

The density in question is

!γ−1
1 x−α
f (x; α, β, γ) =
β β
× exp[−{(x − α)/β}γ ]1(x > α)
Three likelihood equations:

Set β derivative equal to 0; get

i1/γ
γ
hX
β̂(α, γ) = (Xi − α) /n
where β̂(α, γ) indicates mle of β could be found
by finding the mles of the other two parameters
and then plugging in to the formula above.

157
It is not possible to find explicitly the remain-
ing two parameters; numerical methods are
needed.

However putting γ < 1 and letting α → X(1)

will make the log likelihood go to ∞.

MLE is not uniquely defined: any γ < 1 and

any β will do.

If the true value of γ is more than 1 then the

probability that there is a root of the likelihood
equations is high; in this case there must be
two more roots: a local maximum and a saddle
point! For a true value of γ > 1 the theory we
detail below applies to the local maximum and
not to the global maximum of the likelihood
equations.

158
Large Sample Theory

Study approximate behaviour of θ̂ by studying

the function U .

Notice U is sum of independent random vari-

ables.

Theorem: If Y1, Y2, . . . are iid with mean µ

then
Yi
P
→µ
n

Called law of large numbers. Strong law

Yi
P
P (lim = µ) = 1
n
and the weak law that
Yi
P
lim P (| − µ| > ) = 0
n
For iid Yi the stronger conclusion holds; for
our heuristics ignore differences between these
notions.
159
Now suppose θ0 is true value of θ. Then

U (θ)/n → µ(θ)
where
∂ log f
 
µ(θ) =Eθ0 (Xi , θ)
∂θ
∂ log f
Z
= (x, θ)f (x, θ0 )dx
∂θ

160
Example: N (µ, 1) data:
X
U (µ)/n = (Xi − µ)/n = X̄ − µ
If the true mean is µ0 then X̄ → µ0 and
U (µ)/n → µ0 − µ

Consider µ < µ0: derivative of `(µ) is likely to

be positive so that ` increases as µ increases.

For µ > µ0: derivative is probably negative and

so ` tends to be decreasing for µ > 0.

Hence: ` is likely to be maximized close to µ0.

Repeat ideas for more general case. Study rv

log[f (Xi , θ)/f (Xi , θ0 )].
You know the inequality
E(X)2 ≤ E(X 2 )
(difference is Var(X) ≥ 0.)

Generalization: Jensen’s inequality: for g a

convex function (g 00 ≥ 0 roughly) then
g(E(X)) ≤ E(g(X))
161
Inequality above has g(x) = x2. Use g(x) =
− log(x): convex because g 00(x) = x−2 > 0.
We get

− log(Eθ0 [f (Xi , θ)/f (Xi , θ0 )]

≤ Eθ0 [− log{f (Xi , θ)/f (Xi , θ0 )}]
But
" #
f (Xi , θ) f (x, θ)
Z
Eθ0 = f (x, θ0 )dx
f (Xi , θ0) f (x, θ0)
Z
= f (x, θ)dx
=1
We can reassemble the inequality and this cal-
culation to get

Eθ0 [log{f (Xi , θ)/f (Xi , θ0)}] ≤ 0

162
Fact: inequality is strict unless the θ and θ0
densities are actually the same.

Let µ(θ) < 0 be this expected value.

Then for each θ we find

`(θ) − `(θ0)
n
log[f (Xi , θ)/f (Xi , θ0)]
P
=
n
→ µ(θ)
This proves likelihood probably higher at θ0
than at any other single θ.

Idea can often be stretched to prove that the

mle is consistent; need uniform convergence
in θ.

163
Definition A sequence θ̂n of estimators of θ is
consistent if θ̂n converges weakly (or strongly)
to θ.

Proto theorem: In regular problems the mle

θ̂ is consistent.

More precise statements of possible conclu-

sions. Use notation
N () = {θ : |θ − θ0 | ≤ } .

Suppose:

θ̂n is global maximizer of `.

θ̂n,δ maximizes ` over N (δ) = {|θ − θ0| ≤ δ}.

A = {|θ̂n − θ0| ≤ }

Bδ, = {|θ̂n,δ − θ0| ≤ }

CL = {∃!θ ∈ N (L/n1/2 ) : U (θ) = 0, U 0(θ) < 0}

164
Theorem:

1. Under conditions I P (A ) → 1 for each  >

0.

2. Under conditions II there is a δ > 0 such

that for all  > 0 we have P (Bδ, ) → 1.

3. Under conditions III for all δ > 0 there is

an L so large and an n0 so large that for
all n ≥ n0, P (CL ) > 1 − δ.

4. Under conditions III there is a sequence Ln

tending to ∞ so slowly that P (CLn ) → 1.

Point: conditions get weaker as conclusions

get weaker. Many possible conditions in lit-
erature. See book by Zacks for some precise
conditions.
165
 Asymptotic Normality

Study shape of log likelihood near the true

value of θ.

Assume θ̂ is a root of the likelihood equations

close to θ0.

Taylor expansion (1 dimensional parameter θ):

U (θ̂) =0
=U (θ0) + U 0(θ0 )(θ̂ − θ0)
+ U 00(θ̃)(θ̂ − θ0)2 /2
for some θ̃ between θ0 and θ̂.

WARNING: This form of the remainder in Tay-

lor’s theorem is not valid for multivariate θ.

166
Derivatives of U are sums of n terms.

So each derivative should be proportional to n

in size.

Second derivative is multiplied by the square of

the small number θ̂ − θ0 so should be negligible
compared to the first derivative term.

Ignoring second derivative term get

−U 0(θ0 )(θ̂ − θ0) ≈ U (θ0)

Now look at terms U and U 0.

167
Normal case:
X
U (θ0) = (Xi − µ0)
has a normal distribution with mean 0 and vari-
√
ance n (SD n).

Derivative is

U 0(µ) = −n .

Next derivative U 00 is 0.

Notice: both U and U 0 are sums of iid random

variables.

Let
∂ log f
Ui = (Xi , θ0)
∂θ
and
∂ 2 log f
Vi = − 2
(Xi, θ)
∂θ

168
In general, U (θ0) = Ui has mean 0 and ap-
P

proximately a normal distribution.

Here is how we check that:

Eθ0 (U (θ0)) = nEθ0 (U1)

∂ log(f (x, θ0))
Z
=n f (x, θ0 )dx
∂θ
∂f (x, θ0)/∂θ
Z
=n f (x, θ0)dx
f (x, θ0)
∂f
Z
=n (x, θ0)dx
∂θ
∂
Z 

=n f (x, θ)dx
∂θ θ=θ0
∂
=n 1
∂θ
=0

169
Notice: interchanged order of differentiation
and integration at one point.

This step is usually justified by applying the

dominated convergence theorem to the defini-
tion of the derivative.

Differentiate identity just proved:

∂ log f
Z
(x, θ)f (x, θ)dx = 0
∂θ
Take derivative of both sides wrt θ; pull deriva-
tive under integral sign:
∂ ∂ log f
Z  
(x, θ)f (x, θ) dx = 0
∂θ ∂θ
Do the derivative and get

∂ 2 log(f )
Z
− 2
f (x, θ)dx
∂θ Z
∂ log f ∂f
= (x, θ) (x, θ)dx
∂θ ∂θ
2
∂ log f
Z 
= (x, θ) f (x, θ)dx
∂θ

170
Definition: The Fisher Information is
I(θ) = −Eθ (U 0(θ)) = nEθ0 (V1 )
We refer to I(θ0) = Eθ0 (V1) as the information
in 1 observation.

The idea is that I is a measure of how curved

the log likelihood tends to be at the true value
of θ. Big curvature means precise estimates.
Our identity above is
I(θ) = V arθ (U (θ)) = nI(θ)

Now we return to our Taylor expansion approx-

imation
−U 0(θ0 )(θ̂ − θ0) ≈ U (θ0)
and study the two appearances of U .

We have shown that U = Ui is a sum of iid

P

mean 0 random variables. The central limit

theorem thus proves that
n−1/2U (θ0) ⇒ N (0, σ 2 )
where σ 2 = Var(Ui) = E(Vi) = I(θ).
171
Next observe that
0
X
−U (θ) = Vi
where again
∂Ui
Vi = −
∂θ
The law of large numbers can be applied to
show

−U 0(θ0)/n → Eθ0 [V1 ] = I(θ0 )

Now manipulate our Taylor expansion as fol-
lows
Vi −1 Ui
P  P
n1/2(θ̂ − θ0) ≈ √
n n
Apply Slutsky’s Theorem to conclude that the
right hand side of this converges in distribution
to N (0, σ 2/I(θ)2 ) which simplifies, because of
the identities, to N {0, 1/I(θ)}.

172
Summary

In regular families: assuming θ̂ = θ̂n is a con-

sistent root of U (θ) = 0.

• n−1/2U (θ0) ⇒ M V N (0, I) where

n o
Iij = Eθ0 V1,ij (θ0 )
and
∂ 2 log f (Xk , θ)
Vk,ij (θ) = −
∂θi∂θj

• If Vk (θ) is the matrix [Vk,ij ] then

Pn
k=1 Vk (θ0 ) → I
n

• If V(θ) = k Vk (θ) then

P

{V(θ0)/n}n1/2(θ̂ − θ0) − n−1/2U (θ0) → 0

in probability as n → ∞.

173
 • Also
{V(θ̂)/n}n1/2(θ̂ − θ0) − n−1/2U (θ0) → 0
in probability as n → ∞.

• n1/2 (θ̂ − θ0) − {I(θ0 )}−1U (θ0) → 0 in prob-

ability as n → ∞.

• n1/2 (θ̂ − θ0) ⇒ M V N (0, I −1).

• In general (not just iid cases)

q
I(θ0)(θ̂ − θ0 ) ⇒ N (0, 1)
q
I(θ̂)(θ̂ − θ0 ) ⇒ N (0, 1)
q
V (θ0)(θ̂ − θ0 ) ⇒ N (0, 1)
q
V (θ̂)(θ̂ − θ0 ) ⇒ N (0, 1)
where V = −`00 is the so-called observed
information, the negative second derivative
of the log-likelihood.

Note: If the square roots are replaced by ma-

trix square roots we can let θ be vector valued
and get M V N (0, I) as the limit law.
174
Why all these different forms? Use limit laws
to test hypotheses and compute confidence in-
tervals. Test Ho : θ = θ0 using one of the 4
quantities as test statistic. Find confidence in-
tervals using quantities as pivots. E.g.: second
and fourth limits lead to confidence intervals
q
θ̂ ± zα/2/ I(θ̂)
and
q
θ̂ ± zα/2/ V (θ̂)
respectively. The other two are more compli-
cated. For iid N (0, σ 2 ) data we have
3 Xi2 n
P
V (σ) = − 2
σ4 σ
and
2n
I(σ) =
σ2
The first line above then justifies confidence
intervals for σ computed by finding all those σ
for which
√ 
 2n(σ̂ − σ) 
 ≤ zα/2
 

 σ 
Similar interval can be derived from 3rd expres-
sion, though this is much more complicated.
175
Usual summary: mle is consistent and asymp-
totically normal with an asymptotic variance
which is the inverse of the Fisher information.

Problems with maximum likelihood

1. Many parameters lead to poor approxima-

tions. MLEs can be far from right answer.
See homework for Neyman Scott example
where MLE is not consistent.

2. Multiple roots of the likelihood equations:

you must choose the right root. Start with
different, consistent, estimator; apply iter-
ative scheme like Newton Raphson to like-
lihood equations to find MLE. Not many
steps of NR generally required if starting
point is a reasonable estimate.

176
 Finding (good) preliminary Point
Estimates

Method of Moments

Basic strategy: set sample moments equal to

population moments and solve for the param-
eters.

Definition: The r th sample moment (about

the origin) is
n
1 X
Xir
n i=1

The r th population moment is

E(X r )

(Central moments are

n
1 X
(Xi − X̄)r
n i=1
and
E [(X − µ)r ] .
177
If we have p parameters we can estimate the
parameters θ1 , . . . , θp by solving the system of
p equations:
µ1 = X̄

µ02 = X 2
and so on to
µ0p = X p
You need to remember that the population mo-
ments µ0k will be formulas involving the param-
eters.

178
Gamma Example

The Gamma(α, β) density is

!α−1 " #
1 x x
f (x; α, β) = exp − 1(x > 0)
βΓ(α) β β
and has
µ1 = αβ
and
µ02 = α(α + 1)β 2.
This gives the equations

αβ = X
α(α + 1)β 2 = X 2
or

αβ = X
2
αβ 2 = X 2 − X .

179
Divide the second equation by the first to find
the method of moments estimate of β is
2
β̃ = (X 2 − X )/X .
Then from the first equation get
2
α̃ = X/β̃ = (X)2 /(X 2 − X ) .

The method of moments equations are much

easier to solve than the likelihood equations
which involve the function
d
ψ(α) = log(Γ(α))
dα
called the digamma function.

180
Score function has components
Xi
P
Uβ = 2
− nα/β
β
and
X
Uα = −nψ(α) + log(Xi ) − n log(β) .
You can solve for β in terms of α to leave you
trying to find a root of the equation
X X
−nψ(α) + log(Xi ) − n log( Xi/(nα)) = 0
To use Newton Raphson on this you begin with
the preliminary estimate α̂1 = α̃ and then com-
pute iteratively
log(X) − ψ(α̂k ) − log(X)/α̂k
α̂k+1 =
1/α − ψ 0(α̂k )
until the sequence converges. Computation
of ψ 0, the trigamma function, requires special
software. Web sites like netlib and statlib are
good sources for this sort of thing.

181
Estimating Equations

Same large sample ideas arise whenever esti-

mates derived by solving some equation.

Example: large sample theory for Generalized

Linear Models.

Suppose Yi is number of cancer cases in some

group of people characterized by values xi of
some covariates.

Think of xi as containing variables like age, or

a dummy for sex or average income or . . ..

Possible parametric regression model: Yi has

a Poisson distribution with mean µi where the
mean µi depends somehow on xi.

182
Typically assume g(µi) = β0 + xiβ; g is link
function.

Often g(µ) = log(µ) and xiβ is a matrix prod-

uct: xi row vector, β column vector.

“Linear regression model with Poisson errors”.

Special case log(µi) = βxi where xi is a scalar.

The log likelihood is simply

X
`(β) = (Yi log(µi) − µi)
ignoring irrelevant factorials. The score func-
tion is, since log(µi) = βxi,
X X
U (β) = (Yixi − xiµi) = xi(Yi − µi)
(Notice again that the score has mean 0 when
you plug in the true parameter value.)

183
The key observation, however, is that it is not
necessary to believe that Yi has a Poisson dis-
tribution to make solving the equation U = 0
sensible. Suppose only that log(E(Yi)) = xiβ.
Then we have assumed that

Eβ (U (β)) = 0
This was the key condition in proving that
there was a root of the likelihood equations
which was consistent and here it is what is
needed, roughly, to prove that the equation
U (β) = 0 has a consistent root β̂.

184
Ignoring higher order terms in a Taylor expan-
sion will give

V (β)(β̂ − β) ≈ U (β)
where V = −U 0. In the mle case we had iden-
tities relating the expectation of V to the vari-
ance of U . In general here we have

x2
X
Var(U ) = i Var(Yi) .
If Yi is Poisson with mean µi (and so Var(Yi) =
µi) this is

x2
X
Var(U ) = i µi .
Moreover we have

Vi = x2
i µi
and so
x2
X
V (β) = i µi .

185
The central limit theorem (the Lyapunov kind)
will show that U (β) has an approximate normal
distribution with variance σU 2 = P x2Var(Y )
i i
and so
2
x2 2
X
β̂ − β ≈ N (0, σU /( i µi ) )
If Var(Yi) = µi, as it is for the Poisson case,
P 2
the asymptotic variance simplifies to 1/ xi µi.

186
Other estimating equations are possible, pop-
ular. If wi is any set of deterministic weights
(possibly depending on µi) then could define
X
U (β) = wi(Yi − µi)
and still conclude that U = 0 probably has a
consistent root which has an asymptotic nor-
mal distribution.

Idea widely used:

Example: Generalized Estimating Equations,

Zeger and Liang.

Abbreviation: GEE.

Called by econometricians Generalized Method

of Moments.

187
An estimating equation is unbiased if
Eθ (U (θ)) = 0

Theorem: Suppose θ̂ is a consistent root of

the unbiased estimating equation
U (θ) = 0.
Let V = −U 0. Suppose there is a sequence of
constants B(θ) such that
V (θ)/B(θ) → 1
and let
A(θ) = V arθ (U (θ))
and
C(θ) = B(θ)A−1 (θ)B(θ).
Then
q
C(θ0)(θ̂ − θ0 ) ⇒ N (0, 1)
q
C(θ̂)(θ̂ − θ0 ) ⇒ N (0, 1)
Other ways to estimate A, B and C lead to
same conclusions. There are multivariate ex-
tensions using matrix square roots.
188
 Optimality theory for point estimates

Why bother doing the Newton Raphson steps?

Why not just use the method of moments es-

timates?

Answer: method of moments estimates not

usually as close to right answer as MLEs.

Rough principle: A good estimate θ̂ of θ is

usually close to θ0 if θ0 is the true value of
θ. Closer estimates, more often, are better
estimates.

This principle must be quantified if we are to

“prove” that the mle is a good estimate. In
the Neyman Pearson spirit we measure average
closeness.

Definition: The Mean Squared Error (MSE)

of an estimator θ̂ is the function

M SE(θ) = Eθ [(θ̂ − θ)2]

189
Standard identity:

M SE = Varθ (θ̂) + Bias2

θ̂
(θ)
where the bias is defined as

Biasθ̂ (θ) = Eθ (θ̂) − θ .

Primitive example: I take a coin from my

pocket and toss it 6 times. I get HT HT T T .
The MLE of the probability of heads is

p̂ = X/n
where X is the number of heads. In this case
I get p̂ = 1
3.

190
Alternative estimate: p̃ = 1
2.

That is, p̃ ignores data; guess coin is fair.

The MSEs of these two estimators are

p(1 − p)
M SEMLE =
6
and
M SE0.5 = (p − 0.5)2
If 0.311 < p < 0.689 then 2nd MSE is smaller
than first.

For this reason I would recommend use of p̃ for

sample sizes this small.

Same experiment with a thumbtack: tack can

land point up (U) or tipped over (O).

If I get U OU OOO how should I estimate p the

probability of U ?

Mathematics is identical to above but is p̃ is

better than p̂?

Less reason to believe 0.311 ≤ p ≤ 0.689 than

with a coin.
191
 Unbiased Estimation

Binomial problem shows general phenomenon.

An estimator can be good for some values of

θ and bad for others.

To compare θ̂ and θ̃, two estimators of θ: Say

θ̂ is better than θ̃ if it has uniformly smaller
MSE:
M SEθ̂ (θ) ≤ M SEθ̃ (θ)
for all θ.

Normally we also require that the inequality be

strict for at least one θ.

192
Question: is there a best estimate – one which
is better than every other estimator?

Answer: NO. Suppose θ̂ were such a best es-

timate. Fix a θ ∗ in Θ and let θ̃ ≡ θ ∗.

Then MSE of θ̃ is 0 when θ = θ ∗. Since θ̂ is

better than θ̃ we must have

M SEθ̂ (θ ∗) = 0
so that θ̂ = θ ∗ with probability equal to 1.

So θ̂ = θ̃.

If there are actually two different possible val-

ues of θ this gives a contradiction; so no such
θ̂ exists.

193
Principle of Unbiasedness: A good estimate
is unbiased, that is,

Eθ (θ̂) ≡ θ .

WARNING: In my view the Principle of Unbi-

asedness is a load of hog wash.

For an unbiased estimate the MSE is just the

variance.

Definition: An estimator φ̂ of a parameter

φ = φ(θ) is Uniformly Minimum Variance
Unbiased (UMVU) if, whenever φ̃ is an unbi-
ased estimate of φ we have

Varθ (φ̂) ≤ Varθ (φ̃)

We call φ̂ the UMVUE. (‘E’ is for Estimator.)

The point of having φ(θ) is to study problems

like estimating µ when you have two parame-
ters like µ and σ for example.
194
 Cramér Rao Inequality

If φ(θ) = θ we can derive some information

from the identity

Eθ (T ) ≡ θ
When we worked with the score function we
derived some information from the identity
Z
f (x, θ)dx ≡ 1
by differentiation and we do the same here.
If T = T (X) is some function of the data X
which is unbiased for θ then
Z
Eθ (T ) = T (x)f (x, θ)dx ≡ θ
Differentiate both sides to get
d
Z
1= T (x)f (x, θ)dx
Zdθ
∂
= T (x) f (x, θ)dx
∂θ
∂
Z
= T (x) log(f (x, θ))f (x, θ)dx
∂θ
= Eθ (T (X)U (θ))
where U is the score function.
195
Since score has mean 0

Covθ (T (X), U (θ)) = 1

Remember correlations between -1 and 1 or

1 = |Covθ (T (X), U (θ))|

q
≤ Varθ (T )Var θ (U (θ)) .
Squaring gives Cramér Rao Lower Bound:
1
Varθ (T ) ≥ .
I(θ)
Inequality is strict unless corr = 1 so that

U (θ) = A(θ)T (X) + B(θ)

for non-random constants A and B (may de-
pend on θ.) This would prove that

`(θ) = A∗(θ)T (X) + B ∗ (θ) + C(X)

for other constants A∗ and B ∗ and finally
A∗(θ)T (x)+B ∗ (θ)
f (x, θ) = h(x)e
for h = eC .

196
Summary of Implications

• You can recognize a UMVUE sometimes.

If Varθ (T (X)) ≡ 1/I(θ) then T (X) is the
UMVUE. In the N (µ, 1) example the Fisher
information is n and Var(X) = 1/n so that
X is the UMVUE of µ.

• In an asymptotic sense the MLE is nearly

optimal: it is nearly unbiased and (approx-
imate) variance nearly 1/I(θ).

• Good estimates are highly correlated with

the score.

• Densities of exponential form (called expo-

nential family) given above are somehow
special.

• Usually inequality is strict — strict unless

score is affine function of a statistic T and
T (or T /c for constant c) is unbiased for θ.

197
What can we do to find UMVUEs when the
CRLB is a strict inequality?

Example: Suppose X has a Binomial(n, p) dis-

tribution. The score function is
1 n
U (p) = X−
p(1 − p) 1−p
CRLB will be strict unless T = cX for some
c. If we are trying to estimate p then choosing
c = n−1 does give an unbiased estimate p̂ =
X/n and T = X/n achieves the CRLB so it is
UMVU.

Different tactic: Suppose T (X) is some unbi-

ased function of X. Then we have

Ep(T (X) − X/n) ≡ 0

because p̂ = X/n is also unbiased. If h(k) =
T (k) − k/n then
n n
pk (1 − p)n−k ≡ 0
X
Ep(h(X)) = h(k)
k=0
k

198
LHS of ≡ sign is polynomial function of p as is
RHS.

Thus if the left hand side is expanded out the

coefficient of each power pk is 0.

Constant term occurs only in term k = 0; its

coefficient is
n
h(0) = h(0) .
0
Thus h(0) = 0.

Now p1 = p occurs only in term k = 1 with

coefficient nh(1) so h(1) = 0.

Since terms with k = 0 or 1 are 0 the quantity

p2 occurs only in k = 2 term; coefficient is

n(n − 1)h(2)/2
so h(2) = 0.

Continue to see that h(k) = 0 for each k.

So only unbiased function of X is X/n.

199
A Binomial random variable is a sum of n iid
Bernoulli(p) rvs. If Y1, . . . , Yn iid Bernoulli(p)
then X = Yi is Binomial(n, p).
P

Could we do better by than p̂ = X/n by trying

T (Y1 , . . . , Yn) for some other function T ?

Try n = 2. There are 4 possible values for

Y1, Y2. If h(Y1 , Y2) = T (Y1 , Y2) − [Y1 + Y2]/2
then
Ep(h(Y1 , Y2)) ≡ 0
and we have

Ep(h(Y1, Y2)) = h(0, 0)(1 − p)2

+[h(1, 0) + h(0, 1)]p(1 − p)
+h(1, 1)p2 .

200
This can be rewritten in the form
n n
pk (1 − p)n−k
X
w(k)
k=0
k
where

w(0) = h(0, 0)
2w(1) = h(1, 0) + h(0, 1)
w(2) = h(1, 1) .
So, as before w(0) = w(1) = w(2) = 0.

Argument can be used to prove:

For any unbiased estimate T (Y1 , . . . , Yn):

Average value of T (y1 , . . . , yn) over y1, . . . , yn

which have exactly k 1s and n − k 0s is k/n.

201
Now let’s look at the variance of T :

Var(T )
= Ep([T (Y1 , . . . , Yn) − p]2 )
= Ep([T (Y1 , . . . , Yn) − X/n + X/n − p]2 )
= Ep([T (Y1 , . . . , Yn) − X/n]2 )+
2Ep([T (Y1 , . . . , Yn) − X/n][X/n − p])
+ Ep([X/n − p]2)
Claim cross product term is 0 which will prove
variance of T is variance of X/n plus a non-
negative quantity (which will be positive un-
less T (Y1 , . . . , Yn) ≡ X/n). Compute the cross
product term by writing

Ep([T (Y1 , . . . , Yn) − X/n][X/n − p])

X X X
= [T (y1 , . . . , yn) − yi/n][ yi/n − p]
y1,...,yn
P P
×p yi (1 − p)n− yi

202
Sum over those y1, . . . , yn whose sum is an in-
teger x; then sum over x:

Ep([T (Y1 , . . . , Yn) − X/n][X/n − p])

n
X X X
= [T (y1 , . . . , yn) − yi/n]
x=0 yi=x
P
P P
yi n− yi
X
× [ yi/n − p]p (1 − p)
 
n
X  X
= [T (y1 , . . . , yn) − x/n] [x/n − p]


x=0
P
yi=x
x n−x
× p (1 − p)

We have already shown that the sum in [] is 0!

This long, algebraically involved, method of

proving that p̂ = X/n is the UMVUE of p is
one special case of a general tactic.

203
To get more insight rewrite
Ep{T (Y1, . . . , Yn)}
n
X X
= T (y1 , . . . , yn)
x=0
P
yi=x
× P (Y1 = y1, . . . , Yn = yn)
n
X X
= T (y1 , . . . , yn)
x=0
P
yi=x
× P (Y1 = y1, . . . , Yn = yn|X = x)P (X = x)
n
yi=x T (y1 , . . . , yn) n x
PP
p (1 − p)n−x
X
= n
x=0 x
x

Notice: large fraction is average value of T

over y such that yi = x.
P

Notice: weights in average do not depend on

p.

Notice: this average is actually

E{T (Y1 , . . . , Yn)|X = x}
X
= T (y1 , . . . , yn)
y1 ,...,yn
× P (Y1 = y1 , . . . , Yn = yn|X = x)
204
Notice: conditional probabilities do not depend
on p.

In a sequence of Binomial trials if I tell you

that 5 of 17 were heads and the rest tails the
actual trial numbers of the 5 Heads are chosen
at random from the 17 possibilities; all of the
17 choose 5 possibilities have the same chance
and this chance does not depend on p.

Notice: with data Y1, . . . , Yn log likelihood is

X X
`(p) = Yi log(p) − (n − Yi) log(1 − p)
and
1 n
U (p) = X−
p(1 − p) 1−p
as before. Again CRLB is strict except for
multiples of X. Since only unbiased multiple
of X is p̂ = X/n UMVUE of p is p̂.

205
 Sufficiency

In the binomial situation the conditional dis-

tribution of the data Y1, . . . , Yn given X is the
same for all values of θ; we say this conditional
distribution is free of θ.

Defn: Statistic T (X) is sufficient for the model

{Pθ ; θ ∈ Θ} if conditional distribution of data
X given T = t is free of θ.

Intuition: Data tell us about θ if different val-

ues of θ give different distributions to X. If two
different values of θ correspond to same den-
sity or cdf for X we cannot distinguish these
two values of θ by examining X. Extension of
this notion: if two values of θ give same condi-
tional distribution of X given T then observing
T in addition to X doesn’t improve our ability
to distinguish the two values.

206
Mathematically Precise version of this in-
tuition: Suppose T (X) is sufficient statistic
and S(X) is any estimate or confidence inter-
val or ... If you only know value of T then:

• Generate an observation X ∗ (via some sort

of Monte Carlo program) from the condi-
tional distribution of X given T .

• Use S(X ∗ ) instead of S(X). Then S(X ∗ )

has the same performance characteristics
as S(X) because the distribution of X ∗ is
the same as that of X.

You can carry out the first step only if the

statistic T is sufficient; otherwise you need to
know the true value of θ to generate X ∗ .

207
Example 1: Y1, . . . , Yn iid Bernoulli(p). Given
Yi = y the indexes of the y successes have
P
n
the same chance of being any one of the
y
possible subsets of {1, . . . , n}. Chance does not
depend on p so T (Y1 , . . . , Yn) = Yi is sufficient
P

statistic.

Example 2: X1 , . . . , Xn iid N (µ, 1). Joint dis-

tribution of X1, . . . , Xn, X is MVN. All entries of
mean vector are µ. Variance covariance matrix
partitioned as
" #
In×n 1n/n
1tn/n 1/n
where 1n is column vector of n 1s and In×n is
n × n identity matrix.

Compute conditional means and variances of

Xi given X; use fact that conditional law is
MVN. Conclude conditional law of data given
X = x is MVN. Mean vector has all entries
x. Variance-covariance matrix is In×n − 1n1tn/n.
No dependence on µ so X is sufficient.

WARNING: Whether or not statistic is suffi-

cient depends on density function and on Θ.
208
Theorem: [Rao-Blackwell] Suppose S(X) is a
sufficient statistic for model {Pθ , θ ∈ Θ}. If T
is an estimate of φ(θ) then:

1. E(T |S) is a statistic.

2. E(T |S) has the same bias as T ; if T is un-

biased so is E(T |S).

3. Varθ (E(T |S)) ≤ Varθ (T ) and the inequality

is strict unless T is a function of S.

4. MSE of E(T |S) is no more than MSE of T .

209
Proof: Review conditional distributions: ab-
stract definition of conditional expectation is:

Defn: E(Y |X) is any function of X such that

E [R(X)E(Y |X)] = E [R(X)Y ]

for any function R(X). E(Y |X = x) is a func-
tion g(x) such that

g(X) = E(Y |X)

Fact: If X, Y has joint density fX,Y (x, y) and

conditional density f (y|x) then
Z
g(x) = yf (y|x)dy

satisfies these definitions.

210
Proof:
Z
E(R(X)g(X)) = R(x)g(x)fX (x)dx
Z Z
= R(x)yfX (x)f (y|x)dydx
Z Z
= R(x)yfX,Y (x, y)dydx
= E(R(X)Y )

Think of E(Y |X) as average Y holding X fixed.

Behaves like ordinary expected value but func-
tions of X only are like constants:

X X
E( Ai(X)Yi|X) = Ai(X)E(Yi |X)

211
Example: Y1, . . . , Yn iid Bernoulli(p). Then
X = Yi is Binomial(n, p). Summary of con-
P

clusions:

• Log likelihood function of X only not of

Y1 , . . . , Y n .

• Only function of X which is unbiased esti-

mate of p is p̂ = X/n.

• If T (Y1 , . . . , Yn) is unbiased for p then av-

erage value of T (y1 , . . . , yn) over y1, . . . , yn
for which yi = x is x/n.
P

• Distribution of T given Yi = x does not

P

depend on p.

• If T (Y1 , . . . , Yn) is unbiased for p then

Var(T ) = Var(p̂) + E[(T − p̂)2 ]

• p̂ is the UMVUE of p.

This proof that p̂ = X/n is UMVUE of p is

special case of general tactic.
212
Proof of the Rao Blackwell Theorem

Step 1: The definition of sufficiency is that

the conditional distribution of X given S does
not depend on θ. This means that E(T (X)|S)
does not depend on θ.

Step 2: This step hinges on the following iden-

tity (called Adam’s law by Jerzy Neyman – he
used to say it comes before all the others)

E[E(Y |X)] = E(Y )

which is just the definition of E(Y |X) with
R(X) ≡ 1.

From this we deduce that

Eθ [E(T |S)] = Eθ (T )
so that E(T |S) and T have the same bias. If
T is unbiased then

Eθ [E(T |S)] = Eθ (T ) = φ(θ)

so that E(T |S) is unbiased for φ.
213
Step 3: relies on very useful decomposition.
(Total sum of squares = regression sum of
squares + residual sum of squares.)

Var(Y) = Var{E(Y |X)} + E[Var(Y |X)]

The conditional variance means

Var(Y |X) = E[{Y − E(Y |X)}2 |X]

Square out right hand side:

Var(E(Y |X)) = E[{E(Y |X) − E[E(Y |X)]}2 ]

= E[{E(Y |X) − E(Y )}2]
and

E[Var(Y |X)] = E[{Y − E(Y |X)}2]

Adding these together gives

E Y 2 − 2Y E[Y |X] + 2(E[Y |X])2

h

2
i
−2E(Y )E[Y |X] + E (Y )

214
Simplify remembering E(Y |X) is function of X
— constant when holding X fixed. So
E[Y |X]E[Y |X] = E[Y E(Y |X)|X]
taking expectations gives
E[(E[Y |X])2 ] = E[E[Y E(Y |X)|X]]
= E[Y E(Y |X)]
So 3rd term above cancels with 2nd term.

Fourth term simplifies

E[E(Y )E[Y |X]] = E(Y )E[E[Y |X]] = E 2(Y )
so that
Var(E(Y |X)) + E[Var(Y |X)] = E[Y 2] − E 2(Y )

Apply to Rao Blackwell theorem to get

Varθ (T ) = Varθ (E(T |S)) + E[(T − E(T |S))2 ]
Second term ≥ 0 so variance of E(T |S) is no
more than that of T ; will be strictly less un-
less T = E(T |S). This would mean that T is
already a function of S. Adding the squares
of the biases of T (or of E(T |S)) gives the in-
equality for MSE.
215
Examples:

In the binomial problem Y1(1 − Y2) is an unbi-

ased estimate of p(1 − p). We improve this by
computing

E(Y1 (1 − Y2)|X)
We do this in two steps. First compute

E(Y1(1 − Y2)|X = x)

216
Notice that the random variable Y1(1 − Y2) is
either 1 or 0 so its expected value is just the
probability it is equal to 1:

E(Y1 (1 − Y2)|X = x)
= P (Y1 (1 − Y2) = 1|X = x)
= P (Y1 = 1, Y2 = 0|Y1 + Y2 + · · · + Yn = x)
P (Y1 = 1, Y2 = 0, Y1 + · · · + Yn = x)
=
P (Y1 + Y2 + · · · + Yn = x)
P (Y1 = 1, Y2 = 0, Y3 + · · · + Yn = x − 1)
= n
px (1 − p)n−x
x
 n − 2
p(1 − p) px−1 (1 − p)(n−2)−(x−1)
= x− 1
n
px (1 − p)n−x
x
n − 2

= x−n 
1

x
x(n − x)
=
n(n − 1)
This is simply np̂(1 − p̂)/(n − 1) (can be bigger
than 1/4, the maximum value of p(1 − p)).

217
Example: If X1 , . . . , Xn are iid N (µ, 1) then X̄
is sufficient and X1 is an unbiased estimate of
µ. Now

E(X1 |X̄) = E[X1 − X̄ + X̄|X̄]

= E[X1 − X̄|X̄] + X̄
= X̄
which is the UMVUE.

218
 Finding Sufficient statistics

Binomial(n, θ): log likelihood `(θ) (part de-

pending on θ) is function of X alone, not of
Y1, . . . , Yn as well.

Normal example: `(µ) is, ignoring terms not

containing µ,

Xi − nµ2/2 = nµX̄ − nµ2/2 .

X
`(µ) = µ

Examples of the Factorization Criterion:

Theorem: If the model for data X has density

f (x, θ) then the statistic S(X) is sufficient if
and only if the density can be factored as

f (x, θ) = g(S(x), θ)h(x)

219
Proof: Find statistic T (X) such that X is a
one to one function of the pair S, T . Apply
change of variables to the joint density of S
and T . If the density factors then

fS,T (s, t) = g(s, θ)h(x(s, t))J(s, t)

where J is the Jacobian, so conditional density
of T given S = s does not depend on θ. Thus
the conditional distribution of (S, T ) given S
does not depend on θ and finally the condi-
tional distribution of X given S does not de-
pend on θ.

Conversely if S is sufficient then the fT |S has

no θ in it so joint density of S, T is

fS (s, θ)fT |S (t|s)

Apply change of variables formula to get

fX (x) = fS (S(x), θ)fT |S (t(x)|S(x))J(x)

where J is the Jacobian. This factors.

220
Example: If X1, . . . , Xn are iid N (µ, σ 2) then
the joint density is

(2π)−n/2 σ −n×
Xi2 /(2σ 2 ) + µ Xi/σ 2 − nµ2/(2σ 2 )}
X X
exp{−
which is evidently a function of

Xi2 ,
X X
Xi
This pair is a sufficient statistic. You can write
this pair as a bijective function of X̄, (Xi −X̄)2
P

so that this pair is also sufficient.

Example: If Y1, . . . , Yn are iid Bernoulli(p) then

pyi (1 − p)1−yi
Y
f (y1 , . . . , yp; p) =
P P
yi n− yi
=p (1 − p)
Define g(x, p) = px (1 − p)n−x and h ≡ 1 to see
that X = Yi is sufficient by the factorization
P

criterion.

221
 Minimal Sufficiency

In any model S(X) ≡ X is sufficient. (Apply

the factorization criterion.) In any iid model
the vector X(1) , . . . , X(n) of order statistics is
sufficient. (Apply the factorization criterion.)
In N (µ, 1) model we have 3 sufficient statistics:

1. S1 = (X1 , . . . , Xn).

2. S2 = (X(1) , . . . , X(n)).

3. S3 = X̄.

Notice that I can calculate S3 from the val-

ues of S1 or S2 but not vice versa and that I
can calculate S2 from S1 but not vice versa.
It turns out that X̄ is a minimal sufficient
statistic meaning that it is a function of any
other sufficient statistic. (You can’t collapse
the data set any more without losing informa-
tion about µ.)
222
Recognize minimal sufficient statistics from `:

Fact: If you fix some particular θ ∗ then the log

likelihood ratio function

`(θ) − `(θ ∗)
is minimal sufficient. WARNING: the function
is the statistic.

Subtraction of `(θ ∗ ) gets rid of irrelevant con-

stants in `. In N (µ, 1) example:

Xi2 /2+µ Xi −nµ2/2

X X
`(µ) = −n log(2π)/2−
depends on Xi2 , not needed for sufficient
P

statistic. Take µ∗ = 0 and get

∗
Xi − nµ2/2
X
`(µ) − `(µ ) = µ
This function of µ is minimal sufficient. No-
tice: from Xi can compute this minimal suf-
P

ficient statistic and vice versa. Thus Xi is

P

also minimal sufficient.

223
 Completeness

In Binomial(n, p) example only one function of

X is unbiased. Rao Blackwell shows UMVUE,
if it exists, will be a function of any sufficient
statistic.

Q: Can there be more than one such function?

A: Yes in general but no for some models like

the binomial.

Definition: A statistic T is complete for a

model Pθ ; θ ∈ Θ if

Eθ (h(T )) = 0
for all θ implies h(T ) = 0.

224
We have already seen that X is complete in
the Binomial(n, p) model. In the N (µ, 1) model
suppose
Eµ(h(X̄)) ≡ 0 .
Since X̄ has a N (µ, 1/n) distribution we find
that
√ −nµ2/2 Z
ne ∞ 2
E(h(X̄)) = √ h(x)e−nx /2 enµxdx
2π −∞
so that
Z ∞
−nx2 /2 nµx
h(x)e e dx ≡ 0 .
−∞
−nx 2 /2
Called Laplace transform of h(x)e .

Theorem: Laplace transform is 0 if and only if

the function is 0 (because you can invert the
transform).

Hence h ≡ 0.

225
 How to Prove Completeness

Only one general tactic: suppose X has density

p
X
f (x, θ) = h(x) exp{ ai(θ)Si(x) + c(θ)}
1
If the range of the function (a1 (θ), . . . , ap(θ)) as
θ varies over Θ contains a (hyper-) rectangle
in Rp then the statistic

(S1(X), . . . , Sp(X))
is complete and sufficient.

You prove the sufficiency by the factorization

criterion and the completeness using the prop-
erties of Laplace transforms and the fact that
the joint density of S1, . . . , Sp

g(s1, . . . , sp; θ) = h∗ (s) exp{ ak (θ)sk + c∗(θ)}

X

226
Example: N (µ, σ 2 ) model density has form
 
1 2+ µ x− µ2
  
exp − 2σ 2 x σ2
− log σ 2σ 2
√
2π
which is an exponential family with
1
h(x) = √
2π
1
a1 (θ) = − 2
2σ

S1(x) = x2
µ
a2 (θ) = 2
σ
S2(x) = x
and
µ2
c(θ) = − 2 − log σ .
2σ
It follows that

Xi2 ,
X X
( Xi )
is a complete sufficient statistic.
227
Remark: The statistic (s2 , X̄) is a one to one
function of ( Xi2, Xi) so it must be com-
P P

plete and sufficient, too. Any function of the

latter statistic can be rewritten as a function
of the former and vice versa.

FACT: A complete sufficient statistic is also

minimal sufficient.

The Lehmann-Scheffé Theorem

Theorem: If S is a complete sufficient statis-

tic for some model and h(S) is an unbiased
estimate of some parameter φ(θ) then h(S) is
the UMVUE of φ(θ).

Proof: Suppose T is another unbiased esti-

mate of φ. According to Rao-Blackwell, T is
improved by E(T |S) so if h(S) is not UMVUE
then there must exist another function h∗(S)
which is unbiased and whose variance is smaller
than that of h(S) for some value of θ. But
Eθ (h∗ (S) − h(S)) ≡ 0
so, in fact h∗(S) = h(S).
228
Example: In the N (µ, σ 2 ) example the random
variable (n − 1)s2 /σ 2 has a χ2
n−1 distribution.
It follows that
"√ R ∞ 1/2  x (n−1)/2−1 −x/2
0 x e dx
#
n − 1s 2
E = .
σ 2Γ((n − 1)/2)
Make the substitution y = x/2 and get
√ Z ∞
σ 2
E(s) = √ y n/2−1e−y dy .
n − 1 Γ((n − 1)/2) 0
Hence
√
2Γ(n/2)
E(s) = σ √ .
n − 1Γ((n − 1)/2)
The UMVUE of σ is then
√
n − 1Γ((n − 1)/2)
s √
2Γ(n/2)
by the Lehmann-Scheffé theorem.

229
 Criticism of Unbiasedness

• UMVUE can be inadmissible for squared

error loss meaning there is a (biased, of
course) estimate whose MSE is smaller for
every parameter value. An example is the
UMVUE of φ = p(1−p) which is φ̂ = np̂(1−
p̂)/(n − 1). The MSE of

φ̃ = min(φ̂, 1/4)
is smaller than that of φ̂.

• Unbiased estimation may be impossible.

Binomial(n, p) log odds is

φ = log(p/(1 − p)) .
Since the expectation of any function of
the data is a polynomial function of p and
since φ is not a polynomial function of p
there is no unbiased estimate of φ

230
• The UMVUE of σ is not the square root
of the UMVUE of σ 2. This method of es-
timation does not have the parameteriza-
tion equivariance that maximum likelihood
does.

• Unbiasedness is irrelevant (unless you av-

erage together many estimators).
Property is an average over possible values
of the estimate in which positive errors are
allowed to cancel negative errors.
Exception to criticism: if you average a
number of estimators to get a single esti-
mator then it is a problem if all the esti-
mators have the same bias.
See assignment 5, one way layout exam-
ple: mle of the residual variance averages
together many biased estimates and so is
very badly biased. That assignment shows
that the solution is not really to insist on
unbiasedness but to consider an alternative
to averaging for putting the individual es-
timates together.

231
 Hypothesis Testing

Hypothesis testing: a statistical problem where

you must choose, on the basis of data X, be-
tween two alternatives. We formalize this as
the problem of choosing between two hypothe-
ses: Ho : θ ∈ Θ0 or H1 : θ ∈ Θ1 where Θ0 and
Θ1 are a partition of the model Pθ ; θ ∈ Θ. That
is Θ0 ∪ Θ1 = Θ and Θ0 ∩ Θ1 = ∅.

A rule for making the required choice can be

described in two ways:

1. In terms of the set

R = {X : we choose Θ1 if we observe X}
called the rejection or critical region of the
test.

2. In terms of a function φ(x) which is equal

to 1 for those x for which we choose Θ1
and 0 for those x for which we choose Θ0.

232
For technical reasons which will come up soon I
prefer to use the second description. However,
each φ corresponds to a unique rejection region
Rφ = {x : φ(x) = 1}.

Neyman Pearson approach treats two hypothe-

ses asymmetrically. Hypothesis Ho referred to
as the null hypothesis (traditionally the hy-
pothesis that some treatment has no effect).

Definition: The power function of a test φ (or

the corresponding critical region Rφ) is

π(θ) = Pθ (X ∈ Rφ ) = Eθ (φ(X))

Interested in optimality theory, that is, the

problem of finding the best φ. A good φ will
evidently have π(θ) small for θ ∈ Θ0 and large
for θ ∈ Θ1. There is generally a trade off which
can be made in many ways, however.

233
 Simple versus Simple testing

Finding a best test is easiest when the hy-

potheses are very precise.

Definition: A hypothesis Hi is simple if Θi

contains only a single value θi.

The simple versus simple testing problem arises

when we test θ = θ0 against θ = θ1 so that Θ
has only two points in it. This problem is of
importance as a technical tool, not because it
is a realistic situation.

Suppose that the model specifies that if θ = θ0

then the density of X is f0(x) and if θ = θ1
then the density of X is f1 (x). How should we
choose φ? To answer the question we begin by
studying the problem of minimizing the total
error probability.

234
Type I error: the error made when θ = θ0 but
we choose H1, that is, X ∈ Rφ.

Type II error: when θ = θ1 but we choose H0 .

The level of a simple versus simple test is

α = Pθ0 (We make a Type I error)

or
α = Pθ0 (X ∈ Rφ ) = Eθ0 (φ(X))

Other error probability denoted β is

β = Pθ1 (X 6∈ Rφ ) = Eθ1 (1 − φ(X)).

Minimize α+β, the total error probability given

by

α + β = Eθ0 (φ(X)) + Eθ1 (1 − φ(X))

Z
= [φ(x)f0 (x) + (1 − φ(x))f1 (x)]dx

235
Problem: choose, for each x, either the value
0 or the value 1, in such a way as to minimize
the integral. But for each x the quantity

φ(x)f0 (x) + (1 − φ(x))f1 (x)

is between f0(x) and f1 (x). To make it small
we take φ(x) = 1 if f1 (x) > f0 (x) and φ(x) =
0 if f1 (x) < f0 (x). It makes no difference
what we do for those x for which f1(x) =
f0 (x). Notice: divide both sides of inequalities
to get condition in terms of likelihood ratio
f1 (x)/f0 (x).

Theorem: For each fixed λ the quantity β+λα

is minimized by any φ which has

 1 f1 (x) > λ

f0 (x)
φ(x) =
 0 f1 (x) < λ

f0 (x)

236
Neyman and Pearson suggested that in prac-
tice the two kinds of errors might well have
unequal consequences. They suggested that
rather than minimize any quantity of the form
above you pick the more serious kind of error,
label it Type I and require your rule to hold
the probability α of a Type I error to be no
more than some prespecified level α0. (This
value α0 is typically 0.05 these days, chiefly for
historical reasons.)

The Neyman and Pearson approach is then to

minimize β subject to the constraint α ≤ α0.
Usually this is really equivalent to the con-
straint α = α0 (because if you use α < α0 you
could make R larger and keep α ≤ α0 but make
β smaller. For discrete models, however, this
may not be possible.

237
Example: Suppose X is Binomial(n, p) and ei-
ther p = p0 = 1/2 or p = p1 = 3/4.

If R is any critical region (so R is a subset of

{0, 1, . . . , n}) then
k
P1/2 (X ∈ R) = n
2
for some integer k. Example: to get α0 = 0.05
with n = 5: possible values of α are 0, 1/32 =
0.03125, 2/32 = 0.0625, etc.

Possible rejection regions for α0 = 0.05:

Region α β
R1 = ∅ 0 1
R2 = {x = 0} 0.03125 1 − (1/4)5
R3 = {x = 5} 0.03125 1 − (3/4)5

So R3 minimizes β subject to α < 0.05.

Raise α0 slightly to 0.0625: possible rejection

regions are R1, R2, R3 and R4 = R2 ∪ R3 .

238
The first three have the same α and β as before
while R4 has α = α0 = 0.0625 an β = 1 −
(3/4)5 − (1/4)5 . Thus R4 is optimal!

Problem: if all trials are failures “optimal” R

chooses p = 3/4 rather than p = 1/2.

But: p = 1/2 makes 5 failures much more likely

than p = 3/4.

Problem is discreteness. Solution:

Expand set of possible values of φ to [0, 1].

Values of φ(x) between 0 and 1 represent the
chance that we choose H1 given that we ob-
serve x; the idea is that we actually toss a
(biased) coin to decide! This tactic will show
us the kinds of rejection regions which are sen-
sible.

In practice: restrict our attention to levels α0

for which best φ is always either 0 or 1. In the
binomial example we will insist that the value
of α0 be either 0 or Pθ0 (X ≥ 5) or Pθ0 (X ≥ 4)
or ...
239
Smaller example: 4 possible values of X and
24 possible rejection regions. Here is a table
of the levels for each possible rejection region
R:

R α
∅ 0
{3}, {0} 1/8
{0,3} 2/8
{1}, {2} 3/8
{0,1}, {0,2}, {1,3}, {2,3} 4/8
{0,1,3}, {0,2,3} 5/8
{1,2} 6/8
{0,1,2}, {1,2,3} 7/8
{0,1,2,3} 1

Best level 2/8 test has rejection region {0, 3},

β = 1 − [(3/4)3 + (1/4)3 ] = 36/64. Best level
2/8 test using randomization rejects when X =
3 and, when X = 2 tosses a coin with P (H) =
1/3, then rejects if you get H. Level is 1/8 +
(1/3)(3/8) = 2/8; probability of Type II error
is β = 1 − [(3/4)3 + (1/3)(3)(3/4)2 (1/4)] =
28/64.
240
Definition: A hypothesis test is a function
φ(x) whose values are always in [0, 1]. If we
observe X = x then we choose H1 with condi-
tional probability φ(X). In this case we have

π(θ) = Eθ (φ(X))

α = E0(φ(X))
and
β = 1 − E1(φ(X))

Note that a test using a rejection region C is

equivalent to

φ(x) = 1(x ∈ C)

241
The Neyman Pearson Lemma: In testing f0
against f1 the probability β of a type II error
is minimized, subject to α ≤ α0 by the test
function:

f1 (x)

 1 f (x)
>λ
0


(x)
φ(x) = γ ff1 (x)

=λ
 0
 0 f1 (x) < λ



f0 (x)
where λ is the largest constant such that
f1 (X)
P0 ( ≥ λ) ≥ α0
f0 (X)
and
f (X)
P0 ( 1 ≤ λ) ≥ 1 − α0
f0 (X)
and where γ is any number chosen so that
f1 (X)
E0(φ(X)) = P0 ( > λ)
f0 (X)
f (X)
+ γP0 ( 1 = λ)
f0 (X)
= α0
The value of γ is unique if P0 ( ff1(X) = λ) > 0.
0 (X)

242
Example: Binomial(n, p) with p0 = 1/2 and
p1 = 3/4: ratio f1/f0 is

3x 2−n
If n = 5 this ratio is one of 1, 3, 9, 27, 81, 243
divided by 32.

Suppose we have α = 0.05. λ must be one

of the possible values of f1/f0 . If we try λ =
243/32 then

P0 (3X 2−5 ≥ 243/32) = P0 (X = 5)

= 1/32 < 0.05
and

P0 (3X 2−5 ≥ 81/32) = P0(X ≥ 4)

= 6/32 > 0.05
So λ = 81/32.

243
Since

P0(3X 2−5 > 81/32) = P0 (X = 5) = 1/32

we must solve

P0 (X = 5) + γP0 (X = 4) = 0.05
for γ and find
0.05 − 1/32
γ= = 0.12
5/32

NOTE: No-one ever uses this procedure. In-

stead the value of α0 used in discrete problems
is chosen to be a possible value of the rejection
probability when γ = 0 (or γ = 1). When the
sample size is large you can come very close to
any desired α0 with a non-randomized test.

244
If α0 = 6/32 then we can either take λ to be
243/32 and γ = 1 or λ = 81/32 and γ = 0.
However, our definition of λ in the theorem
makes λ = 81/32 and γ = 0.

When the theorem is used for continuous dis-

tributions it can be the case that the cdf of
f1 (X)/f0 (X) has a flat spot where it is equal to
1 − α0. This is the point of the word “largest”
in the theorem.

Example: If X1, . . . , Xn are iid N (µ, 1) and we

have µ0 = 0 and µ1 > 0 then
f1 (X1 , . . . , Xn)
f0 (X1 , . . . , Xn)
=
Xi − nµ2 Xi + nµ2
X X
exp{µ1 1/2 − µ0 0/2}
which simplifies to

Xi − nµ2
X
exp{µ1 1/2}

245
Now choose λ so that

Xi − nµ2
X
P0(exp{µ1 1/2} > λ) = α0
Can make it equal because f1 (X)/f0 (X) has a
continuous distribution. Rewrite probability as

Xi > [log(λ) + nµ2

X
P0 ( 1/2]/µ1)
=
log(λ) + nµ2
!
1−Φ 1/2
n1/2µ1
Let zα be upper α critical point of N (0, 1); then

zα0 = [log(λ) + nµ2

1/2]/[n
1/2
µ1 ] .
Solve to get a formula for λ in terms of zα0 , n
and µ1.

246
The rejection region looks complicated: reject
if a complicated statistic is larger than λ which
has a complicated formula. But in calculating
λ we re-expressed the rejection region in terms
of
Xi
P
√ > z α0
n
The key feature is that this rejection region is
the same for any µ1 > 0. [WARNING: in the
algebra above I used µ1 > 0.] This is why the
Neyman Pearson lemma is a lemma!

247
Definition: In the general problem of testing
Θ0 against Θ1 the level of a test function φ is

α = sup Eθ (φ(X))
θ∈Θ0
The power function is

π(θ) = Eθ (φ(X))
A test φ∗ is a Uniformly Most Powerful level α0
test if

1. φ∗ has level α ≤ αo

2. If φ has level α ≤ α0 then for every θ ∈ Θ1

we have

Eθ (φ(X)) ≤ Eθ (φ∗ (X))

248
Proof of Neyman Pearson lemma: Given a
test φ with level strictly less than α0 we can
define the test
1 − α0 α −α
φ∗ (x) = φ(x) + 0
1−α 1−α
has level α0 and β smaller than that of φ.
Hence we may assume without loss that α = α0
and minimize β subject to α = α0. However,
the argument which follows doesn’t actually
need this.

249
 Lagrange Multipliers

Suppose you want to minimize f (x) subject to

g(x) = 0. Consider first the function

hλ(x) = f (x) + λg(x)

If xλ minimizes hλ then for any other x

f (xλ ) ≤ f (x) + λ[g(x) − g(xλ)]

Now suppose you can find a value of λ such
that the solution xλ has g(xλ) = 0. Then for
any x we have

f (xλ ) ≤ f (x) + λg(x)

and for any x satisfying the constraint g(x) = 0
we have
f (xλ ) ≤ f (x)
This proves that for this special value of λ the
quantity xλ minimizes f (x) subject to g(x) = 0.

Notice that to find xλ you set the usual partial

derivatives equal to 0; then to find the special
xλ you add in the condition g(xλ) = 0.
250
 Return to proof of NP lemma

For each λ > 0 we have seen that φλ minimizes

λα + β where φλ = 1(f1 (x)/f0 (x) ≥ λ).

As λ increases the level of φλ decreases from 1

when λ = 0 to 0 when λ = ∞. There is thus a
value λ0 where for λ > λ0 the level is less than
α0 while for λ < λ0 the level is at least α0.
Temporarily let δ = P0(f1 (X)/f0 (X) = λ0 ). If
δ = 0 define φ = φλ . If δ > 0 define

f1(x)




1 f0(x)
> λ0
φ(x) = γ ff1(x)

(x)
= λ0
 0
 0 f1(x) < λ0



f0(x)
where P0 (f1 (X)/f0 (X) > λ0 ) + γδ = α0. You
can check that γ ∈ [0, 1].

251
Now φ has level α0 and according to the the-
orem above minimizes lambda0 α + β. Suppose
φ∗ is some other test with level α∗ ≤ α0. Then

λ0 α φ + βφ ≤ λ 0 α φ ∗ + β φ ∗
We can rearrange this as

βφ∗ ≥ βφ + (αφ − αφ∗ )λ0

Since
α φ∗ ≤ α 0 = α φ
the second term is non-negative and

βφ ∗ ≥ β φ
which proves the Neyman Pearson Lemma.

252
Example application of NP: Binomial(n, p)
to test p = p0 versus p1 for a p1 > p0 the NP
test is of the form

φ(x) = 1(X > k) + γ1(X = k)

where we choose k so that

Pp0 (X > k) ≤ α0 < Pp0 (X ≥ k)

and γ ∈ [0, 1) so that

α0 = Pp0 (X > k) + γPp0 (X = k)

This rejection region depends only on p0 and
not on p1 so that this test is UMP for p = p0
against p > p0. Since this test has level α0 even
for the larger null hypothesis it is also UMP for
p ≤ p0 against p > p0.

253
Application of the NP lemma: In the N (µ, 1)
model consider Θ1 = {µ > 0} and Θ0 = {0}
or Θ0 = {µ ≤ 0}. The UMP level α0 test of
H0 : µ ∈ Θ0 against H1 : µ ∈ Θ1 is

φ(X1 , . . . , Xn ) = 1(n1/2 X̄ > zα0 )

Proof: For either choice of Θ0 this test has

level α0 because for µ ≤ 0 we have

Pµ(n1/2 X̄ > zα0 )

= Pµ (n1/2(X̄ − µ) > zα0 − n1/2µ)
= P (N (0, 1) > zα0 − n1/2µ)
≤ P (N (0, 1) > zα0 )
= α0
(Notice the use of µ ≤ 0. The central point
is that the critical point is determined by the
behaviour on the edge of the null hypothesis.)

254
Now if φ is any other level α0 test then we have

E0(φ(X1 , . . . , Xn)) ≤ α0
Fix a µ > 0. According to the NP lemma

Eµ(φ(X1 , . . . , Xn)) ≤ Eµ(φµ (X1 , . . . , Xn))

where φµ rejects if

fµ (X1 , . . . , Xn)/f0 (X1 , . . . , Xn ) > λ

for a suitable λ. But we just checked that this
test had a rejection region of the form

n1/2X̄ > zα0

which is the rejection region of φ∗ . The NP
lemma produces the same test for every µ > 0
chosen as an alternative. So we have shown
that φµ = φ∗ for any µ > 0.

255
Fairly general phenomenon: for any µ > µ0 the
likelihood ratio fµ/f0 is an increasing function
of Xi. The rejection region of the NP test
P

is thus always a region of the form Xi > k.

P

The value of the constant k is determined by

the requirement that the test have level α0 and
this depends on µ0 not on µ1.

Definition: The family fθ ; θ ∈ Θ ⊂ R has

monotone likelihood ratio with respect to a
statistic T (X) if for each θ1 > θ0 the likelihood
ratio fθ1 (X)/fθ0 (X) is a monotone increasing
function of T (X).

Theorem: For a monotone likelihood ratio

family the Uniformly Most Powerful level α test
of θ ≤ θ0 (or of θ = θ0) against the alternative
θ > θ0 is

 1 T (x) > tα

φ(x) = γ T (X) = tα
 0 T (x) < t

α
where
Pθ0 (T (X) > tα) + γPθ0 (T (X) = tα) = α0 .

256
Typical family where this works: one parameter
exponential family. Usually there is no UMP
test.

Example: test µ = µ0 against two sided alter-

native µ 6= µ0. There is no UMP level α test.

If there were its power at µ > µ0 would have

to be as high as that of the one sided level α
test and so its rejection region would have to
be the same as that test, rejecting for large
positive values of X̄ − µ0. But it also has to
have power as good as the one sided test for
the alternative µ < µ0 and so would have to
reject for large negative values of X̄ − µ0. This
would make its level too large.

Favourite test: usual 2 sided test rejects for

large values of |X̄ − µ0|. Test maximizes power
subject to two constraints: first, level α; sec-
ond power is minimized at µ = µ0. Second
condition means power on alternative is larger
than on the null.
257
Definition: A test φ of Θ0 against Θ1 is un-
biased level α if it has level α and, for every
θ ∈ Θ1 we have

π(θ) ≥ α .

When testing a point null hypothesis like µ =

µ0 this requires that the power function be
minimized at µ0 which will mean that if π is
differentiable then

π 0(µ0) = 0

258
Example: N (µ, 1): data X = (X1 , . . . , Xn). If
φ is any test function then
∂
Z
0
π (µ) = φ(x)f (x, µ)dx
∂µ
Differentiate under the integral and use
∂f (x, µ) X
= (xi − µ)f (x, µ)
∂µ
to get the condition
Z
φ(x)x̄f (x, µ0)dx = µ0α0

Minimize β(µ) subject to two constraints

Eµ0 (φ(X)) = α0
and
Eµ0 (X̄φ(X)) = µ0α0.

259
Fix two values λ1 > 0 and λ2 and minimize

λ1α + λ2Eµ0 [(X̄ − µ0)φ(X)] + β

The quantity in question is just
Z
[φ(x)f0 (x)(λ1 + λ2(x̄ − µ0))
+ (1 − φ(x))f1 (x)]dx .
As before this is minimized by

f1(x)
 1

f0(x)
> λ1 + λ2(x̄ − µ0)
φ(x) = f1(x)
 0

f0(x)
< λ1 + λ2(x̄ − µ0)

260
The likelihood ratio f1 /f0 is simply

exp{n(µ1 − µ0)X̄ + n(µ2

0 − µ 2
1)/2}
and this exceeds the linear function

λ1 + λ2 (X̄ − µ0)
for all X̄ sufficiently large or small. That is,

λ1α + λ2Eµ0 [(X̄ − µ0)φ(X)] + β

is minimized by a rejection region of the form

{X̄ > KU } ∪ {X̄ < KL}

Satisfy constraints: adjust KU and KL to get
level α and π 0(µ0) = 0. 2nd condition shows
rejection region symmetric about µ0 so test
rejects for
√
n|X̄ − µ0| > zα/2

Mimic Neyman Pearson lemma proof to check

that if λ1 and λ2 are adjusted so that the un-
constrained problem has the rejection region
given then the resulting test minimizes β sub-
ject to the two constraints.
261
A test φ∗ is a Uniformly Most Powerful Unbi-
ased level α0 test if

1. φ∗ has level α ≤ α0.

2. φ∗ is unbiased.

3. If φ has level α ≤ α0 and φ is unbiased then

for every θ ∈ Θ1 we have

Eθ (φ(X)) ≤ Eθ (φ∗ (X))

Conclusion: The two sided z test which re-

jects if
|Z| > zα/2
where
Z = n1/2(X̄ − µ0)
is the uniformly most powerful unbiased test
of µ = µ0 against the two sided alternative
µ 6= µ0.
262
 Nuisance Parameters

The t-test is UMPU.

Suppose X1, . . . , Xn iid N (µ, σ 2). Test µ = µ0

or µ ≤ µ0 against µ > µ0. Parameter space
is two dimensional; boundary between the null
and alternative is

{(µ, σ); µ = µ0, σ > 0}

If a test has π(µ, σ) ≤ α for all µ ≤ µ0 and

π(µ, σ) ≥ α for all µ > µ0 then π(µ0, σ) = α for
all σ because the power function of any test
must be continuous. (Uses dominated conver-
gence theorem; power function is an integral.)

Think of {(µ, σ); µ = µ0} as parameter space

for a model. For this parameter space

(Xi − µ0)2
X
S=
is complete and sufficient. Remember defini-
tions of both completeness and sufficiency de-
pend on the parameter space.
263
Suppose φ( Xi, S) is an unbiased level α test.
P

Then we have
X
Eµ0,σ (φ( Xi, S)) = α
for all σ. Condition on S and get
X
Eµ0,σ [E(φ( Xi , S)|S)] = α
for all σ. Sufficiency guarantees that
X
g(S) = E(φ( Xi, S)|S)
is a statistic and completeness that

g(S) ≡ α

Now let us fix a single value of σ and a µ1 > µ0.

To make our notation simpler I take µ0 = 0.
Our observations above permit us to condition
on S = s. Given S = s we have a level α test
which is a function of X̄.

264
If we maximize the conditional power of this
test for each s then we will maximize its power.
What is the conditional model given S = s?
That is, what is the conditional distribution of
X̄ given S = s? The answer is that the joint
density of X̄, S is of the form

fX̄,S (t, s) = h(s, t) exp{θ1t + θ2s + c(θ1 , θ2)}

where θ1 = nµ/σ 2 and θ2 = −1/σ 2.

This makes the conditional density of X̄ given

S = s of the form

fX̄|s(t|s) = h(s, t) exp{θ1t + c∗(θ1 , s)}

265
Note disappearance of θ2 and null is θ1 = 0.
This permits application of NP lemma to the
conditional family to prove that UMP unbiased
test has form

φ(X̄, S) = 1(X̄ > K(S))

where K(S) chosen to q make conditional level
α. The function x 7→ x/ a − x2 is increasing in
x for each a so that we can rewrite φ in the
form

φ(X̄, S) =
q
1(n1/2X̄/ n[S/n − X̄ 2]/(n − 1) > K ∗(S))
for some K ∗. The quantity
n1/2X̄
T =q
n[S/n − X̄ 2]/(n − 1)
is the usual t statistic and is exactly indepen-
dent of S (see Theorem 6.1.5 on page 262 in
Casella and Berger). This guarantees that

K ∗(S) = tn−1,α
and makes our UMPU test the usual t test.
266
 Optimal tests

• A good test has π(θ) large on the alterna-

tive and small on the null.

• For one sided one parameter families with

MLR a UMP test exists.

• For two sided or multiparameter families

the best to be hoped for is UMP Unbiased
or Invariant or Similar.

• Good tests are found as follows:

1. Use the NP lemma to determine a good
rejection region for a simple alternative.
2. Try to express that region in terms of
a statistic whose definition does not de-
pend on the specific alternative.
3. If this fails impose an additional criterion
such as unbiasedness. Then mimic the
NP lemma and again try to simplify the
rejection region.

267
 Likelihood Ratio tests

For general composite hypotheses optimality

theory is not usually successful in producing
an optimal test. instead we look for heuristics
to guide our choices. The simplest approach
is to consider the likelihood ratio
fθ1 (X)
fθ0 (X)
and choose values of θ1 ∈ Θ1 and θ0 ∈ Θ0
which are reasonable estimates of θ assuming
respectively the alternative or null hypothesis is
true. The simplest method is to make each θi
a maximum likelihood estimate, but maximized
only over Θi.

268
Example 1: N (µ, 1): test µ ≤ 0 against µ > 0.
(Remember UMP test.) Log likelihood is

−n(X̄ − µ)2/2
If X̄ > 0 then global maximum in Θ1 at X̄. If
X̄ ≤ 0 global maximum in Θ1 at 0. Thus µ̂1
which maximizes `(µ) subject to µ > 0 is X̄ if
X̄ > 0 and 0 if X̄ ≤ 0. Similarly, µ̂0 is X̄ if
X̄ ≤ 0 and 0 if X̄ > 0. Hence
fθ̂ (X)
1
= exp{`(µ̂1) − `(µ̂0)}
fθ̂ (X)
0
which simplifies to

exp{nX̄|X̄|/2}
Monotone increasing function of X̄ so rejection
region will be of the form X̄ > K. To get level
α reject if n1/2X̄ > zα. Notice simpler statistic
is log likelihood ratio
!
fµ̂1 (X)
λ ≡ 2 log = nX̄|X̄|
fµ̂0 (X)

269
Example 2: In the N (µ, 1) problem suppose
we make the null µ = 0. Then the value of
µ̂0 is simply 0 while the maximum of the log-
likelihood over the alternative µ 6= 0 occurs at
X̄. This gives
λ = nX̄ 2
which has a χ21 distribution. This test leads to
the rejection region λ > (zα/2 )2 which is the
usual UMPU test.

Example 3: For the N (µ, σ 2) problem test-

ing µ = 0 against µ 6= 0 we must find two
estimates of µ, σ 2 . The maximum of the likeli-
hood over the alternative occurs at the global
mle X̄, σ̂ 2. We find

`(µ̂, σ̂ 2 ) = −n/2 − n log(σ̂)

Maximize ` over null hypothesis. Recall

1 X
`(µ, σ) = − 2 (Xi − µ)2 − n log(σ)
2σ

270
On null µ = 0 so find σ̂0 by maximizing
1 X 2
`(0, σ) = − 2 Xi − n log(σ)
2σ
This leads to

σ̂02 = Xi2/n
X

and
`(0, σ̂0) = −n/2 − n log(σ̂0 )
This gives

λ = −n log(σ̂ 2 /σ̂02)
Since
σ̂ 2 (Xi − X̄)2
P

2
=P
σ̂0 (Xi − X̄)2 + nX̄ 2
we can write

λ = n log(1 + t2 /(n − 1))

where
n1/2X̄
t=
s
is the usual t statistic. Likelihood ratio test
rejects for large values of |t| — the usual test.
271
Notice that if n is large we have

λ ≈ n[1 + t2 /(n − 1) + O(n−2)] ≈ t2 .

Since the t statistic is approximately standard
normal if n is large we see that

λ = 2[`(θ̂1) − `(θ̂0 )]
has nearly a χ2
1 distribution.

This is a general phenomenon when the null

hypothesis being tested is of the form φ = 0.
Here is the general theory. Suppose that the
vector of p + q parameters θ can be partitioned
into θ = (φ, γ) with φ a vector of p parameters
and γ a vector of q parameters. To test φ = φ0
we find two mles of θ. First the global mle
θ̂ = (φ̂, γ̂) maximizes the likelihood over Θ1 =
{θ : φ 6= φ0 } (because typically the probability
that φ̂ is exactly φ0 is 0).

272
Now we maximize the likelihood over the null
hypothesis, that is we find θ̂0 = (φ0 , γ̂0 ) to
maximize
`(φ0 , γ)
The log-likelihood ratio statistic is

2[`(θ̂) − `(θ̂0 )]

Now suppose that the true value of θ is φ0, γ0

(so that the null hypothesis is true). The score
function is a vector of length p + q and can
be partitioned as U = (Uφ , Uγ ). The Fisher
information matrix can be partitioned as
" #
Iφφ Iφγ
.
Iγφ Iγγ

273
According to our large sample theory for the
mle we have

θ̂ ≈ θ + I −1U
and
−1 U
γ̂0 ≈ γ0 + Iγγ γ

If you carry out a two term Taylor expansion

of both `(θ̂) and `(θ̂0 ) around θ0 you get
1 t −1
t −1
`(θ̂) ≈ `(θ0) + U I U + U I V (θ)I −1U
2
where V is the second derivative matrix of `.
Remember that V ≈ −I and you get

2[`(θ̂) − `(θ0)] ≈ U tI −1U .

A similar expansion for θ̂0 gives
−1
2[`(θ̂0) − `(θ0 )] ≈ Uγt Iγγ Uγ .

274
If you subtract these you find that

2[`(θ̂) − `(θ̂0 )]
can be written in the approximate form

U tM U
for a suitable matrix M . It is now possible to
use the general theory of the distribution of
X tM X where X is M V N (0, Σ) to demonstrate
that

Theorem: The log-likelihood ratio statistic

λ = 2[`(θ̂) − `(θ̂0)]
has, under the null hypothesis, approximately
a χ2
p distribution.

275
Aside:

Theorem: Suppose X ∼ M V N (0, Σ) with Σ

non-singular and M is a symmetric matrix. If
ΣM ΣM Σ = ΣM Σ then X tM X has a χ2 ν dis-
tribution with df ν = trace(M Σ).

Proof: We have X = AZ where AAt = Σ

and Z is standard multivariate normal. So
X tM X = Z tAtM AZ. Let Q = AtM A. Since
AAt = Σ condition in the theorem is

AQQAt = AQAt
Since Σ is non-singular so is A. Multiply by
A−1 on left and (At)−1 on right; get QQ = Q.

Q is symmetric so Q = P ΛP t where Λ is diago-

nal matrix containing the eigenvalues of Q and
P is orthogonal matrix whose columns are the
corresponding orthonormal eigenvectors. So
rewrite
Z tQZ = (P t Z)tΛ(P Z) .
276
W = P tZ is M V N (0, P t P = I); i.e. W is stan-
dard multivariate normal. Now
t
λi Wi2
X
W ΛW =

We have established that the general distribu-

tion of any quadratic form X tM X is a linear
combination of χ2 variables. Now go back to
the condition QQ = Q. If λ is an eigenvalue
of Q and v 6= 0 is a corresponding eigenvec-
tor then QQv = Q(λv) = λQv = λ2v but also
QQv = Qv = λv. Thus λ(1−λ)v = 0. It follows
that either λ = 0 or λ = 1. This means that
the weights in the linear combination are all 1
or 0 and that X tM X has a χ2 distribution with
degrees of freedom, ν, equal to the number of
λi which are equal to 1. This is the same as
the sum of the λi so

ν = trace(Λ)

277
But

trace(M Σ) = trace(M AAt)

= trace(At M A)
= trace(Q)
= trace(P ΛP t )
= trace(ΛP t P )
= trace(Λ)

In the application Σ is I the Fisher information

and M = I −1 − J where
" #
0 0
J= −1
0 Iγγ
It is easy to check that M Σ becomes
" #
I 0
−IγφIφφ 0
where I is a p × p identity matrix. It follows
that ΣM ΣM Σ = ΣM Σ and trace(M Σ) = p.

278
 Confidence Sets

Defn: A level β confidence set for a param-

eter φ(θ) is a random subset C, of the set of
possible values of φ such that for each θ

Pθ (φ(θ) ∈ C) ≥ β
Confidence sets are very closely connected with
hypothesis tests:

From confidence sets to tests

Suppose C is a level β = 1 − α confidence set

for φ.

To test φ = φ0 : reject if φ 6∈ C. This test has

level α.

279
 From tests to confidence sets

Conversely, suppose that for each φ0 we have

available a level α test of φ = φ0 who rejection
region is say Rφ0 .

Define C = {φ0 : φ = φ0 is not rejected}; get

level 1 − α confidence set for φ.

Example: Usual t test gives rise in this way to

the usual t confidence intervals
s
X̄ ± tn−1,α/2 √ .
n

280
 Confidence sets from Pivots

Definition: A pivot (pivotal quantity) is a

function g(θ, X) whose distribution is the same
for all θ. (θ in pivot is same θ as being used to
calculate distribution of g(θ, X).

Using pivots to generate confidence sets:

Pick a set A in space of possible values for g.

Let β = Pθ (g(θ, X) ∈ A); since g is pivotal β is

the same for all θ.

Given data X solve the relation

g(θ, X) ∈ A
to get
θ ∈ C(X, A) .

281
Example: (n−1)s2 /σ 2 ∼ χ2
n−1 is a pivot in the
N (µ, σ 2 ) model.

Given β = 1 − α consider the two points

χ2
n−1,1−α/2 and χ 2
n−1,α/2.
Then

P (χ2 2 2 2
n−1,1−α/2 ≤ (n − 1)s /σ ≤ χn−1,α/2 ) = β
for all µ, σ.

Solve this relation:

(n − 1)1/2 s (n − 1)1/2 s
P( ≤σ≤ )=β
χn−1,α/2 χn−1,1−α/2
so interval
 
(n − 1) 1/2 s (n − 1)1/2 s
 , 
χn−1,α/2 χn−1,1−α/2
is a level 1 − α confidence interval.

282
In the same model we also have

P (χ2
n−1,1−α ≤ (n − 1)s 2 2
/σ ) = β
which can be solved to get
(n − 1)1/2 s
P (σ ≤ )=β
χn−1,1−α
This gives a level 1 − α interval

(0, (n − 1)1/2 s/χn−1,1−α) .

The right hand end of this interval is usually
called a confidence upper bound.

In general the interval from

(n − 1)1/2 s/χn−1,α1 to (n − 1)1/2 s/χn−1,1−α2

has level β = 1 − α1 − α2. For fixed β can mini-
mize length of resulting interval numerically —
rarely used. See homework for an example.

283
 Decision Theory and Bayesian Methods

Example: Decide between 4 modes of trans-

portation to work:

• B = Ride my bike.

• C = Take the car.

• T = Use public transit.

• H = Stay home.

Costs depend on weather: R = Rain or S =

Sun.

284
Ingredients of Decision Problem: No data
case.

• Decision space D = {B, C, T, H} of possible

actions.

• Parameter space Θ = {R, S} of possible

“states of nature”.

• Loss function L = L(d, θ) loss incurred if

do d and θ is true state of nature.

285
In the example we might use the following table
for L:

C B T H
R 3 8 5 25
S 5 0 2 25

Notice that if it rains I will be glad if I drove.

If it is sunny I will be glad if I rode my bike. In
any case staying at home is expensive.

In general we study this problem by comparing

various functions of θ. In this problem a func-
tion of θ has only two values, one for rain and
one for sun and we can plot any such function
as a point in the plane. We do so to indicate
the geometry of the problem before stating the
general theory.

286
 Losses of deterministic rules
30
25

•H
20
Rain

15
10

•B
•T
5

•C
0

0 5 10 15 20 25 30

Sun

287
 Statistical Decision Theory

Statistical problems have another ingredient,

the data. We observe X a random variable
taking values in say X . We may make our
decision d depend on X. A decision rule is
a function δ(X) from X to D. We will want
L(δ(X), θ) to be small for all θ. Since X is
random we quantify this by averaging over X
and compare procedures δ in terms of the risk
function

Rδ (θ) = Eθ (L(δ(X), θ))

To compare two procedures we must compare

two functions of θ and pick “the smaller one”.
But typically the two functions will cross each
other and there won’t be a unique ‘smaller
one’.

288
Example: In estimation theory to estimate a
real parameter θ we used D = Θ,

L(d, θ) = (d − θ)2
and find that the risk of an estimator θ̂(X) is

Rθ̂ (θ) = E[(θ̂ − θ)2 ]

which is just the Mean Squared Error of θ̂. We
have already seen that there is no unique best
estimator in the sense of MSE. How do we
compare risk functions in general?

289
• Minimax methods choose δ to minimize
the worst case risk:

sup{Rδ (θ); θ ∈ Θ)}.

We call δ ∗ minimax if

sup Rδ ∗ (θ) = inf sup Rδ (θ)

θ δ θ
Usually the sup and inf are achieved and
we write max for sup and min for inf. This
is the source of “minimax”.

• Bayes methods choose δ to minimize an

average
Z
rπ (δ) = Rδ (θ)π(θ)dθ

for a suitable density π. We call π a prior

density and r the Bayes risk of δ for the
prior π.

290
Example: Transport problem has no data so
the only possible (non-randomized) decisions
are the four possible actions B, C, T, H. For B
and T the worst case is rain. For the other two
actions Rain and Sun are equivalent. We have
the following table:

C B T H
R 3 8 5 25
S 5 0 2 25
Maximum 5 8 5 25

Smallest maximum: take car, or transit.

Minimax action: take car or public transit.

291
Now imagine: toss coin with probability λ of
getting Heads, take my car if Heads, otherwise
take transit. Long run average daily loss would
be 3λ + 5(1 − λ) when it rains and 5λ + 2(1 − λ)
when it is Sunny. Call this procedure dλ ; add
it to graph for each value of λ. Varying λ
from 0 to 1 gives a straight line running from
(3, 5) to (5, 2). The two losses are equal when
λ = 3/5. For smaller λ worst case risk is for
sun; for larger λ worst case risk is for rain.

Added to graph: loss functions for each dλ ,

(straight line) and set of (x, y) pairs for which
min(x, y) = 3.8 — worst case risk for dλ when
λ = 3/5.

292
 Losses
30
25

•H
20
Rain

15
10

•B
•T
5

•C
0

0 5 10 15 20 25 30

Sun

293
The figure then shows that d3/5 is actually
the minimax procedure when randomized pro-
cedures are permitted.

In general we might consider using a 4 sided

coin where we took action B with probabil-
ity λB , C with probability λC and so on. The
loss function of such a procedure is a convex
combination of the losses of the four basic pro-
cedures making the set of risks achievable with
the aid of randomization look like the follow-
ing:

294
 Losses
30
25

•
20
Rain

15
10

•
•
5

•
0

0 5 10 15 20 25 30

Sun

295
Randomization in decision problems permits
assumption that set of possible risk functions
is convex — an important technical conclusion
used to prove many basic decision theory re-
sults.

Graph shows many points in the picture cor-

respond to bad decision procedures. Rain or
not taking my car to work has a lower loss
than staying home; the decision to stay home
is inadmissible.

Definition: A decision rule δ is inadmissible

if there is a rule δ ∗ such that

Rδ ∗ (θ) ≤ Rδ (θ)
for all θ and there is at least one value of θ
where the inequality is strict. A rule which is
not inadmissible is called admissible.

Admissible procedures have risks on lower left

of graphs, i.e., lines connecting B to T and T
to C are the admissible procedures.
296
Connection between Bayes procedures and ad-
missible procedures:

Prior distribution in example specified by two

probabilities, πS and πR which add up to 1.

If L = (LS , LR ) is the risk function for some

procedure then the Bayes risk is

r π = π R LR + π S LS .
Consider set of L such that this Bayes risk is
equal to some constant.

On picture this is line with slope −πS /πR.

297
Consider three priors: π1 = (0.9, 0.1), π2 =
(0.5, 0.5) and π3 = (0.1, 0.9).

For π1: imagine a line with slope -9 =0.9/0.1

starting on the far left of the picture and slid-
ing right until it bumps into the convex set of
possible losses in the previous picture. It does
so at point B as shown in the next graph.

Sliding this line to the right corresponds to

making rπ larger and larger so that when it
just touches the convex set we have found the
Bayes procedure.

298
 Losses
30
25

•
20
Rain

15
10

•
•
5

•
0

0 5 10 15 20 25 30

Sun

299
Here is a picture showing the same lines for the
three priors above.
Losses
30
25

•
20
Rain

15
10

•
•
5

•
0

0 5 10 15 20 25 30

Sun

300
Bayes procedure for π1 (you’re pretty sure it
will be sunny) is to ride your bike. If it’s a toss
up between R and S you take the bus. If R is
very likely you take your car. Prior (0.6, 0.4)
produces the line shown here:
Losses
30
25

•
20
Rain

15
10

•
•
5

•
0

0 5 10 15 20 25 30

Sun

Any point on line BT is Bayes for this prior.

301
Decision Theory and Bayesian Methods
Summary for no data case

• Decision space is the set of possible actions

I might take. We assume that it is con-
vex, typically by expanding a basic decision
space D to the space D of all probability
distributions on D.

• Parameter space Θ of possible “states of

nature”.

• Loss function L = L(d, θ) which is the loss

I incur if I do d and θ is the true state of
nature.

• We call δ ∗ minimax if

max L(δ ∗ , θ) = min max L(δ, θ) .

θ δ θ

302
• A prior is a probability distribution π on Θ,.

• The Bayes risk of a decision δ for a prior π

is
Z
rπ (δ) = Eπ (L(δ, θ)) = L(δ, θ)π(θ)dθ

if the prior has a density. For finite param-

eter spaces Θ the integral is a sum.

• A decision δ ∗ is Bayes for a prior π if

rπ (δ ∗ ) ≤ rπ (δ)
for any decision δ.

303
• For infinite parameter spaces: π(θ) > 0 on
R
Θ is a proper prior if π(θ)dθ < ∞; divide π
R
by integral to get a density. If π(θ)dθ = ∞
π is an improper prior density.

• Decision δ is inadmissible if there is δ ∗ such

that
L(δ ∗ , θ) ≤ L(δ, θ)
for all θ and there is at least one value of
θ where the inequality is strict. A decision
which is not inadmissible is called admis-
sible.

• Every admissible procedure is Bayes, per-

haps only for an improper prior. (Proof
uses the Separating Hyperplane Theorem
in Functional Analysis.)

304
• Every Bayes procedure with finite Bayes
risk (for prior with density > 0 for all θ)
is admissible.

Proof: If δ is Bayes for π but not admissible

there is a δ ∗ such that

L(δ ∗ , θ) ≤ L(δ, θ)
Multiply by the prior density; integrate:

rπ (δ ∗ ) ≤ rπ (δ)
If there is a θ for which the inequality in-
volving L is strict and if the density of π
is positive at that θ then the inequality for
rπ is strict which would contradict the hy-
pothesis that δ is Bayes for π.

Notice: theorem actually requires the ex-

tra hypotheses: positive density, and risk
functions of δ and δ ∗ continuous.

305
• A minimax procedure is admissible. (Ac-
tually there can be several minimax proce-
dures and the claim is that at least one of
them is admissible. When the parameter
space is infinite it might happen that set
of possible risk functions is not closed; if
not then we have to replace the notion of
admissible by some notion of nearly admis-
sible.)

• The minimax procedure has constant risk.

Actually the admissible minimax procedure
is Bayes for some π and its risk is constant
on the set of θ for which the prior density
is positive.

306
Decision Theory and Bayesian Methods
Summary when there is data

• Decision space is the set of possible actions

I might take. We assume that it is con-
vex, typically by expanding a basic decision
space D to the space D of all probability
distributions on D.

• Parameter space Θ of possible “states of

nature”.

• Loss function L = L(d, θ): loss I incur if I

do d and θ is true state of nature.

• Add data X ∈ X with model {Pθ ; θ ∈ Θ}:

model density is f (x|θ).

• A procedure is a map δ : X 7→ D.

• The risk function for δ is the expected loss:

Rδ (θ) = R(δ, θ) = E [L{δ(X), θ}] .

307
• We call δ ∗ minimax if

max R(δ ∗, θ) = min max R(δ, θ) .

θ δ θ

• A prior is a probability distribution π on Θ,.

• Bayes risk of decision δ for prior π is

rπ (δ) = Eπ (R(δ, θ))

Z
= L(δ(x), θ)f (x|θ)π(θ)dxdθ

if the prior has a density. For finite param-

eter spaces Θ the integral is a sum.

• A decision δ ∗ is Bayes for a prior π if

rπ (δ ∗ ) ≤ rπ (δ)
for any decision δ.

308
• For infinite parameter spaces: π(θ) > 0 on
R
Θ is a proper prior if π(θ)dθ < ∞; divide π
R
by integral to get a density. If π(θ)dθ = ∞
π is an improper prior density.

• Decision δ is inadmissible if there is δ ∗ such

that
R(δ ∗ , θ) ≤ R(δ, θ)
for all θ and there is at least one value of
θ where the inequality is strict. A decision
which is not inadmissible is called admis-
sible.

• Every admissible procedure is Bayes, per-

haps only for an improper prior.

309
• If every risk function is continuous then ev-
ery Bayes procedure with finite Bayes risk
(for prior with density > 0 for all θ) is ad-
missible.

• A minimax procedure is admissible.

• The minimax procedure has constant risk.

The admissible minimax procedure is Bayes
for some π; its risk is constant on the set
of θ for which the prior density is positive.

310
 Bayesian estimation

Focus on problem of estimation of 1 dimen-

sional parameter.

Mean Squared Error corresponds to using

L(d, θ) = (d − θ)2 .

Risk function of procedure (estimator) θ̂ is

Rθ̂ (θ) = Eθ [(θ̂ − θ)2]

Now consider prior with density π(θ).

Bayes risk of θ̂ is
Z
rπ = Rθ̂ (θ)π(θ)dθ
Z Z
= (θ̂(x) − θ)2 f (x; θ)π(θ)dxdθ

311
Choose θ̂ to minimize rπ ?

Recognize that f (x; θ)π(θ) is really a joint den-

sity
Z Z
f (x; θ)π(θ)dxdθ = 1

For this joint density: conditional density of X

given θ is just the model f (x; θ).

Justifies notation f (x|θ).

Compute rπ a different way by factoring the

joint density a different way:
f (x|θ)π(θ) = π(θ|x)f (x)
where now f (x) is the marginal density of x
and π(θ|x) denotes the conditional density of θ
given X.

Call π(θ|x) the posterior density.

Found via Bayes theorem (which is why this is

Bayesian statistics):
f (x|θ)π(θ)
π(θ|x) = R
f (x|φ)π(φ)dφ
312
With this notation we can write
Z Z 
rπ (θ̂) = (θ̂(x) − θ)2 π(θ|x)dθ f (x)dx

Can choose θ̂(x) separately for each x to mini-

mize the quantity in square brackets (as in the
NP lemma).

Quantity in square brackets is a quadratic func-

tion of θ̂(x); minimized by
Z
θ̂(x) = θπ(θ|x)dθ

which is
E(θ|X)
and is called the posterior expected mean
of θ.

313
Example: estimating normal mean µ.

Imagine, for example that µ is the true speed

of sound.

I think this is around 330 metres per second

and am pretty sure that I am within 30 metres
per second of the truth with that guess.

I might summarize my opinion by saying that

I think µ has a normal distribution with mean
ν =330 and standard deviation τ = 10.

That is, I take a prior density π for µ to be

N (ν, τ 2).

Before I make any measurements best guess of

µ minimizes
1
Z
(µ̂ − µ)2 √ exp{−(µ − ν)2 /(2τ 2 )}dµ
τ 2π
This quantity is minimized by the prior mean
of µ, namely,
Z
µ̂ = Eπ (µ) = µπ(µ)dµ = ν .

314
Now collect 25 measurements of the speed of
sound.

Assume: relationship between the measure-

ments and µ is that the measurements are un-
biased and that the standard deviation of the
measurement errors is σ = 15 which I assume
that we know.

So model is: given µ, X1, . . . , Xn iid N (µ, σ 2).

The joint density of the data and µ is then

−n/1 −n
(Xi − µ)2/(2σ 2 )}
X
(2π) σ exp{−
×
(2π)−1/2 τ −1 exp{−(µ − ν)2/τ 2}.
Thus (X1 , . . . , Xn , µ) ∼ M V N . Conditional dis-
tribution of θ given X1 , . . . , Xn is normal.

Use standard MVN formulas to calculate con-

ditional means and variances.

315
Alternatively: exponent in joint density has
form
1h 2 2 2
i
− µ /γ − 2µψ/γ
2
plus terms not involving µ where
1 n 1
 
= +
γ2 σ2 τ2
and
ψ Xi ν
P
= + 2
γ2 σ2 τ
So: conditional of µ given data is N (ψ, γ 2).

In other words the posterior mean of µ is

n X̄ + 1 ν
σ2 τ2
n + 1
σ2 τ2
which is a weighted average of the prior mean
ν and the sample mean X̄.

Notice: weight on data is large when n is large

or σ is small (precise measurements) and small
when τ is small (precise prior opinion).
316
Improper priors: When the density does not
integrate to 1 we can still follow the machinery
of Bayes’ formula to derive a posterior.

Example: N (µ, σ 2 ); consider prior density

π(µ) ≡ 1.
This “density” integrates to ∞; using Bayes’
theorem to compute the posterior would give

π(µ|X) =
(2π)−n/2 σ −n exp{− (Xi − µ)2 /(2σ 2 )}
P

(2π)−n/2 σ −n exp{− (Xi − ν)2/(2σ 2 )}dν

R P

It is easy to see that this cancels to the limit

of the case previously done when τ → ∞ giving
a N (X̄, σ 2 /n) density.

I.e., Bayes estimate of µ for this improper prior

is X̄.

317
Admissibility: Bayes procedures correspond-
ing to proper priors are admissible. It follows
that for each w ∈ (0, 1) and each real ν the
estimate
wX̄ + (1 − w)ν
is admissible. That this is also true for w = 1,
that is, that X̄ is admissible is much harder to
prove.

Minimax estimation: The risk function of X̄

is simply σ 2 /n. That is, the risk function is
constant since it does not depend on µ. Were
X̄ Bayes for a proper prior this would prove
that X̄ is minimax. In fact this is also true but
hard to prove.

318
Example: Given p, X has a Binomial(n, p) dis-
tribution.

Give p a Beta(α, β) prior density

Γ(α + β) α−1
π(p) = p (1 − p)β−1
Γ(α)Γ(β)
The joint “density” of X and p is

n−X Γ(α + β) α−1

n
X
p (1 − p) p (1 − p)β−1 ;
X Γ(α)Γ(β)
posterior density of p given X is of the form

cpX+α−1 (1 − p)n−X+β−1
for a suitable normalizing constant c.

This is Beta(X + α, n − X + β) density.

319
Mean of Beta(α, β) distribution is α/(α + β).

So Bayes estimate of p is
X +α α
= wp̂ + (1 − w)
n+α+β α+β
where p̂ = X/n is the usual mle.

Notice: again weighted average of prior mean

and mle.

Notice: prior is proper for α > 0 and β > 0.

To get w = 1 take α = β = 0; use improper

prior
1
p(1 − p)
Again: each wp̂ + (1 − w)po is admissible for
w ∈ (0, 1).

Again: it is true that p̂ is admissible but our

theorem is not adequate to prove this fact.

320
The risk function of wp̂ + (1 − w)p0 is

R(p) = E[(wp̂ + (1 − w)p0 − p)2 ]

which is

w2 Var(p̂) + (wp + (1 − w)p − p)2

=
w2 p(1 − p)/n + (1 − w)2 (p − p0)2
Risk function constant if coefficients of p2 and
p in risk are 0.

Coefficient of p2 is

−w2/n + (1 − w)2
so w = n1/2/(1 + n1/2).

Coefficient of p is then

w2/n − 2p0 (1 − w)2

which vanishes if 2p0 = 1 or p0 = 1/2.

321
Working backwards: to get these values for w
and p0 require α = β. Moreover

w2/(1 − w)2 = n
gives
√
n/(α + β) = n
√
or α = β = n/2. Minimax estimate of p is
√
n 1 1
√ p̂ + √
1+ n 1 + n2

Example: X1, . . . , Xn iid M V N (µ, Σ) with Σ

known.

Take improper prior for µ which is constant.

Posterior of µ given X is then M V N (X̄, Σ/n).

322
Multivariate estimation: common to extend
the notion of squared error loss by defining

(θ̂i − θi)2 = (θ̂ − θ)t(θ̂ − θ) .

X
L(θ̂, θ) =
For this loss risk is sum of MSEs of individual
components.

Bayes estimate is again posterior mean. Thus

X̄ is Bayes for an improper prior in this prob-
lem.

It turns out that X̄ is minimax; its risk function

is the constant trace(Σ)/n.

If the dimension p of θ is 1 or 2 then X̄ is also

admissible but if p ≥ 3 then it is inadmissible.

Fact first demonstrated by James and Stein

who produced an estimate which is better, in
terms of this risk function, for every µ.

So-called James Stein estimator is essentially

never used.
323
Hypothesis Testing and Decision Theory

Decision analysis of hypothesis testing takes

D = {0, 1} and

L(d, θ) = 1(make an error)

or more generally L(0, θ) = `11(θ ∈ Θ1) and
L(1, θ) = `21(θ ∈ Θ0) for two positive con-
stants `1 and `2. We make the decision space
convex by allowing a decision to be a probabil-
ity measure on D. Any such measure can be
specified by δ = P (reject) so D = [0, 1]. The
loss function of δ ∈ [0, 1] is

L(δ, θ) = (1 − δ)`11(θ ∈ Θ1) + δ`01(θ ∈ Θ0) .

324
Simple hypotheses: Prior is π0 > 0 and π1 >
0 with π0 + π1 = 1.

Procedure: map from sample space to D – a

test function.

Risk function of procedure φ(X) is a pair of

numbers:

Rφ(θ0 ) = E0(L(δ, θ0 ))
and
Rφ(θ1 ) = E1(L(δ, θ1 ))
We find

Rφ (θ0) = `0E0(φ(X)) = `0α

and

Rφ (θ1) = `1E1(1 − φ(X)) = `1β

325
The Bayes risk of φ is

π0 ` 0 α + π 1 ` 1 β
We saw in the hypothesis testing section that
this is minimized by

φ(X) = 1(f1 (X)/f0 (X) > π0`0/(π1 `1))

which is a likelihood ratio test. These tests are
Bayes and admissible. The risk is constant if
β`1 = α`0; you can use this to find the minimax
test in this context.

326
Hypothesis Testing and Decision Theory

Decision analysis of hypothesis testing takes

D = {0, 1} and

L(d, θ) = 1(make an error)

or more generally L(0, θ) = `11(θ ∈ Θ1) and
L(1, θ) = `21(θ ∈ Θ0) for two positive con-
stants `1 and `2. We make the decision space
convex by allowing a decision to be a probabil-
ity measure on D. Any such measure can be
specified by δ = P (reject) so D = [0, 1]. The
loss function of δ ∈ [0, 1] is

L(δ, θ) = (1 − δ)`11(θ ∈ Θ1) + δ`01(θ ∈ Θ0) .

327
Simple hypotheses: Prior is π0 > 0 and π1 >
0 with π0 + π1 = 1.

Procedure: map from sample space to D – a

test function.

Risk function of procedure φ(X) is a pair of

numbers:

Rφ(θ0 ) = E0(L(δ, θ0 ))
and
Rφ(θ1 ) = E1(L(δ, θ1 ))
We find

Rφ (θ0) = `0E0(φ(X)) = `0α

and

Rφ (θ1) = `1E1(1 − φ(X)) = `1β

328
The Bayes risk of φ is

π0 ` 0 α + π 1 ` 1 β
We saw in the hypothesis testing section that
this is minimized by

φ(X) = 1(f1 (X)/f0 (X) > π0`0/(π1 `1))

which is a likelihood ratio test. These tests are
Bayes and admissible. The risk is constant if
β`1 = α`0; you can use this to find the minimax
test in this context.

329