Complete Statistics

February 4, 2016 Debdeep Pati

1 Complete Statistics

Suppose X ∼ Pθ , θ ∈ Θ. Let (X(1) , . . . , X(n) ) denote the order statistics.

Definition 1. A statistic T = T (X) is complete if

Eθ g(T ) = 0 for all θ

implies

Pθ (g(T ) = 0) = 1 for all θ.

(Note: Eθ denotes expectation computed with respect to Pθ ).

Example: X = (X1 , . . . , Xn ) iid N(θ, 1). T (X) = (X1 , X2 ) is a statistic which is not
complete because

E(X1 − X2 ) = 0 for all θ.

and X1 − X2 here is a function of T where g : R2 → R given by (x1 , x2 ) 7→ x1 − x2 but

P (X1 − X2 = 0) 6= 1 for all θ.
More formally: T is not complete because the function g satisfies

Eθ g(T ) = 0 for all θ but

Pθ (g(T ) = 0) 6= 1 for all θ.

Example: X = (X1 , . . . , Xn ) iid Unif(θ, θ + 1). T = T (X) = (X(1) , X(n) ) is MSS. But T is
not complete. We know that S(X) = X(n) − X(1) is ancillary. Thus

E(X(n) − X(1) ) = c

where c does not depend on θ and therefore,

E(X(n) − X(1) − c) = 0 for all θ

but clearly,

P (X(n) − X(1) − c = 0) 6= 1 for all θ.

Hence taking g(T ) = X(n) − X(1) − c, we can see that T is not complete.
Example: X = X1 , . . . , Xn iid Unif(0, θ) T = T (X) = X(n) is MSS.
Fact: T is also complete.

1
Proof. Assume there exists g such that

Eθ g(T ) = 0 for all θ.

T has cdf H(t) = t/θ)n , 0 ≤ t ≤ θ and pdf ntn−1 /θn , 0 ≤ t ≤ θ. Then

Z θ
ntn−1
Eg(T ) = g(t) n dt = 0 for all θ > 0.
0 θ
implies
Z θ
g(t)ntn−1 dt = 0 for all θ > 0.
0

implies (by differentiating both sides and using the Fundamental Theorem of Calculus)

g(θ)nθn−1 dt = 0 for all θ > 0.

implies

g(t) = 0 for all t > 0.

implies

Pθ (g(T ) = 0) = 1 for all θ > 0.

Theorem 1. Suppose X1 , . . . , Xn iid with pdf (pmf )

k
X 
f (x | θ) = c(θ)h(x) exp wj (θ)tj (x)
j=1

for θ = (θ1 , . . . , θk ) ∈ Θ. Let X = (X1 , . . . , Xn ). Define

X n Xn n
X 
T (X) = t1 (Xi ), t2 (Xi ), . . . , tk (Xi )
i=1 i=1 i=1

Then

1. T (X) is sufficient statistic for θ.

2. If Θ contains an open set in Rk , then T (X) is complete. (More precisely, if

{(w1 (θ), w2 (θ), . . . , wk (θ)) : θ ∈ Θ}

contains an open set in Rk , then T (X) is complete.)

2
Remarks:

1. The statistic T (X) in the Theorem is called the natural sufficient statistic.

2. η = (η1 , . . . , ηk ) ≡ (w1 (θ), . . . , wk (θ)) is called the natural parameter of the exponen-
tial family.

3. Condition (2) is the “open set condition” (OSC). The OSC is easily verified by in-
spection. Let A ⊂ Rk . A contains an open set in Rk iff A contains a k-dimensional
ball. That is, there exists x ∈ Rk and r > 0 such that B(x, r) ⊂ A. Here B(x, r)
denotes the ball of radius r about x. Let A ⊂ R (take k = 1). A contains an open
set in R if and only if A contains an interval. That is, there exists c < d such that
(c, d) ⊂ A.

Facts:

1. Under weak conditions (which are almost always true, a complete sufficient statistic
is also minimal. Abbreviation: CSS ⇒ MSS. (but MSS does not imply CSS as we
saw earlier).

2. A one-to-one function of a CSS is also a CSS (See later remarks). Reminder: A 1-1
function of an MSS is also an MSS.

Example: The Bernoulli pmf is an exponential family (1pef):

  
x 1−x θ
p(x | θ) = θ (1 − θ) , x ∈ {0, 1} = (1 − θ)I(x ∈ {0, 1}) exp x log .
1−θ
P
If X1 , . . . , Xn are iid p(x | θ), then T = i Xi is a SS. It is also complete if Θ contains
an interval. Here is a direct proof of completeness (not relying on our general theorem on
exponential families).

Proof. We know T ∼ Binomial(n, θ). If Eg(T ) = 0 for all θ ∈ Θ ⊂ (0, 1), then
n  
X n k
g(k) θ (1 − θ)n−k = 0
k
k=0

for all θ ∈ Θ. Then

n   k
n
X n θ
(1 − θ) g(k) =0
k 1−θ
k=0

3
for all θ ∈ Θ. Then
n
X
ak uk = 0
k=0

n


for all θ ∈ Θ where ak = g(k) k and u = θ/(1 −Pθ). This if Θ contains an interval,
then the above implies that the polynomial ψ(u) = nk=1 ak uk is identically zero in some
interval. This implies all the coefficients ak must be zero, which further implies g(k) = 0
for k = 0, 1, . . . , n so that Pθ (g(T ) = 0) = 1 for all θ ∈ Θ.

In a homework exercise, you show that T = ni=1 Xi is complete when X1 , . . . , Xn are

P
iid Poisson(λ) in a similar way, using the fact that an infinite power series (an analytic
function) is identically zero in some interval if and only if all the coefficients are zero.
Example: The N(θ, 1) Pfamily is a 1pef with w(θ) = θ, t(x) = x. Let X = (X1 , . . . , Xn )
iid N(θ, 1). T (X) = ni=1 Xi is the natural SS (It is sufficient for any Θ). Is T complete?
This depends on Θ.

1. Θ = R. Yes. (OSC holds)

2. Θ = [0.01, 0.02]. Yes. (OSC holds)

3. Θ = (1, 2) ∪ {4, 7}. Yes. (OSC holds)

4. Θ = Z (the integers). OSC fails so Theorem says nothing. But can show that it is
not complete.

5. Θ = Q (the rationals). OSC fails so Theorem says nothing. Yes or no? Don’t know.

6. Θ = Cantor set. OSC fails so Theorem says nothing. Yes or no? Don’t know.

7. Θ = finite set. OSC fails so Theorem says nothing. But can show that it is not
complete.

Remark: In general, it is typically true that if Θ is finite and the support of T = T (X) is
infinite, then T is not complete.
Example: The N(µ, σ 2 ) family with θ = (µ, σ 2 ) is a 2pef with
 
µ −1
w(θ) = , , t(x) = (x, x2 ).
σ 2 2σ 2

Let X = (X1 , . . . , Xn ) iid N(µ, σ 2 ). Then T (X) = ( ni=1 Xi , ni=1 Xi2 ) is the natural SS.
P P
(It is a SS for any Θ). T (X) is a one-to-one function of U (X) = (x̄, s2 ). So T is CSS iff U
is CSS. Is T (or U ) complete? That depends on Θ.

4
 1. Θ1 = {(µ, σ 2 ) : σ 2 > 0}. OSC holds. Yes, complete.

2. Θ2 = {(µ, σ 2 ) : σ 2 = σ02 }. OSC fails. Theorem says nothing. But we can prove that
U is not complete.

Proof. Let g(x1 , x2 ) = x2 − σ02 . Then Eg(U ) = E(s2 − σ02 ) = σ 2 − σ02 = 0 for all
θ ∈ Θ2 .

3. Θ3 = {(µ, σ 2 ) : µ = µ0 , σ 2 > 0}. OSC fails. Theorem says nothing. But we can
prove that U is not complete.

Proof. Let g(x1 , x2 ) = x1 − µ0 . Then Eg(U ) = E(x̄ − µ0 ) = µ − µ0 = 0 for all

θ ∈ Θ3 .

4. Θ4 = {(µ, σ 2 ) : µ = σ 2 , σ 2 > 0}. OSC fails. Theorem says nothing. But we can
prove that U is not complete.

Proof. Let g(x1 , x2 ) = x1 − x2 . Then Eg(U ) = E(x̄ − s2 ) = µ − σ 2 = 0 for all

θ ∈ Θ4 .

(Note: It is more natural to describe the families Θ2 , Θ3 , Θ4 as 1pef’s. If you do this,

you get different natural sufficient statistics, which turn out to be complete.)

5. Θ5 = {(µ, σ 2 ) : µ2 = σ 2 , σ 2 > 0}. OSC fails. Theorem says nothing. But we can
prove that U is not complete.

Proof. Homework

6. Θ6 = [1, 3] × [4, 6]. OSC holds. Yes, complete.

7. Θ7 = Θ6 ∪ {(5, 1), (4, 2)}. OSC holds. Yes, complete.

8. Θ8 = complicated wavy curve. OSC fails. Theorem says nothing. But hard to
conclude anything.

Corollary 1. Suppose X ∈ Rm has joint pdf (pmf )

k
X 
f (x | θ) = c(θ)h(x) exp wj (θ)tj (x)
j=1

for all x ∈ Rm where θ = (θ1 , . . . , θk ) ∈ Θ. Define T (X) = (t1 (X), t2 (X), . . . , tk (X)).
Then

5
 1. T (X) is sufficient statistic for θ.

2. If Θ contains an open set in Rk , then T (X) is complete.

(More precisely, if

{(w1 (θ), w2 (θ), . . . , wk (θ)) : θ ∈ Θ}

contains an open set in Rk , then T (X) is complete.)

Example: Return to Simple Linear Regression: X1 , . . . , Xn independent with Xi ∼ N(β0 +

β1 zi , σ 2 ). θ = (β0 , β1 , σ 2 ) and Θ = R2 × (0, ∞). Recall that the joint density of X =
(X1 , . . . , Xn ) is
 X n n n 
1 X X X
f (x | θ) = (2πσ 2 )−n/2 exp − x2i − 2β0 xi − 2β1 z i xi + (β0 + β1 zi )2
2σ 2
i i=1 i=1 i=1
  
2 −n/2 1 X 2
= (2πσ ) exp − 2 (β0 + β1 zi ) ×1
2σ
i
 
−1 X 2 β0 X β1 X
× exp x i + x i + z x
i i
2σ 2 σ2 σ2
i i i
X 3 
= c(θ)h(x) exp wj (θ)tj (x)
j=1

where
 
−1 β0 β1
w(θ) = (w1 (θ), w2 (θ), w3 (θ)) = , , ,
2σ 2 σ 2 σ 2
n
X n
X Xn 
2
t(x) = (t1 (x), t2 (x), t3 (x)) = xi , xi , z i xi .
i=1 i=1 i=1

The data vector X may be regarded as a single observation P from an n-dimensionalP 3pef.
Since Θ ⊂ R3 satisfied the OSC, the statistic T (X) = t(X) = ( ni=1 Xi2 , ni=1 Xi , ni=1 zi Xi )
P
is a CSS.
Notation: X ∼ Pθ , θ ∈ Θ.S(X) = ψ(T (X)) for some ψ. Θ1 ⊂ Θ2 ⊂ Θ.
Sufficiency:

1. If S(X) is sufficient, then T (X) is sufficient.

2. If T (X) is sufficient for Θ2 , then T (X) is sufficient for Θ1 .

Completeness:

6
 1. If T (X) is complete, then S(X) is complete.

2. If T (X) is complete for Θ1 , then T (X) is complete for Θ2 (under mild regularity
conditions).

Proof. Proof of 1. Eθ g(S(X)) = 0 for all θ ∈ Θ =⇒ Eθ g(ψ(T (X))) = 0 for all θ ∈

Θ =⇒ Pθ {g(ψ(T (X))) = 0} = 1 for all θ ∈ Θ (by completeness of T (X)) which implies
Pθ {g(S(X)) = 0} = 1 for all θ.
Proof of 2. Eθ g(T (X)) = 0 for all θ ∈ Θ2 =⇒ Eθ g(T (X)) = 0 for all θ ∈ Θ1 =⇒
Pθ {g(T (X)) = 0} = 1 for all θ ∈ Θ1 (by completeness of T (X) for Θ1 ) which implies
Pθ {g(T (X)) = 0} = 1 for all θ ∈ Θ2 (under mild assumptions).

Ancillarity:

1. If T (X) is ancillary, then S(X) is ancillary.

2. If T (X) is ancillary for Θ2 , then T (X) is ancillary for Θ1 .

d d
Proof. Proof of 1. Uses Y = Z =⇒ ψ(Y ) = ψ(Z).
Proof of 2. Trivial

2 Basu’s results

Suppose X ∼ Pθ , θ ∈ Θ.

Lemma 1. (Basu’s Lemma) If T (X) is complete and sufficient (for θ ∈ Θ), and S(X) is
ancillary, then S(X) and T (X) are independent for all θ ∈ Θ.

In other words, a complete sufficient statistic is independent of any ancillary statistic.

2.1 Remarks:

Let S = S(X), T = T (X). Let Eθ denote expectation w.r.t. Pθ .

1. The joint distribution of (S, T ) depends on θ, so in general it is possible for S and T

to be independent for some values of θ, but not for others. (Basu’s Lemma says this
does not happen in this case.)

7
 2. For any rv’s Y and Z, we know that E(Y | Z) = g(Z), i.e., the conditional expectation
is a function of Z. If the joint distribution of (Y, Z) depends on a parameter θ, then
Eθ (Y | Z) = g(Z, θ), i.e., the conditional expectation is a function of both Z and θ.
(However, this function may turn out to be constant in one or both variables.)

3. In general, E(Y ) = E{E(Y | Z)} and Eθ (Y ) = Eθ Eθ (Y | Z).

4. To show that Y and Z are independent, it suffices to show that L(Y | Z) = L(Y )
which means that P (Y ∈ A | Z) = P (Y ∈ A) for all (Borel) sets A. Let w(Y ) =
I(Y ∈ A). Then P (Y ∈ A) = Ew(Y ) and P (Y ∈ A | Z) = E(w(Y ) | Z). The
indicator function w(Y ) is a bounded (Borel measurable) function. Therefore we
have:
To show that Y and Z are independent, it suffices to show E(w(Y ) | Z) = Ew(Y )
for all bounded (Borel measurable) functions.

5. Thus, to show that S and T are independent for all θ, it suffices to show that
Eθ (w(S) | T ) = Eθ w(S) for all θ and all bounded measurable functions w(S).

2.2 Proof of Basu’s Lemma

Proof. Let w(S) be a given bounded function of S. Consider both sides of the identity:

Eθ w(S) = Eθ [Eθ (w(S) | T )]

for all θ. Consider the LHS. Since S is ancillary, the distribution of w(S) is the same for
all θ so that the LHS is constant in θ. Call this constant c.
Now consider the RHS. We know that Eθ (w(S) | T ) will be some function of θ and T .
However, since T is a sufficient statistic, L(X | T ) does not depend on θ. Since S = S(X),
this implies L(S | T ) does not depend on θ so that in turn L(w(S) | T ) does not depend
on θ. Thus, by sufficiency, Eθ (w(S) | T ) is constant in θ and must be a function of T only.
Call this function ψ(T ). That is,

ψ(T ) = Eθ (w(S) | T ).

The original identity can now be written as

c = Eθ ψ(T ) for all θ or equivalently

0 = Eθ (ψ(T ) − c) for all θ.

Since T is complete, this implies

P (ψ(T ) − c = 0) = 1 for all θ or equivalently

ψ(T ) = c with probability one for all θ.

8
This means

Eθ (w(S) | T ) = Eθ w(S) with probability one for all θ.

Since w(S) is an arbitrary bounded function, by the earlier discussion this implies S and
T are independent for all θ.

2.3 Applications of Basu’s theorem

1. Example: Let X = (X1 , . . . , Xn ) iid Unif(0, θ). Recall: T (X) = X(n) = max{Xi } is
a CSS. Unif(0, θ) is a scale family, so any scale invariant statistic S(X) is ancillary.
Thus, by Basus lemma, all of the following are independent of X(n) for all θ:

X(1) X(1) X(n−1)

 
x̄
, , ,...,
s X(n) X(n) X(n)

2. Example: (Using Basu’s Lemma to obtain an indirect proof of completeness.) Let

X = (X1 , . . . , Xn ) iid Unif(θ, θ + 1). Recall: T (X) = (X(1) , X(n) ) is a MSS. S(X) =
X(n) − X(1) is ancillary. Since S is a function of T , the rvs S and T cannot be
independent. Thus T cannot be complete (for then we would get a contradiction
with Basus Lemma).

3. Example: Let X = (X1 , . . . , Xn ) be iid N(µ, σ 2 ). Let x̄, s2 , Z be the sample mean,
sample variance, and standardized residuals (z-scores) of the data X (Recall: Z =
(Z1 , . . . , Zn ) with Zi = (Xi − x̄)/s.
Fact: x̄, s2 , Z are mutually independent.

Proof. We first show that the pair (x̄, s2 ) is independent of Z, and then show that x̄
and s2 are independent. Each stage uses Basu’s Lemma.
Stage 1: (x̄, s2 ) is independent of Z: Consider the family of all N (µ, σ 2 ) distribu-
tions (with both parameters allowed to vary). Recall: (x̄, s2 ) is a CSS. This is a
location-scale family so that any location-scale invariant statistic is ancillary. Z is
location-scale invariant. Thus, Z is ancillary so that (by Basu’s Lemma) it must be
independent of (x̄, s2 ) for all (µ, σ 2 ).
Stage 2: x̄ and s2 are independent: Fix σ 2 at an arbitrary value σ02 and consider the
family 2
P of N(µ, σ0 ) distributions, µ unknown. Recall: This is a 1pef and the natural
SS i Xi is a CSS. x̄ is a 1-1 function of this and so also a CSS. This is a location
family so that any location invariant statistic is ancillary. s2 is location invariant.
Thus, s2 is ancillary so (by Basu’s Lemma) it must be independent of x̄ for all µ (and
also for all σ 2 since σ02 is arbitrary).

9
4. Example: (an amusing calculation via Basu) Let X = (X1 , . . . , Xn ) iid N(0, σ 2 ), σ 2 >
0.
Goal: Compute ES where

Xi )2
P
(
S = Pi 2
i Xi

This is a 1pef with θ = σ 2 , t(x) = x2 and w(θ) = −1/(2σ 2 ). Therefore T (X) = i Xi2
P
is CSS. This is also a scale family so that scale invariant statistics are ancillary. S
is scale invariant implies S ancillary which implies (by Basu) S independent of T .
Thus E(ST ) = (ES)(ET ) which implies ES = E(ST )/ET which becomes

E( i Xi )2 nσ 2
P
ES = P 2 = = 1.
E i Xi nσ 2

10