Unit 3 Minimum Variance

Unbiased Estimator

Institute of Communications Engineering, EE, NCTU 1

 Sufficient Statistics

Institute of Communications Engineering, EE, NCTU 2

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Is there a set of data that is sufficient for estimation?

 What do we mean by sufficiency for a particular objective?
 Suppose Â1= x[n]/N, then are the followings sufficient?
 S1 = {x[0], x[1],…, x[N-1]}
 S2 = {x[0]+x[1], x[2],…, x[N-1]}
 S3 = x[n]
 Then, what is the minimum set of sufficient statistics?
 We want to have a set of data T(x) such that given T(x), any
individual data x(n) is statistically independent of A
 Givenx[n] = T0 , do we still need the individual data?

Institute of Communications Engineering, EE, NCTU 3

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 We say the conditional PDF : p( x | x[n] = T0 ; A)

should not depend on A if T0 is sufficient
 E.g.

 For (a), a value of A near A0 is more likely even given T0

 For (b), however, p( x | x[n] = T0 ; A) is a constant

Institute of Communications Engineering, EE, NCTU 4

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Now, we need to determine p( x | x[n] = T0 ; A) to

show x[n] = T0 is sufficient
 By Baye’s rule

 Since T(x) is a direct function of x,

 Clearly, we have

Institute of Communications Engineering, EE, NCTU 5

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Thus, we have

Institute of Communications Engineering, EE, NCTU 6

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Since T(x) = x[n] ~ N(NA, N2)

 Thus

which does not depend on A

Institute of Communications Engineering, EE, NCTU 7

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 In general, to identify potential sufficient statistics is

difficult
 An efficient procedure of finding the sufficient statistics
is to employ the Neyman-Fisher factorization theorem
 Observe that

 If we can factorize p(x;) into p(x;)=g(T(x),) h(x)

where
 g is a function depends on x only through T(x)
 h is a function depends only on x
  T(x) is a sufficient statistics for 
 The converse is also true
If T(x) is a sufficient statistics  p(x;)=g(T(x),) h(x)

Institute of Communications Engineering, EE, NCTU 8

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Recall p(x;A)

 On the other hand, we want to estimate 2 of y[n]=A+x[n]

 Suppose A is given, then define x[n] = y[n]-A

 Clearly, T(x) = x2[n] is a sufficient statistics for 2

Institute of Communications Engineering, EE, NCTU 9

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Ex. we want to estimate the phase of a sinusoid

x[n]=A cos(2 f0 n + ) + w[n], n=0,1,…,N-1
 Suppose A and f0 are given

 Expand the exponent

Institute of Communications Engineering, EE, NCTU 10

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 In this case, no single sufficient statistics exists, however

g(T1(x),T2 (x),  )

h(x)

Institute of Communications Engineering, EE, NCTU 11

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Proof of the Neyman-Fisher Factorization Theorem

 ()

 By assumption, p(x;) = g(T(x), )h(x)

 Since

Institute of Communications Engineering, EE, NCTU 12

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 We have

 Then

which does not depend on 

 Hence T(x) is a sufficient statistic

Institute of Communications Engineering, EE, NCTU 13

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 ()
 Suppose T(x) is a sufficient statistic
 Since

 And

 As p(x| T(x)=T0 ;) = P(x| T(x)=T0) ( not a function of )

 We define p(x| T(x)=T0 ;) = w(x)(T(x) - T0) with
 w(x)(T(x) - T0) =1
 Then

Institute of Communications Engineering, EE, NCTU 14

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 As a result, we can let

such that

 This results in

where

Institute of Communications Engineering, EE, NCTU 15

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 The r statistics T1(x), T2(x),…, Tr(x) are jointly sufficient

if p(x | T1(x), T2(x),…, Tr(x) ; ) does not depend on 
 If p(x ; ) = g(T1(x), T2(x),…, Tr(x) , ) h(x)
 {T1(x), T2(x),…, Tr(x)} are sufficient statistics for 
 Now, we know how to obtain the sufficient statistics
 How do we apply them to help obtain the MVU estimator?
 The Rao-Blackwell-Lehmann-Scheffe (RBLS) Theorem
 If is an unbiased estimator of  and T(x) is a sufficient
statistic for , then is unbiased and
 A valid estimator for  (not dependent on )
 Of lesser or equal variance than that of for all 
 If T(x) is complete, then is the MVU estimator

Institute of Communications Engineering, EE, NCTU 16

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Proof
 1> validity

by definition p(x|T(x) ; ) is not a function of   after the

integration w.r.t. x, the result is not a function of  but T
 2> unbiasedness

By assumption

Institute of Communications Engineering, EE, NCTU 17

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 3> show

0 0
 Recall that is solely a function of T(x)

 Also

Institute of Communications Engineering, EE, NCTU 18

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Finally, a statistic is complete if there is only one function,

say g, of the statistic that is unbiased
 is solely a function of T(x)
 If T(x) is complete  is unique and unbiased
 Then is the same for any unbiased
 Besides, for any unbiased estimator
 Then, must be the MVU
 In summary, the MVU can be found by
 Taking any unbiased and carrying out
 Alternatively, since there is only one function of T(x) that
leads to an unbiased estimator
 find the unique g(T(x)) that makes unbiased

Institute of Communications Engineering, EE, NCTU 19

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Completeness of a sufficient statistic

 Suppose our problem is to estimate A from x[n] = A + w[n],
then T(x) =  x[n] is complete and g(T(x)) = T(x)/N
 Suppose,  a second function h for which E{h(T(x))} = A
 E{g(T(x)) -h(T(x))} = A – A =0,  A
 Define v(T) = g(T(x)) -h(T(x)). We want to show v(T) = 0,
given E{v(T)} = 0. Since T ~ N(NA, N2) 

 Let  = T /N and v’() = v(N )

W()
Institute of Communications Engineering, EE, NCTU 20
Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Which is a convolution of v’() and w() and =0 A  v’()=0

 Recall a signal v(t) is zero iff F(v(t)) = 0

  F(v’()*w()) = V’(f)W(f) = 0,  A
 Since W(f) is still Gaussian  V’(f) = 0  v’() = 0,  
  g(T(x)) = h(T(x))

Institute of Communications Engineering, EE, NCTU 21

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Incomplete sufficient statistic

 Now consider x[0] = A + w[0], while w[0] ~ U[-1/2, 1/2]
 T(x) = x[0] is sufficient. But, is T(x) completely sufficient?
 Let v(T) = g(T(x)) -h(T(x))

 However, x = x[0] = T, so that

but

Institute of Communications Engineering, EE, NCTU 22

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 So that

 A nonzero v(T) = sin (2T) will satisfy this condition

Institute of Communications Engineering, EE, NCTU 23

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Hence, v(T) = g(T) – h(T) = sin (2T)

or h(T) = T - sin (2T)
  Â = x[0] - sin (2 x[0]) is also unbiased for A
  x[0] is not complete,
 not sure if  = x[0] is an MVU estimator
 To summarize, we say a sufficient statistic is complete if

is satisfied only by the zero function or by v(T) = 0,  T

Institute of Communications Engineering, EE, NCTU 24

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 As a summary, the RBLS method can be used to find the

MVU even when an efficient estimator does not exist
 The procedure we learn by now to find an MVU estimator
 1> Find a sufficient statistic T(x) for 
by the Neyman-Fisher factorization theorem
 2> Determine if T(x) is complete, if so,
 3> Find a function g(T(x)) that yields an unbiased
estimation of  which is the MVU of 
 Alternatively,

where is an unbiased estimator

Institute of Communications Engineering, EE, NCTU 25

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Ex: Mean of uniform noise

 x[n] = w[n] , n=0,1,…,N-1, w[n] ~ U[0, ]
 Want to find the MVU estimator for the mean  = /2
 The initial approach of using the CRLB to find an efficient
estimator cannot even be tried for it does not satisfy the
regularity condition : E{ ln P(x;) /  } = 0
 Is a sample mean the MVU estimator for this case?

 Its variance is

Institute of Communications Engineering, EE, NCTU 26

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Now, we follow the procedure we learned so far

 Define

 Then

where =2
 The PDF is

or

Institute of Communications Engineering, EE, NCTU 27

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Alternatively

so that

 T(x) = max(x)
 We need to determine a function g to make T(x) unbiased
 To do so requires us to determine E{T(x)}

Institute of Communications Engineering, EE, NCTU 28

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 The CDF of T(x)

 The PDF follows as

Institute of Communications Engineering, EE, NCTU 29

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 But d Pr{x[n] < }/d  is the PDF of x[n] or

 Integrating, we obtain

Institute of Communications Engineering, EE, NCTU 30

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Which finally yields

 We now have

Institute of Communications Engineering, EE, NCTU 31

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 To make it unbiased, we multiply T(x) by (N+1)/N



which is the MVU estimator whose corresponding

variance is

and

 < 2/(12N) of the sample mean

Institute of Communications Engineering, EE, NCTU 32

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Extension to a vector parameter 

 We want to seek an unbiased vector estimator such that
each element has minimum variance
 Similarly, a vector T(x) = [T1(x), T2(x)…, Tr(x)]T is said
to be sufficient for the estimation of 
 If P(x| T(x) ; ) = P(x| T(x))
 If P(x ; ) =g(T(x), ) h(x)  T(x) is of minimum
dimension

Institute of Communications Engineering, EE, NCTU 33

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Back to a similar example

 x[n]=A cos(2 f0 n) + w[n], n=0,1,…,N-1
 Now,  =[A, f0 , 2 ]T is the unknown vector parameter
 The PDF is

 Expanding the exponent, we obtain

 Cannot reduce the PDF due to

Institute of Communications Engineering, EE, NCTU 34

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 If f0 is known, then  =[A, 2 ]T

 We are able to factorize the PDF which gives

 N 1 

 T1 ( x )   n 0
x[ n ]cos(2  f 0 n ) 
T ( x)     
T2 ( x )   
N 1




n 0
2
x [n ] 


Institute of Communications Engineering, EE, NCTU 35

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 Now apply the above result to our previous discussion

 x[n]=A + w[n], n=0,1,…,N-1
 Again,  =[A, 2 ]T
 Set f0 = 0 for x[n]=A cos(2 f0 n) + w[n], we have
 N 1 
 T1 ( x )    x[n ] 
T ( x)      n 0

T2 ( x )   x 2 [n ]
N 1

 
 n 0 
 Taking the expected values produces
 NA   NA 
E{T ( x )}     2 
 NE { x 2
[ n ]}  N ( 2
 A )

Institute of Communications Engineering, EE, NCTU 36

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 So T2(x) only helps estimate the second moment not the

variance
 If we transform T (x) into
 1 
 T1 ( x )   x 
N  
g (T ( x ))      1 N 1
2

2
1 1   x 2
[ n ]  x
 N T2 ( x )   N T1 ( x )    N n 0

   
 Then, E{g(T(x))} gives
 E x  
   A 
E{g (T ( x ))}    1 N 1 2 2 
 2 2 
E   x [n ]  x    A  E  x 
2
  N n 0  
Institute of Communications Engineering, EE, NCTU 37
 Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

E x   A   2
 Since x ~ N (A, 2/N)  2 2
N

 Substituting this back into E{g(T(x))} yields

 A   A 
E{g (T ( x ))}   2 2 
 2 
  A  E  x    A  A  N 
2 2 2  2

 Therefore, multiply the second element by N/(N-1) yields

an unbiased estimator of 2
 x   1 T ( x) 
N 1


 g (T ( x ))   1  N 1 2   1  2 
2  
 N  1  
x [ n ]  Nx  1 
  T2 ( x )  N  T1 ( x )   
n 0    N  1  N   

Institute of Communications Engineering, EE, NCTU 38
Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

N 1 N 1
Since
  x[ n ]  x    x 2 [n ]  Nx 2
 2

n 0 n 0

 Eventually, we have
 x   x 
 ˆ  g (T ( x ))   1  N 1 2 

1 N 1 
 2      
2
x [ n ]  Nx x[ n ]  x
 N  1  n 0   N  1
  n 0 
 Is this the MVU of  =[A, 2 ]T ?
 We have shown that for Gaussian PDF, T(x) is complete
 Actually, this is also true for the vector exponential
family of PDFs
 Is ˆ efficient ?

Institute of Communications Engineering, EE, NCTU 39

Unit 3: Minimum Variance Unbiased Estimator Sau-Hsuan Wu

 In fact  and are independent with

 Therefore

 While, the CRLB is

Institute of Communications Engineering, EE, NCTU 40