Detection and Estimation Theory

Sau-Hsuan Wu
Institute of Communications Engineering
National Chiao Tung University

Institute of Communications Engineering, EE, NCTU 1

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Detection theory
 Determine whether or not an event of interest occurs
 E.g. Whether or not an aircraft occurs
 Estimation theory
 Determine the values of parameters pertaining to an event
of interest
 E.g. The altitude and position of an aircraft

Institute of Communications Engineering, EE, NCTU 2

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Under what basis do we do detection and estimation

 The signal source model and the observations

Institute of Communications Engineering, EE, NCTU 3

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Another application of detection and estimation theory

 Determine the existence of a submarine

 Estimate the bearing

Institute of Communications Engineering, EE, NCTU 4

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Many more applications

 Biomedicine: estimate the heart rate or even heart diseases
 Image analysis: estimate the size, position and orientation of
an object in an image
 Seismology: estimate the underground distance of an oil
deposit
 Communications: estimate the signal distortion and
determine the transmitted symbols
 Economics: estimate the Don-Jones industrial average

Institute of Communications Engineering, EE, NCTU 5

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 The mathematical estimation problem

 We want to know (or estimate) an unknown quantity 
 This quantity, , may not be obtained directly, or it is a
notion (quantity) derived from a set of observations
 E.g. the Don-Jones average

 We seem to have a model x[n]=A + Bn + w[n], n=0,…,N-1

Institute of Communications Engineering, EE, NCTU 6

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 The observations are corrupted by noise w[n]

 The noise may be modeled as a white Gaussian noise N(0,2)

 We want to determine  = [A, B] from the set of corrupted
observations x = x[0],…, x[N-1] which can be modeled as

P(x ; A, B) =

Institute of Communications Engineering, EE, NCTU 7

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Our goal is to determine  according to the observations,

namely
 Examples:
 x[n] = A1 + w[n], n=0,…,N-1
 x[n] = A2 + B2 n + w[n], n=0,…,N-1
 Â1= x[n]/N
 and ? Can we try the least square method
 How many other kinds of methods can we use
 Minimum variance unbiased estimator (MVU)
 Best linear unbiased estimator (BLUE)
 Maximum likelihood estimator (ML)
 Bayesian estimator: MMSE, MAP, and …

Institute of Communications Engineering, ECE, NCTU 8

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 The mathematical detection problem

 We want to determine the presence or absence of an object
 Similarly, we have some observations x[n], n=0,…,N-1
 Based on our problem setting, x[n] can be modeled as
H0 : x[n] = s0 + w[n]
H1 : x[n] = s1 + w[n]

Institute of Communications Engineering, ECE, NCTU 9

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 In this special case, s0 = 0 and s1 = 1

 Then, we have two hypotheses for x[n] in this case
H0 : x[n] = w[n]
H1 : x[n] = 1 + w[n]
 Suppose the noise w[n] ~ N(0, 2)
 And we have one observation only

Institute of Communications Engineering, ECE, NCTU 10

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 A reasonable approach is to decide H1 if x[0] > 1/2

 Type I error: We decide H1 when H0 is true (False Alarm)
 Type II error: We decide H0 when H1 is true (Miss)

Institute of Communications Engineering, ECE, NCTU 11

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 As the threshold  increases, one error increases and the

other decreases

 P(H0 ; H1) = 1- P(H1 ; H1)

Institute of Communications Engineering, ECE, NCTU 12

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 In general, we have multiple observations x[n], n=0,…,N-1

H0 : x[n] = w[n]
H1 : x[n] = A + w[n]
 We would decide H1 if
T = x[n]/N > 
 The detection performance increases with d2 = NA2/ 2
 Other approach for detection
 Neyman-Pearson approach
 When H0 and H1 are thought of random variables
 Maximum likelihood (ML) detector
 Maximum a posterior (MAP) detector
 Composite hypothesis testing

Institute of Communications Engineering, ECE, NCTU 13

 Detection Theory

Institute of Communications Engineering, ECE, NCTU 14

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Neyman-Pearson (NP) approach

 We have observations x[n], n=0,…, N-1, x{x[0],…,x[N-1]}
 Given P(x ; Hi ), we decide H1 if P(x ; H1) / P(x ; H0) > 
 Goal: maximize PD = P(H1 ; H1)
subject to PFA = P(H1 ; H0)  
  is called the significance level
 For example, H0 : x[n] = w[n]
H1 : x[n] = A + w[n], n=0,…, N-1,

Institute of Communications Engineering, ECE, NCTU 15

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

Institute of Communications Engineering, ECE, NCTU 16

Introduction to Detection and Estimation Theory Sau-Hsuan Wu


Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Bayesian approaches
 In some detection problems, it makes sense to assign
probabilities to the various hypotheses Hi
 Define a detector (x) = P(decide H1|x) [0, 1]
 The error probability of detection becomes
Pe = P(H1 | H0) P(H0) + P(H0 | H1) P(H1)
= P(H0) (x) P(x |H0)dx + P(H1)[1- (x)] P(x |H1)dx
= P(H1) +  (x)[P(x |H0) P(H0)-P(x |H1) P(H1)] dx
 We want to design a (x) [0, 1] to minimize Pe
 (x) = 0 if P(x |H0) P(H0)-P(x |H1) P(H1) > 0
 (x) = 1 if P(x |H0) P(H0)-P(x |H1) P(H1)  0
 Or, (x) = 1 if P(x |H1) / P(x |H0)  (x)  P(H0) / P(H1) = 
= 0 if (x) < 

Institute of Communications Engineering, ECE, NCTU 18

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 The function (x)=P(x |H1) / P(x |H0) is the likelihood ratio

 And the detection process is called the likelihood ratio test
 Maximum a posteriori (MAP) detector
 We may rewrite
 (x) = 0 if P(x |H0) P(H0)-P(x |H1) P(H1) > 0
 (x) = 1 if P(x |H0) P(H0)-P(x |H1) P(H1)  0
 Into
 (x) = arg max {0, 1} P(x |H) P(H )
= arg max {0, 1} P(H|x) P(x) = arg max {0, 1} P(H|x)
 Maximum likelihood (ML) detector
 If P(H0) = P(H1)
 (x) = arg max {0, 1} P(x |H)

Institute of Communications Engineering, ECE, NCTU 19

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 For example, H0 : x[n] = w[n]

H1 : x[n] = A + w[n], n = 0,…, N-1
 P(H0) = P(H1) = ½ and w[n] ~ N(0, 2)
 As a result,  = P(H0)/P(H1) = 1, and
 (x) = P(x |H1)/P(x |H0) >  = 1 yields

 Or equivalently

Institute of Communications Engineering, ECE, NCTU 20

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Thus, we decide H1 if
 Given that

 As a result

 Since Q(-x) = 1-Q(x), we have

Institute of Communications Engineering, ECE, NCTU 21

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 For MAP detector: arg max {0, 1} P(H|x)

 Thus, we decide H1 if P(H1|x) > P(H0|x)
 Suppose we have only one observation x[0]
 The decision region varies with the prior probability

(a) P(H0) = P(H1) = ½ (b) P(H0) = ¼, P(H1) = ¾

Institute of Communications Engineering, ECE, NCTU 22

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 In more general cases, we want to design a detector (x) to

minimize the average cost of decision errors
 Consider a cost function Cab of deciding Ha when Hb is true
 Given the general set of costs {C00, C01, C10, C11}
 The average cost, or also called Bayesian Risk, is given by
R = C00 P(H0 | H0) P(H0) + C01 P(H0 | H1) P(H1)
+ C10 P(H1 | H0) P(H0) + C11 P(H1 | H1) P(H1)
= C00 P(H0)[1- (x)]P(x |H0)dx + C10P(H0) (x)P(x |H0)dx
+ C01 P(H1)[1- (x)]P(x |H1)dx + C11P(H1) (x)P(x |H1)dx
 R = C00 P(H0) + C01 P(H1) +
(x)[(C10 – C00)P(H0)P(x |H0) + (C11 – C01)P(H1)P(x |H1)]dx
 (x) = 0 if >0
(x) = 1 if 0

Institute of Communications Engineering, ECE, NCTU 23

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 An alternative form
 [(C10 – C00)P(H0) P(x |H0) +(C11 – C01)P(H1) P(x |H1)]
= [C10 P(H0|x) + C11P(H1|x) - C00P(H0|x) - C01P(H1|x)]P(x)
 [C1(x) - C0(x)] P(x)
 where Ci(x)=Ci0P(H0|x)+Ci1P(H1|x) is the cost of deciding Hi
 Therefore,
(x) = 0 if C1(x) - C0(x) > 0 or C1(x) > C0(x)
(x) = 1 if C1(x) - C0(x)  0
 Or, we can define two decision regions
R1  {x|C1(x)  C0(x)}
R0  {x|C1(x) > C0(x)}
 (x) = 1 if x  R1
(x) = 0 if x  R0

Institute of Communications Engineering, ECE, NCTU 24

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Multiple hypothesis testing

 Suppose we must decide among M hypotheses for which
H0 : x ~ P(x |H0), H1 : x ~ P(x |H1), …, HM-1 : x ~ P(x |HM-1)
 The average cost is given by
R = i j Cij P(Hi | Hj) P(Hj)
 Redefine the decision rule
i(x)=P(decide Hi | x)[0, 1], and i i(x) =1
 R = i jCij  i(x)P(Hj |x) P(x)dx
 Redefine the cost of Ci(x) = j CijP(Hj|x)
 And define the decision region for Hi as
Ri  {x|Ci(x)  Cj(x),  j  i }
 The decision rule i(x) that minimizes R is
 i(x) = 1 if x Ri , and i(x) = 0 otherwise
Institute of Communications Engineering, ECE, NCTU 25
 Estimation Theory

Institute of Communications Engineering, ECE, NCTU 26

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Let us recall the problems we talked about

 x[n] = A1 + w[n], n=0,…,N-1
 x[n] = A2 + B2 n + w[n], n=0,…,N-1
 We define a parameter  to represent the quantities to be
estimated, e.g.  = A1 and  = [A2, B2] in the above cases
 We model the data by its probability density function
(PDF), assuming that the data are inherently random
 As an example:

 We have a class of PDFs where each one is different due to a

different value of , i.e. the PDFs are parameterized by  .
 The parameter  is assumed deterministic but unknown

Institute of Communications Engineering, EE, NCTU 27

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 We conclude for the time being that we are hoping to

have an estimator that gives
 An unbiased mean of the estimate:

 A minimum variance for the estimate:

 Is an unbiased estimator always the optimal estimator?
 Consider a widely used optimality criterion:
the minimum mean squared error (MSE) criterion

Institute of Communications Engineering, EE, NCTU 28

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 As an example, consider the modified estimator

 We attempt to find the ‘a’ which yields the minimum MSE

 Since E(Ă)=aA and var(Ă) = a2 2 / N, we have
mse (Ă) = a2 2 / N + (a-1)2A2
 Taking the derivative w.r.t. to a and setting it to zero leads to
aopt = A2 / (A2 + 2/ N )
 The optimal value of a depends upon the unknown parameter
A  The estimator is not realizable
 How do we resolve this dilemma?
 Since, mse (Ă) = var(Ă) + b2, as an alternative,
we set b=0 and search for the estimator that minimizes var(Ă)
 minimum variance unbiased (MVU) estimator

Institute of Communications Engineering, EE, NCTU 29

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 However, does a MVU estimator always exist for all ?

 Example:
 x[0] ~ N(,1)
 x[1] ~ N(,1) if   0
~ N(,2) if   0
 Both of the two estimators are unbiased

 Therefore

 None of the estimator has a variance uniformly less than or

equal to 18/36

Institute of Communications Engineering, EE, NCTU 30

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Is there a systematic way to find the MVU if it exists?

 We start by defining the set of data that is sufficient for
estimation?
 What do we mean by sufficiency in estimation?
 We want to have a set of data T(x) such that given T(x), any
individual data x(n) is statistically independent of A

 Suppose Â1= x[n]/N, then are the followings sufficient?

 S1 = {x[0], x[1],…, x[N-1]}
 S2 = x[n]
 GivenT0 = x[n] , do we still need the individual data?

Institute of Communications Engineering, EE, NCTU 31

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 We say the conditional PDF : p( x | x[n] = T0 ; A)

should not depend on A if T0 is sufficient
 E.g.

 For (a), a value of A near A0 is more likely even given T0

 For (b), however, p( x | x[n] = T0 ; A) is a constant

Institute of Communications Engineering, EE, NCTU 32

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Now, we need to determine p( x | x[n] = T0 ; A) to

show x[n] = T0 is sufficient
 By Baye’s rule

 Since T(x) is a direct function of x,

 Clearly, we have

Institute of Communications Engineering, EE, NCTU 33

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Thus, we have

Institute of Communications Engineering, EE, NCTU 34

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Since T(x) = x[n] ~ N(NA, N2)

 Thus

which does not depend on A

Institute of Communications Engineering, EE, NCTU 35

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 In general, to identify potential sufficient statistics is

difficult
 An efficient procedure for finding the sufficient statistics
is to employ the Neyman-Fisher factorization theorem
 Observe that

 If we can factor p(x;) into p(x;)=g(T(x),) h(x)

where
 g is a function depends on x only through T(x)
 h is a function depends only on x
  T(x) is a sufficient statistics for 
 The converse is also true
If T(x) is a sufficient statistics  p(x;)=g(T(x),) h(x)

Institute of Communications Engineering, EE, NCTU 36

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Recall p(x;A)

 On the other hand, we want to estimate 2 of y[n]=A+x[n]

 Suppose A is given, then define x[n] = y[n]-A

 Clearly, T(x) = x2[n] is a sufficient statistics for 2

Institute of Communications Engineering, EE, NCTU 37

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Ex. we want to estimate the phase of a sinusoid

x[n]=A cos(2 f0 n + ) + w[n], n=0,1,…,N-1
 Suppose A and f0 are given

 Expand the exponent

Institute of Communications Engineering, EE, NCTU 38

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 In this case, no single sufficient statistics exists, however

g(T1(x),T2 (x),  )

h(x)

Institute of Communications Engineering, EE, NCTU 39

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 The r statistics T1(x), T2(x),…, Tr(x) are jointly sufficient

if p(x | T1(x), T2(x),…, Tr(x) ; ) does not depend on 
 If p(x ; ) = g(T1(x), T2(x),…, Tr(x) , ) h(x)
 {T1(x), T2(x),…, Tr(x)} are sufficient statistics for 
 Now, we know how to obtain the sufficient statistics
 How do we apply them to help obtain the MVU estimator?
 The Rao-Blackwell-Lehmann-Scheffe Theorem
 If is an unbiased estimator of  and T(x) is a sufficient
statistic for , then is unbiased and
 A valid estimator for  (not dependent on )
 Of lesser or equal variance than that of for all 
 If T(x) is complete, then is the MVU estimator

Institute of Communications Engineering, EE, NCTU 40

Introduction to Detection and Estimation Theory Sau-Hsuan Wu

 Finally, a statistic is complete if there is only one function,

say g, of the statistic that is unbiased
 is solely a function of T(x)
 If T(x) is complete  is unique and unbiased
 Besides, for any unbiased estimator
 Then, must be the MVU
 In summary, the MVU can be found by
 Taking any unbiased and carrying out
 Alternatively, since there is only one function of T(x) that
leads to an unbiased estimator
 find the unique g(T(x)) that makes unbiased

Institute of Communications Engineering, EE, NCTU 41