Statistics 512 Notes 26: Decision Theory

Continued
Posterior Analysis
We now develop a method for finding the Bayes rule. The
Bayes risk for a prior distribution
( )
is the expected loss
of a decision rule
( ) d X
when X is generated from the
following probability model:
First, the state of nature is generated according to the prior
distribution
( )
Then, the data X is generated according to the distribution
( ; ) f X
, which we will denote by
( | ) f X
Under this probability model (call it the Bayes model), the
marginal distribution of X is (for the continuous case)
( ) ( | ) ( )
X
f x f x d

Applying Bayes rule, the conditional distribution of

given X is
,
( , )
( | ) ( )
( | )
( )
( | ) ( )
X
X
f x
f x
h x
f x
f x d

The conditional distribution

( | ) h x
is called the X x of
. The words prior and posterior derive from the facts that
( )
is specified before (prior to) observing X and
( | ) h x
is calculated after (posterior to) observing X x . We will
discuss later more about the interpretation of prior and
posterior distributions.
Suppose that we have observed X x . We define the
posterior risk of an action
( ) a d x
as the expected loss,
where the expectation is taken with respect to the posterior
distribution of . For continuous random variables, we
have
( | )
[ ( ), ( ))] ( , ( )) ( | )
h X x
E l d x l d x h X x d

Theorem: Suppose there is a function

0
( ) d x
that minimizes
the posterior risk. Then
0
( ) d x
is a Bayes rule.
Proof: We will this for the continuous case. The discrete
case is proved analogously. The Bayes risk of a decision
function d is
( )
( )
,
( ) [ ( ), ]
[ ( ( , ( )) | ]
( , ( )) ( | ) ( )
( , ( )) ( , )
( , ( )) ( | ) ( )
X
x
X
B d E R d
E E l d X
l d x f x dx d
l d x f x dxd
l d x h x d f x dx

(We have used the relations
,
( | ) ( ) ( , ) ( ) ( | )
X X
f x f x f x h x

)
Now the inner integral is the posterior risk and since
( )
X
f x
is nonnegative,
( ) B d
can be minimized by choosing
0
( ) ( ) d x d x
.

The practical importance of this theorem is that it allows us

to use just the observed data,
x
, rather than considering all
possible values of X to find the action for the Bayes rule
**
d
given the data X x ,
**
( ) d x . In summary the
algorithm for finding
**
( ) d x is as follows:
Step 1: Calculate the posterior distribution
( | ) h X x
.
Step 2: For each action
a
, calculate the posterior risk,
which is
[ ( , ) | ] ( , ) ( | ) E l a X x l a h X d

The action
*
a that minimizes the posterior risk is the Bayes
rule action
** *
( ) d x a
Example: Consider again the engineering example.
Suppose that we observe
2
45 X x
. In the notation of
that example, the prior distribution is
1
( ) .8
,
2
( ) .2
.
We first calculate the posterior distribution:
2 1 1
1 2
2
2
1
( | ) ( )
( | )
( | ) ( )
.3*.8

.3*.8 .2*.2
.86
i i
i
f x
h x
f x

Hence,
2 2
( | ) .14 h x
We next calculate the posterior risk (PR) for
1
a
and
2
a
:
1 1 1 1 2 2 1 2 2
( ) ( , ) ( | ) ( , ) ( | )
0 400*.14
56
PR a l a h x l a h x +
+

and
2 1 2 1 2 2 2 2 2
( ) ( , ) ( | ) ( , ) ( | )
100*.86 0
86
PR a l a h x l a h x +
+

Comparing the two, we see that

1
a
has the smaller posterior
risk and is thus the Bayes rule.
Decision Theory for Point Estimation
Estimation theory can be cast in a decision theoretic
framework. The action space is the same as the parameter
space and the decision functions are point estimates

( ) d X . In this framework, a loss function may be

specified, and one of the strengths of the theory is that
general loss functions are allowed and thus the purpose of
estimation is made explicit. But the case of squared error
loss is especially tractable. For squared error loss, the risk
is
[ ]
{ }
( )
2
1
2
( , ) ( , , )

n
R d E d X X
E

1

1
]
K
Suppose that a Bayesian approach is taken and the prior
distribution on is
( )
. Then from the above theorem,
the Bayes rule for squared error loss can be found by
minimizing the posterior risk, which is
2
( )

[( ) | ] E X x


. We have
[ ] ( )
2
2
( ) ( ) ( )
2
( ) ( )

[( ) | ] ( ) | ( ) |

| |
E X x Var X x E X x
Var X x E X x




1 1
+
] ]
1
+
]
The first term of this last expression does not depend on

and the second term is minimized by
( )

| E X x


1

]
.
Thus, the Bayes rule for squared error loss is the mean of
the posterior distribution of ,

( | ) h X d

.
Example: A (possibly) biased coin is thrown once, and we
want to estimate the probability of the coin landing heads
on future tosses based on this single toss. Suppose that we
have no idea how biased the coin is; to reflect this state of
knowledge, we use a uniform prior distribution on :
( ) 1, 0 1 g
Let 1 X if a head appears, and let 0 X if a tail appears.
The distribution of X given is
, 1
( | )
1 , 0
x
f x
x

'

The posterior distribution is

( | ) 1
( | )
( | )
f x
h x
f x d

In particular,
1
0
1
0
( | 1) 2
(1 ) 1
( | 0) 2(1 )
(1 )
h X
d
h X
d

Suppose that 1 X . The Bayes estimate of is the mean

of the posterior distribution
( | 1) h X
, which is
1
0
2
(2 )
3
d

The Bayes estimate in the case where 0 X is

1

3

.
Note that these estimates differ from the classical
maximum likelihood estimates, which are 0 and 1.
Comparison of the risk functions of the Bayes estimate and
the MLE:
The risk function of the above Bayes estimate is
( )
2 2
2
2 2
2
1 2

( 0) ( 1)
3 3
1 2
(1 )
3 3
1 1 1

9 3 3
Bayes
E P X P X




_ _
1
+

1
]
, ,
_ _
+

, ,
+
The risk function of the MLE

MLE
X is
( )
( ) ( )
( ) ( )
2
2 2
2 2
2

0 ( 0) 1 ( 1)
0 (1 ) 1
(1 )
MLE
E P X P X




1
+
1
]
+
+
The following graph shows the risk function of the Bayes
estimate and the MLE the Bayes estimate is the solid line
and the MLE is the dashed line. The Bayes estimate has
smaller risk than that of the MLE over most of the range of
[0,1] but neither estimator dominates the other.
Admissibility
Minimax estimators and Bayes estimators are good
estimators in the sense that their risk functions have
certain good properties; minimax estimators minimize the
worst case risk and Bayes estimators minimize a weighted
average of the risk. It is also useful to characterize bad
estimators.
Definition: An estimator

is inadmissible if there exists

another estimator

'
that dominates

, meaning that

( , ') ( , ) for all and

( , ') ( , ) for at least one

If there is no estimator ' that dominates , then is
R R
R R
admissible

<
Example: Let X be a sample of size one from a
( ,1) N
distribution and consider estimating with squared error
loss.
Let

( ) 2 X X . Then

( )
2
2
2

( , ) [(2 ) ]
(2 ) [2 ]
4
R E X
Var X E X

+
+
The risk function of the MLE

( )
MLE
X X is
( )
2
2

( , ) [( ) ]
( ) [ ]
1
MLE
R E X
Var X E X

Thus, the MLE dominates

( ) 2 X X and

( ) 2 X X is
inadmissible.
Consider another estimator

( ) 3 X . We will show that

is admissible. Suppose not. Then there exists a different

estimator

'
with smaller risk. In particular,

(3, ') (3, ) 0 R R . Hence,
( )
2 2
1

0 (3, ') ( ' 3) exp ( 3) / 2
2
R x dx

.
Thus,

'( ) 3 X and there is no estimator that dominates

. Even though

is admissible, it is clearly a bad estimator.

In general, it is very hard to know whether a particular
estimate is admissible, since one would have to check that
it was not strictly dominated by any other estimate in order
to show that it was admissible. The following theorem
states that any Bayes estimate is admissible.
Theorem (Complete Class Theorem): Suppose that one of
the following two assumptions holds:
(1) is discrete and * d is a Bayes rule with respect to a
prior probability mass function

such that
( ) 0 >
for all
.
(2) is an interval and * d is a Bayes rule with respect to a
prior density function
( ) g
such that
( ) 0 >
for all
and
( , ) R d
is a continuous function of for all d .
Then * d is admissible.
Proof: We will prove the theorem for assumption (2). The
proof is by contradiction. Suppose that * d is inadmissible.
There is then another estimate, d , such that
( , *) ( , ) R d R d
for all and with strict inequality for
some , say
0

. Since
( , *) ( , ) R d R d
is a continuous
function of , there is an 0 > and an interval h t such
that
0 0
( , *) ( , ) for R d R d h h > +
Then,
[ ] [ ]
0
0
0
0
( , *) ( , ) ( ) ( , *) ( , ) ( )
( ) 0
h
h
h
h
R d R d d R d R d d
d

> >

But this contradicts the fact that * d is a Bayes rule because

a Bayes rule has the property that
[ ]
( *) ( ) ( , *) ( , ) ( ) 0 B d B d R d R d d

.
The proof is complete.

The theorem can be regarded as both a positive and

negative result. It is positive in that it identifies a certain
class of estimates as being admissible, in particular, any
Bayes estimate. It is negative in that there are apparently
so many admissible estimates one for every prior
distribution that satisfies the hypotheses of the theorem
and some of these might make little sense (like

( ) 3 X
for the normal distribution above).