STA732

Statistical Inference
Lecture 09: Bayesian estimation

Yuansi Chen
Spring 2023
Duke University

https://www2.stat.duke.edu/courses/Spring23/sta732.01/

1
Recap from Lecture 08

1. Construct minimum risk equivariant (MRE) estimator via

conditioning on maximal invariant statistics
2. Pitman estimator of location
3. MRE for location is unbiased under squared error loss
4. MRE usually admissible

2
Where we are

• We have finished the first approach of arguing for “the best”

estimator in point estimation: by restricting to a small set of
estimatiors
• Unbiased estimators
• Equivariant estimators
• We begin the second approach: global measure of optimality
• average risk
• minimax risk

3
Goal of Lecture 09

1. Bayes risk, Bayes estimator

2. Examples
3. Bayes estimators are usually biased
4. Bayes estimators are usually admissible

Chap. 7 in Keener or Chap. 4 in Lehmann and Casella

4
Bayes risk, Bayes estimator
Recall the components of a decision problem

• Data 𝑋
• Model family P = {𝑃𝜃 ∶ 𝜃 ∈ Ω}, a collection of probability
distributions on the sample space
• Loss function 𝐿, 𝐿(𝜃, 𝑑) measures the loss incurred by the
decision 𝑑 when compared with the parameter obtained from 𝜃
• Risk function 𝑅, 𝑅(𝜃, 𝛿) = 𝔼𝜃 [𝐿(𝜃, 𝛿)]

5
The frequentist motivation of the Bayesian setup

Motivation
It is in general hard to find uniformly minimum risk estimator.
Oftentimes, we have risks that cross. This difficulty will not arise if
the performance is measured via a single number.

Def. Bayes risk

The Bayes risk is the average-case risk, integrated w.r.t. some
measure Λ, called prior.

6
The frequentist motivation of the Bayesian setup

Motivation
It is in general hard to find uniformly minimum risk estimator.
Oftentimes, we have risks that cross. This difficulty will not arise if
the performance is measured via a single number.

Def. Bayes risk

The Bayes risk is the average-case risk, integrated w.r.t. some
measure Λ, called prior.

Remark
For now, assume Λ(Ω) = 1 (Λ is a prob measure). Later we might
deal with improper prior.

6
Bayes risk

𝑅Bayes (Λ, 𝛿) = ∫ 𝑅(𝜃, 𝛿)𝑑Λ(𝜃)

Ω
= 𝔼𝑅(Θ, 𝛿)

where Θ is the randoma variable with distribution Λ.

𝔼𝑅(Θ, 𝛿) = 𝔼[𝔼[𝐿(Θ, 𝛿(𝑋)) ∣ 𝑋]]

Both 𝑋 and Θ are considered random.

The frequentist understanding: average risk makes sense without believing the
parameter is random

7
Bayes estimator

An estimator 𝛿 which minimizes the average risk 𝑅Bayes (Λ, ⋅) is a

Bayes estimator.

8
Construct Bayes estimator

Thm 7.1 in Keener

Suppose Θ ∼ Λ, 𝑋 ∣ Θ = 𝜃 ∼ 𝑃𝜃 , and 𝐿(𝜃, 𝑑) ≥ 0 for all 𝜃 ∈ Ω and
all 𝑑. If

• 𝔼[𝐿(Θ, 𝛿0 )] < ∞ for some 𝛿0

• for a.e. 𝑥, there exists a 𝛿Λ (𝑥) minimizing

𝔼[𝐿(Θ, 𝑑) ∣ 𝑋 = 𝑥]

with respect to 𝑑

Then 𝛿Λ is a Bayes estimator.

In words: the Bayes estimator can be found by minimizing the conditional

distribution 𝔼[𝐿(𝜃, 𝑑) ∣ 𝑋 = 𝑥], one 𝑥 at a time

9
proof of Thm 7.1

10
Posterior

Def. Posterior
The conditional distribution of Θ given 𝑋, written as ℒ(Θ ∣ 𝑋) is
called the posterior distribution

Remark
• Λ is usually interpreted as prior belief about Θ before seeing
the data
• ℒ(Θ ∣ 𝑋) is the belief after seeing the data

11
Posterior calcultation with density

Suppose prior density 𝜆(𝜃), likelihood 𝑝𝜃 (𝑥), then the posterior

density is

𝜆(𝜃)𝑝𝜃 (𝑥)
𝜆(𝜃 ∣ 𝑥) =
𝑞(𝑥)

where 𝑞(𝑥) = ∫Ω 𝜆(𝜃)𝑝𝜃 (𝑥)𝑑𝜃 is the marginal density of 𝑋.

Then the Bayes estimator has the form

𝛿Λ (𝑥) = arg min ∫ 𝐿(𝜃, 𝑑)𝜆(𝜃 ∣ 𝑥)𝑑𝜃

𝑑 Ω

12
Posterior mean is Bayes estimator for squared error loss

2
Suppose 𝐿(𝜃, 𝑑) = (𝑔(𝜃) − 𝑑) then the Bayes estimator is the
posterior mean
proof:

13
Examples
Binomial model with Beta prior

Suppose 𝑋 ∣ Θ = 𝜃 ∼ Binomial(𝑛, 𝜃) with density 𝜃𝑥 (1 − 𝜃)𝑛−𝑥 (𝑛𝑥),

Θ ∼ Beta(𝛼, 𝛽) with density 𝜃𝛼−1 (1 − 𝜃)𝛽−1 Γ(𝛼)Γ(𝛽)
Γ(𝛼+𝛽) . Find the
Bayes estimator under squared error loss.

14
Weighted squared error loss

2
Suppose 𝐿(𝜃, 𝑑) = 𝑤(𝜃) (𝑔(𝜃) − 𝑑) . Find a Bayes estimator.

15
Normal mean estimation

𝑋 ∣ Θ = 𝜃 ∼ 𝒩(𝜃, 𝜎2 ),
Θ ∼ 𝒩(𝜇, 𝜏 2 ).
Find the Bayes estimator of mean under squared error loss
What if we have 𝑛 i.i.d. data points?

16
Binary classification

Suppose the parameter space Ω = {0, 1}.

ℙ(𝑋 = 𝑥 ∣ Θ = 0) = 𝑓0 (𝑥) and ℙ(𝑋 = 𝑥 ∣ Θ = 1) = 𝑓1 (𝑥). The
prior is 𝜋(1) = 𝑝, 𝜋(0) = 1 − 𝑝.
⎧
{0 𝑑 = 𝜃
Determine a Bayes estimator under 0-1 loss 𝐿(𝜃, 𝑑) =
⎨
{
⎩1 𝑑 ≠ 𝜃

17
Bayes estimators are usually biased
Unbiased estimator under squared error loss is not Bayes

Thm Lehmann Casella 4.2.3

If 𝛿 is unbiased for 𝑔(𝜃) with 𝑅Bayes (Λ, 𝛿) < ∞ then 𝛿 is not Bayes
under squared error loss unless its average risk is zero

𝔼 [(𝛿(𝑋) − 𝑔(Θ))2 ] = 0

18
proof:

19
Bayes estimators are usually
admissible
Uniqueness of Bayes estimator under strictly convex loss

Thm. Lehmann Casella 4.1.4

Let 𝑄 be the marginal distribution of 𝑋, i.e.,
𝑄(𝐸) = ∫ ℙ𝜃 (𝐸)𝑑Λ(𝜃). Suppose 𝐿 is strictly convex. If

1. 𝑅Bayes (Λ, 𝛿Λ ) < ∞,

2. 𝑄(𝐸) = 0 implies 𝑃𝜃 (𝐸) = 0, ∀𝜃,

then the Bayes estimator 𝛿Λ is unique (a.e. with respect to 𝑃𝜃 for all
𝜃).

20
proof: Use the following lemma
Lem. Lehmann Casella exercise 1.7.26
Let 𝜙 be a strictly convex function over an interval 𝐼. If there exists a
value 𝑎0 ∈ 𝐼 minimizing 𝜙, then 𝑎0 is unique.

21
A unique Bayes estimator is admissible

Thm. Lehmann Casella 5.2.4

A unique Bayes estimator (a.s. for all 𝑃𝜃 ) is admissible.

22
proof:

23
Summary

• Bayes estimator is defined as the minimizer of the average risk

over a prior on 𝜃.
• Bayes estimator can be constructed by conditioning the risk
on each 𝑥
• Bayes estimators are biased under squared error loss
• Bayes estimators are admissible under strictly convex loss

24
What is next?

• Where do priors come from?

• Pros and cons of Bayes

25
Thank you

26
27