The normal-normal Bayes estimation problem

Let X⃗ = (X1 , ..., Xn ) be a random sample from a N (µ, σ 2 ) population. The variance σ 2
is assumed to be known. Suppose that prior information about µ can be represented by
a µ ∼ N (m, v 2 ) prior distribution, where the prior parameters (m, v 2 ) are known. The
functional form of the posterior distribution for µ

f (µ|⃗x) ∝ f X x|µ) · f (µ)

⃗ (⃗

is obtained as follows. Ignoring the inessential integrating constant, the likelihood and the
prior density have the form
1 1
x|µ) ∝ exp { − − µ)2 }, f (µ) ∝ exp { − − m)2 }.
P
fX
⃗ (⃗ 2σ 2 i (xi 2v 2
(µ

The product yields

n o
1 1
f (µ|⃗x) ∝ exp − x2i − 2µ 2
i xi + nµ ) − (µ2 − 2mµ + m2 ) .
P P
2σ 2
( i 2v 2

Ignoring the inessential constant factors exp{− 2σ1 2 x2i } and exp{− 2v12 m2 }, we get
P
i

2 µ2
n o
f (µ|⃗x) ∝ exp − 12 ( nµ − 2 σµ2 xi − 2 vm2 µ)
P
σ2
+ v2 i

or
nµ2 µ2
n  o
f (µ|⃗x) ∝ exp − 1
− 2 σµ2 m
i xi − 2 v 2 µ
P
2 σ2
+ v2
n h i hP x i o
1 nv 2 +σ 2 i i m
∝ exp − 2 σ2 v2
µ2 − 2 σ2
+ v2
µ .
| {z } | {z }
2 2
1/σ∗ µ∗ /σ∗

Note that the r.h.s. has the form of a density kernel of a normal distribution for µ. To see
this, consider the density for an X ∼ N (µx , σx2 ) given by
µ2x
f (x) = √ 1 exp{− 12 ( σ12 x2 − 2 µσx2 x + σx2
)}.
2πσx2 x x

Hence, the posterior f (µ|⃗x) is a N (µ∗ , σ∗2 ) density with

σ2v2 nx̄v 2 + mσ 2
σ∗2 = 1 P
, µ∗ = , where x̄ = n i xi ,
nv 2 + σ 2 nv 2 + σ 2
and the posterior mean is
(σ 2 /v 2 )
h i h i
n
(µ|⃗x) = µ∗ = n+(σ 2 /v 2 )
x̄ + n+(σ 2 /v 2 )
m.

Recall that the classical ML estimate for µ is x̄. Thus, when using the posterior mean as the
Bayesian estimate, we obtain a weighted average of the classical estimate and an estimate
based on prior information, namely the prior mean m.
Furthermore note that the Bayesian estimate
(σ 2 /v 2 )
h i h i
n
(µ|⃗x) = µ∗ = n+(σ 2 /v 2 )
x̄ + n+(σ 2 /v 2 )
m

1
depends on the prior variance v 2 . Smaller values of v 2 correspond to greater confidence in
prior information, and this would make the Bayesian estimate closer to m. In contrast, as v 2
becomes larger, the Bayesian estimate approaches the classical estimate x̄.
For the limit of v 2 → ∞ the prior density becomes a so-called diffuse or improper prior
density. In this case the prior information is so poor that it is completely ignored in forming
the estimate.
Recall that the ML estimator X̄ is an unbiased estimator of µ. This implies that the
posterior mean µ∗ is biased for a finite v 2 . The bias comes from the prior in this case.