Scribd
Upload
4 views

bayes

Uploaded by

Chinaski Bukowski
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
4 views

bayes

Uploaded by

Chinaski Bukowski
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Bayes Estimation

February 26-28, 2008

Bayesian estimation begins with an assumed probability distribution on the parameter space Θ. With
this approach θ itself is a random variable and the observations X1 , . . . , Xn are conditionally independent
given the value of θ. Consequently, in Bayesian statistics, (X, θ) is a random variable on the state space
S n × Θ.
The distribution of θ on Θ is called the prior distribution. We shall denote its density by π. Together,
the prior distribution and the parametric family {Pθ ; θ ∈ Θ} determine the joint distribution of (X, θ).

1 Bayes Formula
For events A and B, recall that the conditional probability is

P (A|B)P (B) = P (A ∩ B) = P (B|A)P (A),


or
P (B|A)P (A)
P (A|B) = .
P (B)
Now, if we set
A = {θ = θ0 } and B = {X = x},
then
P {X = x|θ = θ0 }P {θ = θ0 }
P {θ = θ0 |X = x} = .
P {X = x}
If the appropriate densities exist, then we can write Bayes formula as
!
fX|Θ (x|θ0 )
fΘ|X (θ0 |x) = R π(θ0 ),
fX|Θ (x|θ̃)π(θ̃) dθ̃

to compute the posterior density fΘ|X (θ0 |x) as the product of the Bayes factor and the prior density.
If T is a sufficient statistic and fX|Θ (x|θ̃) = h(x)g(θ, T (x)), then the Bayes factor

fX|Θ (x|θ0 ) h(x)g(θ, T (x)) g(θ, T (x))


R =R =R
fX|Θ (x|θ̃)fΘ (θ̃) dθ̃ h(x)g(θ̃, T (x)) dθ̃ g(θ̃, T (x)) dθ̃

is a function of T .
Example 1 (Normal observations and normal prior). Suppose that

1
• θ is N (θ0 , 1/λ), and
• that given θ, X consists of n conditionally independent N (θ, 1) random variables.
q
λ
Then the prior density is fΘ (θ) = 2π exp(− λ2 (θ − θ0 )2 ), and
n
1X
fX|Θ (x|θ) = (2π)−n/2 exp(− (xi − θ)2 )
2 i=1
n
n 1X
= (2π)−n/2 exp(− (θ − x̄)2 − (xi − x̄)2 ).
2 2 i=1

The posterior density is proportional to


1
k(x) exp(− (n(θ − x̄)2 + λ(θ − θ0 )2 ))
2
n+λ
= k̃(x) exp(− (θ − θ̃(x))2 ).
2
where
λθ0 + nx̄
θ̃(x) = .
λ+n
Thus, the posterior distribution is
N (θ1 (x), 1/(λ + n)).
Note that it is a function of the sufficient statistics T (x) = x1 + · · · + xn . If n is small, then θ1 (x) is
near θ0 . If n is large, θ1 (x) is near x̄.

2 Bayes Action
Recall that given a loss function L and an estimator d the risk function R : Θ × D → R is the expected loss
for that decision.
R(θ, d) = Eθ L(θ, d(X))
and the mean risk, or Bayes risk,
Z Z Z
r(π, d) = R(θ, d)π(θ) dθ = L(θ, d(x))fX (x|θ)π(θ) dxdθ.
Θ Θ Rn

The decision function that minimizes risk is called the Bayes action.
If the loss function is L1 (θ, a) = |θ − a|, then the posterior median minimizes risk and thus the Bayes
action θ̂1 (x) satisfies
Z θ̂1 (x)
1
= fΘ|X (θ|x) dθ.
2 −∞
If the loss function is L2 (θ, a) = (θ − a)2 , then the posterior mean minimizes risk and thus the Bayes
action Z
θ̂2 (x) = E[θ|X = x] = θfΘ|X (θ|x) dθ.

For the example of a normal prior and normal observations, θ̂1 (x) = θ̂2 (x) = θ̃(x).

2
10
8
dbeta(x, 1, 11)

6
4
2
0

0.0 0.2 0.4 0.6 0.8 1.0

x
Figure 1: Beta posterior distribution with t = 0, 1, · · · , 10 successes in 10 Bernoulli trials based on a uniform
prior

Example 2. Let the prior distribution π on θ be a beta distribution with parameters α and β and consider
Bernoulli observations X1 , . . . , Xn with parameter θ. T (X) = X1 + · · · + Xn is a sufficient statistic. The
posterior distribution
Γ(α + β) α−1
fΘ|X (θ|x) ∝ L(θ|x)π(θ) = θT (x) (1 − θ)n−T (x) θ (1 − θ)β−1 , 0 ≤ θ ≤ 1.
Γ(α)Γ(β)
Thus,
fΘ|X (θ|x) ∝ θT (x)+α−1 (1 − θ)n−T (x)+β−1 0 ≤ θ ≤ 1.
and the posterior distribution is Beta(T (x) + α, n − T (x) + β). If we want to estimate θ using a quadratic
risk function, then
T (x) + α
θ̂(x) = E[θ|X = x] = .
n+α+β
The uniform distribution on [0, 1] has a Beta(1, 1) distribution. In this case

T (x) + 1
θ̂(x) = .
n+2
The posterior densities are graph using R in Figure 1 using
> curve(dbeta(x,1,11),0,1)
> for (i in 2:11){curve(dbeta(x,i,12-i),0,1,add=TRUE)}

You might also like