Computational Bayesian Statistics: Credible Intervals

A credible interval is a Bayesian version of a frequentist’s confidence interval. We begin

by reviewing frequentist confidence intervals.

1 Frequentist Confidence Intervals

Suppose that X1 , X2 , . . . , Xn is a random sample from a distribution that depends on a
one-dimensional parameter θ. To construct a confidence interval for θ we must

1. Come up with an estimator θb for θ.

2. Come up with some function of both θb and θ whose distribution is known and, in
particular, not dependent on the unknown θ. This function is known as a pivotal
quantity.
3. Use the distribution from Step 2 to come up with “critical values” that the pivotal
quantity is between with a prespecified probability.
4. Using the inequalities from Step 3, solve for bounds for the unknown θ.

Example 1: Suppose that X1 , X2 , . . . , Xn is a random sample from the N (µ, 1) distribution.

Derive a 90% confidence interval for µ

Step One: An estimator of µ is the sample mean µb = X.

Step Two: The distribution of X is N (µ, 1/n). Thus, we know that

X −µ X −µ
q = √ ∼ N (0, 1).
1/n 1/ n

Since this distribution is completely known (not only is it known to be

X−µ
normal but it does not depend on any unknown parameters), 1/ √ is a
n
pivotal quantity.
Step Three: Suppose that Z ∼ N (0, 1), we can find (using a z−table
or computer that

P (−1.645 < Z < 1.645) = 0.90.

X−µ
Since √
1/ n
has the same distribution, we also have that
!
X −µ
P −1.645 < √ < 1.645 = 0.90.
1/ n
 Step Four: Solving for µ “in the middle” gives
!
1 1
P X − 1.645 √ < µ < X + 1.645 √ = 0.90.
n n

The 90% confidence interval for µ is given by

!
1 1
X − 1.645 √ , X − 1.645 √ .
n n

Interpretation: Note that µ is not a random variable in the frequen-

tist paradigm. Here, it is the endpoints of the interval that are random.
Once data is collected for some fixed sample size n and the sample mean
is computed. The confidence interval becomes numerical. Suppose the
interval is then, for example, (−3.421, 4.718). The true mean µ is fixed
and is either in this interval or not in this interval. It is not in there
with probability 0.90. The randomness came from the random sampling
from the normal distribution. Different samples will result in different
sample means which will result in different confidence intervals. In the
long run, with repreated sampling, the true mean µ will be correctly
captured by the interval 90% of the time. t
u

Example 2: Suppose that X1 , X2 , . . . , Xn is a random sample from the exponential

distribution with rate λ. Derive a 90% confidence interval for λ

Step One: Since the mean of this distribution is 1/λ, one decent idea
for an estimator of λ is λ
b = 1/X.

Step Two: The distribution of X can be found using moment generating

functions. It is X ∼ Γ(n, nλ).
1/X has an inverse gamma distribution but we will see that is is suffi-
cient to work with the distribution of X.
We need to find a function of 1/X and λ whose distribution is known
and free of the unknown λ. Any function of X and λ can be rewritten
as a function of 1/X and λ. This is why we can work with X instead
of 1/X.
If Y ∼ Γ(α, β) and c > 0 is a constant, we can show that cY ∼
Γ(α, β/c). Applying this to X ∼ Γ(n, nλ), we see that λX ∼ Γ(n, n).
Since this is a completely known distribution, λX is a pivotal quantity.
Step Three: Suppose that Y ∼ Γ(n, n). We can find, using a computer,
critical values a and b that solve
Z b
1 n n−1 −ny
0.90 = P (a < Y < b) = n y e dy.
a Γ(n)
There are many pairs of values for a and b that will solve this problem.
Ideally, one would choose a and b as a solution to this integral equa-
tion with the restriction that b − a is minimized so that the resulting
confidence interval will be shortest.
(In reality though, if you are up on your Mathematical Statistics, you
would transform the pivotal quantity λX into something having a χ2
distribution and use a χ2 -table. A χ2 distribution is a special case of
the gamma distribution that depends on only one parameter as opposed
to two paramters for the gamma distribution. There are many tables
floating around that give that give critical values for χ2 distributions
that have been obtained through numerical integration of the χ2 pdf!)
Step Four: Assuming that you have solved for a and b (they will depend
on n and on the 0.90), you would write

a<Y <b

⇓
a < λX < b
and solve for λ “in the middle” to get
a b
<λ< .
X X
The 90% confidence interval for λ is then given by

a a
 
,
X X

where, again, a and b will depend on n and 0.90. (And ideally would
be written as chi-squared critical values!)
Interpretation: Once again, we have an interval with random end-
points that will contain the true value of λ with probability 0.90. Once
we collect the sample and compute the numerical value of X and hence
the numerical confidence interval, we will have a fixed interval that ei-
ther contains λ or doesn’t contain λ. In the long run, with repreated
sampling, the true mean λ will be correctly captured by the interval
90% of the time. t
u
2 Bayesian Credible Intervals
As Bayesians, we are thinking of parameters, such as µ for the normal distribution, as random
variables. Thus, it now makes sense to write statements like

P (−3.421 < µ < 4.718) = 0.90.

This probability can be computed by integrating a prior pdf for µ but, if we want to let the
data speak, we’d better use the posterior pdf for µ given the data!
Like all Bayesian results, the credible interval will be affected by the choice of the prior
distribution for µ. In the following examples, we will compare results for different priors.

Example 1: Suppose that X1 , X2 , . . . , Xn is a random sample from the N (µ, 1) distribution.

Compute a 90% credible interval for µ under the assumptions

(a) µ has a flat prior

(b) µ has a N (µ0 , σ02 ) prior for known hyperparameters µ0 and σ02

Solution to (a):
The likelihood is Pn
1
(xi −µ)2
f (~x|µ) = (2π)−n/2 e− 2 i=1 .

The prior is
f (µ) ∝ 1, −∞ < µ < ∞.

The posterior is
f (µ|~x) ∝ f (~x|µ) · f (µ)
1
Pn
(xi −µ)2
∝ e− 2 i=1 ·1
1
xi − n
P P
x2i +µ µ2
= e− 2 2

1
xi − n
P P
x2i +µ µ2
∝ e− 2 2

.
= ..
n 2
∝ e− 2 (µ−x) .
~ = ~x is N (x, 1/n).
That is, the posterior distribution for µ given X
We wish to find critical values a and b such that
~ = ~x) = 90%
P (a < µ < b|X
 µ−x
In this conditional world, we have that √
1/ n
∼ N (0, 1).
We know that for a N (0, 1) random variable Z,
P (−1.645 < Z < 1.645) = 0.90.
(Just as in the frequentist case, there are other non-symmetric values as well!)
So,
0.90 = P (−1.645 < Z < 1.645)
 
= P −1.645 < µ−x
√ ~ = ~x
< 1.645 X
1/ n

.
= ..
 
= P x − 1.645 √1n < µ < x + 1.645 √1n

The 90% credible interval for µ, using a flat prior, is

!
1 1
x − 1.645 √ , x + 1.645 √ .
n n

Interpretation: This looks very similar to what we had in the frequentist case but the
interpretation is very different. The lowercase notation for x indicates that the sample mean,
and hence the endpoints of the interval, have been fixed and computed. As a frequentist,
the parameter µ would be fixed and would be either in the interval or not. It would not be
in there “with some probability”.
In the Bayesian case though, we are thinking of µ as random, and we can say, using a flat
prior and observing the data as x1 , x2 , . . . , xn , that µ is in this interval with probability 0.90!

Solution to (b):
The likelihood is still Pn
1
(xi −µ)2
f (~x|µ) = (2π)−n/2 e− 2 i=1 .
The prior is now
1
1 −
2σ 2
(µ−µ0 )2
f (µ) = q e 0 .
2πσ02
The posterior is then
f (µ|~x) ∝ f (~x|µ) · f (µ)
1
1
Pn − (µ−µ0 )2
(xi −µ)2 2σ 2
∝ e− 2 i=1 ·e 0

.
= ..
1
− (µ=µ∗ )2
∝ e 2(σ 2 )∗
where P
∗ µ0 +σ02 xi
µ = 1+nσ02

σ02
(σ 2 )∗ = 1+nσ02

Thus, we have that  

µ|~x ∼ N µ∗ , (σ 2 )∗ .

We wish to find critical values a and b such that

~ = ~x) = 90%
P (a < µ < b|X
∗
In this conditional world, we have that √µ−µ2 ∼ N (0, 1).
(σ )∗

We know that for a N (0, 1) random variable Z,

P (−1.645 < Z < 1.645) = 0.90.

So,
0.90 = P (−1.645 < Z < 1.645)
 
∗
= P −1.645 < √µ−µ2 ~ = ~x
< 1.645 X
(σ )∗

.
= ..
 
= P x − 1.645 √1n < µ < x + 1.645 √1n

The 90% credible interval for µ, using the conjugate N (µ0 , σ02 ), is
 q q 
µ∗ − 1.645 (σ 2 )∗ , µ∗ + 1.645 (σ 2 )∗ .

Note that, as σ02 → ∞, the N (µ0 , σ02 ) conjugate prior is an ever flattening bell curve that
is “squishing down” to a flat line. So, the flat uninformative prior can be thought of as a
limiting case of the conjugate prior. Indeed, in this case and for fixed n,
µ0 + σ02 xi
P P
∗ xi
µ = → =x
1 + nσ02 n
and
σ02 1
(σ 2 )∗ = 2
→
1 + nσ0 n
as σ02 → ∞. These limiting parameters match the parameters in the posterior distribution
for µ under the assumption of a flat prior.
Still, our resulting interval is kind of convoluted. In my personal opinion, too many people
fall back on conjugate priors for computational convenience. I would go with the flat prior
here unless I really knew something a priori about µ. I wouldn’t worry so much about the fact
that it is a normal distribution used for the prior as much as the parameters. Perhaps prior
experimentation/data suggests that the mean is around 3. I would then use a normal prior
with mean 3 and a variance that expresses how confident I am about that prior estimate.
(Small prior variance for high confidence and larger prior variance for less confidence.) tu

Example 2: Suppose that X1 , X2 , . . . , Xn is a random sample from the exponential distri-

bution with rate λ.
Compute an 85% credible interval for λ under the assumptions

(a) λ has a flat prior

(b) λ has a Γ(α, β) prior for known hyperparameters α and β

Solution to (a):
The likelihood is n
Pn
f (~x|λ) = λn e−λ xi
Y
i=1 I(0∞) (xi ).
i=1

The prior is
f (λ) ∝ 1, λ > 0.

The posterior is
f (λ|~x) ∝ f (~x|λ)
Pn
∝ λn e−λ i=1
xi
·1
Pn
= λn e−λ i=1
xi

As a function of λ. this looks like a gamma distribution. In particular,

n
X
λ|~x ∼ Γ(n + 1, xi ).
i=1

We wish to find critical values a and b such that

~ = ~x) = 0.85.
P (a < λ < b|X

If you’ve had MathStat, the best approach would be to multiply a < λ < b through by
appropriate constants in order to move from working with a gamma distribution to a chi-
squared distribution. Then, give your cutoffs in terms of symbolic chi-squared critical values
of numerical ones from a χ2 -table.
Otherwise, you need to numerically solve
Z b n
!n+1
1 P
λn e−λ xi
X
xi dλ.
a Γ(n + 1) i=1
There are many sets of values for a and b that will solve this. For simplicity, you could take
a = 0 and just solve for b. Alternatively, you could take b = ∞ and try to solve for a.
In frequentist statistics, taking a = 0 is often done for simplicity but sometimes people will
use Calculus (or numerics) to find the shortest possible confidence interval. After all, we are
giving an interval estimate for λ and we would like to be as precise as possible.

In Bayesian statistics, if you wanted to optimize your credible interval, the goal would be to
find an interval of values for λ that have the highest posterior density.
Definition:
In Bayesian statistics, a 100(1 − α)% highest posterior density region for a parameter θ
is a subset C of the parameter space that is defined by

C = {θ : f (θ|~x) ≥ k}

where k is the largest value such that

Z
f (θ|~x) dθ = 1 − α.
C

Basically this means that you want to find the highest horizontal line that, when intersected
with the posterior pdf, defines θ values that, when integrating between, will give you 1 − α.
For the exponential example, this will give two values as depicted below.

These values are the endpoints of the highest posterior density region for λ. Note that, in
the case of multimodal posterior densities, the highest posterior density region may consist
of a collection of disjoint intervals. (This is why it’s called a “region” and not a highest
posterior density “interval”.)

Solution to (b): Similar because the prior is a conjugate prior– we will just get a different
gamma posterior.