The Laplace approximation applied to

Bayesian logistic regression

There are multiple ways that we could try to fit a distribution with a Gaussian form. For
example, we could try to match the mean and variance of the distribution. The Laplace
approximation is another possible way to approximate a distribution with a Gaussian. It can
be seen as an incremental improvement of the MAP approximation to Bayesian inference,
and only requires some additional derivative computations.
In Bayesian logistic regression, we can only evaluate the posterior distribution up to a
constant: we can evaluate the joint probability p(w, D), but not the normalizer P(D). We
match the shape of the posterior using p(w, D), and then the approximation can be used to
approximate P(D).
The Laplace approximation sets the mode of the Gaussian approximation to the mode of
the posterior distribution, and matches the curvature of the log probability density at that
location. We need to be able to evaluate first and second derivatives of log P(w, D).
The rest of the note just fills in the details. We’re not adding much to MacKay’s textbook
pp341–342, or Murphy’s book p255. Although we try to go slightly more slowly and show
some pictures of what can go wrong.

1 Matching the distributions

First of all we find the most probable setting of the parameters:
w∗ = arg max p(w | D) = arg max log p(w, D). (1)
w w

The conditional probability on the left is what we intuitively want to optimize. The maxi-
mization on the right gives the same answer, but contains the term we will actually compute.
Reminder: why do we take the log?1
[The website version of this note has a question here.]
We usually find the mode of the distribution by minimizing an ‘energy’, which is the negative
log-probability of the distribution up to a constant. For a posterior distribution, we can
define the energy as:
E(w) = − log p(w, D), w∗ = arg min E(w). (2)
w

We minimize it as usual, using a gradient-based numerical optimizer.

∂E
The minimum of the energy is a turning point. For a scalar variable w the first derivative ∂w
is zero and the second derivative gives the curvature of this turning point:
∂2 E ( w )
H= . (3)
∂w2 w=w∗

The notation means that we evaluate the second derivative at the optimum, w = w∗ . If H is
large, the slope (the first derivative) changes rapidly from a steep descent to a steep ascent.
We should approximate the distribution with a narrow Gaussian. Generalizing to multiple
variables w, we know ∇w E is zero at the optimum and we evaluate the Hessian, a matrix
with elements:
∂2 E ( w )
Hij = . (4)
∂wi ∂w j ∗ w=w

1. Because log is a monotonic transformation, maximizing the log of a function is equivalent to maximizing the
original function. Often the log of a distribution is more convenient to work with, less prone to numerical problems,
and closer to an ideal quadratic function that optimizers like.

MLPR:w10b Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 1

This matrix tells us how sharply the distribution is peaked in different directions.
For comparison, we can find the optimum and curvature that we would get if our distribution
were Gaussian. For a one-dimensional distribution, N (µ, σ2 ), the energy (the negative log-
probability up to a constant) is:

( w − µ )2
EN (w) = . (5)
2σ2
The minimum is w∗ = µ, and the second derivative H = 1/σ2 , implying the variance is
σ2 = 1/H. Generalizing to higher dimensions, for a Gaussian N (µ, Σ), the energy is:

1
EN (w) = ( w − µ ) > Σ −1 ( w − µ ), (6)
2
with w∗ = µ and H = Σ−1 , implying the covariance is Σ = H −1 .
Therefore matching the minimum and curvature of the ‘energy’ (negative log-probability) to
those of a Gaussian energy gives the Laplace approximation to the posterior distribution:

p(w | D) ≈ N (w; w∗ , H −1 ) (7)

[The website version of this note has a question here.]

2 Approximating the normalizer Z

Evaluating our approximation for a D-dimensional distribution gives:

p(w, D) | H |1/2
 
1
p(w | D) = ≈ N (w; w∗ , H −1 ) = exp − ( w − w ∗ >
) H ( w − w ∗
) . (8)
P(D) (2π ) D/2 2

At the mode w∗ = w, the exponential term disappears and we get:

p(w∗ , D) | H |1/2 p(w∗ , D)(2π ) D/2

≈ , P(D) ≈ . (9)
P(D) (2π ) D/2 | H |1/2

An equivalent expression is

P(D) ≈ p(w∗ , D) |2πH −1 |1/2 , (10)

where | · | means take the determinant of the matrix.

When some people say “the Laplace approximation”, they are referring to this approximation
of the normalization P(D), rather than the intermediate Gaussian approximation to the
distribution.

3 Computing logistic regression predictions

Now we return to the question of how to make Bayesian predictions (all implicitly condi-
tioned on a set of model choices M):
Z
P(y | x, D) = p(y, w | x, D) dw (11)
Z
= P(y | x, w) p(w | D) dw. (12)

MLPR:w10b Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 2

We can approximate the posterior with a Gaussian, p(w | D) ≈ N (w; w∗ , H −1 ), using the
Laplace approximation (or variational methods, next week). Using this approximation, we
still have an integral with no closed form solution:
Z
P(y = 1 | x, D) ≈ σ (w> x) N (w; w∗ , H −1 ) dw (13)
h i
= EN (w; w∗ ,H −1 ) σ(w> x) . (14)

However, this expectation can be simplified. Only the inner product a = w> x matters, so
we can take the average over this scalar quantity instead. The linear combination a is a
linear combination of Gaussian beliefs, so our beliefs about it are also Gaussian. By now you
should be able to show that

p( a) = N ( a; w∗ > x, x> H −1 x). (15)

Therefore, the predictions given the approximate posterior, are given by a one-dimensional
integral:

P(y = 1 | x, D) ≈ EN (a; w∗ > x, x> H −1 x) [σ( a)] (16)

Z
= σ ( a) N ( a; w∗ > x, x> H −1 x) da. (17)

One-dimensional integrals can be computed numerically to high precision.

Bishop p. 220 and Murphy Section 8.4.4.2 review a further approximation, which is quicker
to evaluate and provides an interpretable closed-form expression:

1
P(y = 1 | x, D) ≈ σ (κ w∗ > x), κ= q . (18)
π > −1
1+ 8x H x

Under this approximation, the predictions use the most probable or MAP weights. However,
the activation is scaled down (with κ) when the activation is uncertain, so that predictions
will be less confident far from the data (as they should be).

4 Is the Laplace approximation reasonable?

If we think that the Energy is well-behaved and sharply peaked around the mode of the
distribution, we might think that we can approximate it with a Taylor series. In one dimension
we write

∂E 1 ∂2 E
E(w∗ + δ) ≈ E(w∗ ) + δ + δ2 (19)
∂w w∗ 2 ∂w2 w∗
1
≈ E(w∗ ) + Hδ2 , (20)
2
∂E
where the second term disappears because ∂w is zero at the optimum. In multiple dimensions
this Taylor approximation generalizes to:

E(w∗ + δ) ≈ E(w∗ ) + 12 δ>Hδ. (21)

A quadratic energy (negative log-probability) implies a Gaussian distribution. The distribu-

tion is close to the Gaussian fit when the Taylor series is accurate.
For models with a fixed number of identifiable parameters, the posterior becomes tightly
peaked in the limit of large datasets. Then the Taylor expansion of the log-posterior doesn’t
need to be extrapolated far and will be accurate. Search term for more information: “Bayesian
central limit theorem”.

MLPR:w10b Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 3

5 The Laplace approximation doesn’t always work well!
Despite the theory above, it is easy for the Laplace approximation to go wrong.
In high dimensions, there are many directions in parameter space where there might only
be a small number of informative datapoints. Then the posterior could look like the first
asymmetrical example in this note.
If the mode and curvature are matched, but the distribution is otherwise non-Gaussian, then
the value of the densities won’t match2 .

As a result, the approximation of P(D) will be poor.

[The website version of this note has a question here.]
One way for a distribution to be non-Gaussian is to be multi-modal. The posterior of logistic
regression only has one mode, but the posterior for neural networks will be multimodal.
Even if capturing one mode is reasonable, an optimizer could get stuck in bad local optima.

In models with many parameters, the posterior will often be flat in some direction, where
parameters trade off each other to give similar predictions. When there is zero curvature in
some direction, the Hessian isn’t positive definite and we can’t get a meaningful approxima-
tion.

6 Further Reading
Bishop covers the Laplace approximation and application to Bayesian logistic regression in
Sections 4.4 and 4.5.
Or read Murphy Sections 8.4 to 8.4.4 inclusive. You can skip 8.4.2 on BIC.
Similar material is covered by MacKay, Ch. 41, pp492–503, and Ch. 27, pp341–342.
The Laplace approximation was used in some of the earliest Bayesian neural networks
although — as presented here — it’s now rarely used. However, the idea does occur in recent
work, such as on continual learning (Kirkpatrick et al., Google Deepmind, 2017) and a more
sophisticated variant is used by the popular statistical package, R-INLA.

2. The final two figures in this note come from previous MLPR course notes, by one of Amos Storkey, Chris
Williams, or Charles Sutton.

MLPR:w10b Iain Murray and Arno Onken, http://www.inf.ed.ac.uk/teaching/courses/mlpr/2020/ 4