Mixture Models and

the EM Algorithm

Christopher M. Bishop
Microsoft Research, Cambridge
1 1

0.5
0.5
2006 Advanced Tutorial
Lecture Series, CUED 0
0
0 0.5 1 0 0.5 1
(a) (b)
 Applications of Machine Learning

• Web search, email spam detection, collaborative

filtering, game player ranking, video games, real-time
stereo, protein folding, image editing, jet engine anomaly
detection, fluorescence in-situ hybridisation, signature
verification, satellite scatterometer, cervical smear
screening, human genome analysis, compiler
optimization, handwriting recognition, breast X-ray
screening, fingerprint recognition, fast spectral analysis,
one-touch microwave oven, monitoring of premature
babies, text/graphics discrimination, event selection in
high energy physics, electronic nose, real-time tokamak
control, crash log analysis, QSAR, backgammon, sleep
EEG staging, fMRI analysis, speech recognition, natural
language processing, face detection, data visualization,
computer Go, satellite track removal, iris recognition, …
Three Important Developments
• 1. Adoption of a Bayesian framework
• 2. Probabilistic graphical models
• 3. Efficient techniques for approximate inference
Illustration: Bayesian Ranking
• Goal is to rank player skill from outcome of games
• Conventional approach: Elo (used in chess)
– maintains a single strength value for each player
– cannot handle team games, or more than 2 players
Bayesian Ranking: TrueSkillTM
• Ralf Herbrich, Thore Graepel, Tom Minka
• Xbox 360 Live (November 2005)
– millions of players
– billions of service-hours
– hundreds of thousands of game outcomes per day
• First “planet-scale” application of Bayesian methods?
• NIPS (2006) oral
Expectation Propagation on a Factor Graph
New Book
• Springer (2006)
• 738 pages, hardcover
• Full colour
• Low price
• 431 exercises + solutions
• Matlab software and
companion text with
Ian Nabney

http://research.microsoft.com/~cmbishop/PRML
Mixture Models and EM
• K-means clustering
• Gaussian mixture model
• Maximum likelihood and EM
• Bayesian GMM and variational inference

Please ask questions!

Old Faithful
 Old Faithful Data Set

Time
between
eruptions
(minutes)

Duration of eruption (minutes)

K-means Algorithm
• Goal: represent a data set in terms of K clusters each of
which is summarized by a prototype
• Initialize prototypes, then iterate between two phases:
– E-step: assign each data point to nearest prototype
– M-step: update prototypes to be the cluster means
• Simplest version is based on Euclidean distance
– re-scale Old Faithful data
Responsibilities
• Responsibilities assign data points to clusters

such that

• Example: 5 data points and 3 clusters

K-means Cost Function

data

responsibilities prototypes
Minimizing the Cost Function
• E-step: minimize J w.r.t.

• M-step: minimize J w.r.t

• Convergence guaranteed since there is a finite number

of possible settings for the responsibilities
Probabilistic Clustering
• Represent the probability distribution of the data as a
mixture model
– captures uncertainty in cluster assignments
– gives model for data distribution
– Bayesian mixture model allows us to determine K
• Consider mixtures of Gaussians
 The Gaussian Distribution
• Multivariate Gaussian

mean covariance

x2 x2 x2

x1 x1 x1
(a) (b) (c)
Likelihood Function
• Data set

• Consider first a single Gaussian

• Assume observed data points generated independently

• Viewed as a function of the parameters, this is known as

the likelihood function
Maximum Likelihood
• Set the parameters by maximizing the likelihood function
• Equivalently maximize the log likelihood
Maximum Likelihood Solution
• Maximizing w.r.t. the mean gives the sample mean

• Maximizing w.r.t covariance gives the sample covariance

Gaussian Mixtures
• Linear super-position of Gaussians

• Normalization and positivity require

• Can interpret the mixing coefficients as prior probabilities

Example: Mixture of 3 Gaussians

0.5

0
0 0.5 1
(a)
Contours of Probability Distribution

0.5

0
0 0.5 1
(b)
Surface Plot
Sampling from the Gaussian
• To generate a data point:
– first pick one of the components with probability
– then draw a sample from that component
• Repeat these two steps for each new data point
Synthetic Data Set

0.5

0
0 0.5 1
(a)
Fitting the Gaussian Mixture
• We wish to invert this process – given the data set, find
the corresponding parameters:
– mixing coefficients
– means
– covariances
• If we knew which component generated each data point,
the maximum likelihood solution would involve fitting
each component to the corresponding cluster
• Problem: the data set is unlabelled
• We shall refer to the labels as latent (= hidden) variables
Synthetic Data Set Without Labels

0.5

0
0 0.5 1
(b)
Posterior Probabilities
• We can think of the mixing coefficients as prior
probabilities for the components
• For a given value of we can evaluate the
corresponding posterior probabilities, called
responsibilities
• These are given from Bayes’ theorem by
Posterior Probabilities (colour coded)

0.5

0
0 0.5 1
(a)
 Latent Variables

1 1 1

0.5 0.5 0.5

0 0 0
0 0.5 1 0 0.5 1 0 0.5 1
(a) (b) (a)
Maximum Likelihood for the GMM
• The log likelihood function takes the form

• Note: sum over components appears inside the log

• There is no closed form solution for maximum likelihood
Over-fitting in Gaussian Mixture Models
• Singularities in likelihood function when a component
‘collapses’ onto a data point:

then consider

• Likelihood function gets larger as we add more

components (and hence parameters) to the model
– not clear how to choose the number K of components
Problems and Solutions
• How to maximize the log likelihood
– solved by expectation-maximization (EM) algorithm
• How to avoid singularities in the likelihood function
– solved by a Bayesian treatment
• How to choose number K of components
– also solved by a Bayesian treatment
EM Algorithm – Informal Derivation
• Let us proceed by simply differentiating the log likelihood
EM Algorithm – Informal Derivation
• Similarly for the covariances

• For mixing coefficients use a Lagrange multiplier to give

EM Algorithm – Informal Derivation
• The solutions are not closed form since they are coupled
• Suggests an iterative scheme for solving them:
– make initial guesses for the parameters
– alternate between the following two stages:
1. E-step: evaluate responsibilities
2. M-step: update parameters using ML results
• Each EM cycle guaranteed not to decrease the likelihood
Relation to K-means
• Consider GMM with common covariances
• Take limit
• Responsibilities become binary

• EM algorithm is precisely equivalent to K-means

Bayesian Mixture of Gaussians
• Include prior distribution over parameters

• Make predictions by marginalizing over parameters

– c.f. point estimate from maximum likelihood
Bayesian Mixture of Gaussians
• Conjugate priors for the parameters:
– Dirichlet prior for mixing coefficients

– Normal-Wishart prior for means and precisions

where the Wishart distribution is given by

Variational Inference
• Exact solution is intractable
• Variational inference:
– extension of EM
– alternate between updating posterior over parameters
and posterior over latent variables
– again convergence is guaranteed
Illustration: a single Gaussian
• Convenient to work with precision

• Likelihood function

• Prior over parameters

Variational Inference
• Goal is to find true posterior distribution

• Factorized approximation

• Alternately update each factor to minimize a measure of

closeness between the true and approximate distributions
Initial Configuration

2
(a)
τ

0
−1 0 µ 1
After Updating

2
(b)
τ

0
−1 0 µ 1
After Updating

2
(c)
τ

0
−1 0 µ 1
Converged Solution

2
(d)
τ

0
−1 0 µ 1
Variational Equations for GMM
Sufficient Statistics

• Small computational overhead compared to maximum

likelihood EM
Bayesian Model Comparison
• Multiple models (e.g. different values of K) with priors

• Posterior probabilities

• For equal priors, models are compared using evidence

• Variational inference maximizes lower bound on

Evidence vs. K for Old Faithful
Bayesian Model Complexity
Take-home Messages
• Maximum likelihood gives severe over-fitting
– singularities
– favours ever larger numbers of components
• Bayesian mixture of Gaussians
– no singularities
– determines optimal number of components
• Variational inference
– effective solution for Bayesian GMM
– little computational overhead compared to EM
 Viewgraphs, tutorials and
publications available from:
http://research.microsoft.com/~cmbishop