Scribd
Upload
4 views11 pages

L08 GMM

The document discusses Gaussian Mixture Models (GMM) as a method for estimating class conditionals using a sum of multivariate Gaussian densities, particularly when dealing with small datasets. It outlines the Expectation-Maximization (EM) algorithm for estimating GMM parameters and highlights issues such as sensitivity to initialization and the selection of the number of Gaussian components. The content is based on lecture notes from Amir Atiya and Mohand Saïd Allili.

Uploaded by

amrmgoma6
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views11 pages

L08 GMM

The document discusses Gaussian Mixture Models (GMM) as a method for estimating class conditionals using a sum of multivariate Gaussian densities, particularly when dealing with small datasets. It outlines the Expectation-Maximization (EM) algorithm for estimating GMM parameters and highlights issues such as sensitivity to initialization and the selection of the number of Gaussian components. The content is based on lecture notes from Amir Atiya and Mohand Saïd Allili.

Uploaded by

amrmgoma6
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Pattern Classification

08. Gaussian Mixture Model

AbdElMoniem Bayoumi, PhD

Spring 2025
Recap: Kernel Density Estimator

0.75
True density
0.25

1 2

0.75
Density Estimation
0.25

2
Gaussian Mixture Model (GMM)
• Assume we have a small data set à not
possible to estimate class conditionals
using kernel density estimator

• Instead, we model each class conditional as


a sum of multivariate Gaussian densities

3
Gaussian Mixture Model (GMM)
• The parameters, i.e., the mean vectors &
covariance matrices, are determined so
that this sum approximates as good as
possible the given class conditional density

$ #
% ((%)! )" +#$ ((%)! )
𝑒 & !
𝑃! 𝑋 = % 𝑤! , #
!"# 2𝜋 & 𝑑𝑒𝑡 & (Σ! )
$

= % 𝑤! 𝑁(𝑋, 𝜇! , Σ! )
!"#
𝒘𝒋 ≡ represents the probability of each mixture component
𝑵(𝑿, 𝝁𝒋 , 𝜮𝒋 ) ≡ multi-variate Gaussian density with mean 𝝁𝒋 and covariance 𝜮𝒋 4
Gaussian Mixture Model (GMM)
-

𝑃! 𝑋 = % 𝑤* 𝑁(𝑋, 𝜇* , Σ* )
*+,

Condition:
-

% 𝑤* = 1
*+,
Because we need:
/ - /

. 𝑃! 𝑋 = % 𝑤* . 𝑁(𝑋, 𝜇* , Σ* ) = 1
./ *+, ./

=1
5
1-D Example𝑲
, 𝒘𝒋 𝑵(𝑿, 𝝁𝒋 , 𝝈𝒋 ) à approx. 𝑷(𝑿| 𝑪𝒊 )
𝒋*𝟏
𝑷(𝑿| 𝑪𝒊 )

𝒘𝟏 𝑵(𝑿, 𝝁𝟏 , 𝝈𝟏 ) 𝒘𝟐 𝑵(𝑿, 𝝁𝟐 , 𝝈𝟐 ) 𝒘𝟑 𝑵(𝑿, 𝝁𝟑 , 𝝈𝟑 )

6
Expectation–Maximization (EM)
• Apply EM algorithm, which is an iterative
algorithm, to estimate the parameters of
the GMM components

• For simplicity assume 2-component case


,i.e.,
𝑃! 𝑋|𝐶- = 𝑤 𝑁 𝑋, 𝜇# , Σ# + 1 − 𝑤 𝑁(𝑋, 𝜇& , Σ& )

7
Expectation–Maximization (EM)
1. Take initial guesses for the parameters: 𝑤, 𝜇! , Σ! , 𝜇" and Σ"
2. Expectation step: compute the responsibilities:
𝑤𝑁
+ 𝑋(𝑚), 𝜇(! , Σ1!
𝛾(# =
+ 𝑁 𝑋(𝑚), 𝜇(! , Σ1! + 1 − 𝑤
𝑤 + 𝑁 𝑋(𝑚), 𝜇(" , Σ1 "

𝛾(# represents the probability that 𝑋(𝑚) is generated from component 1

3. Maximization step: compute the weighted means & covariance matrices:


∑$ %! '(#)
!"# & ∑$
!"# !*%
&! '(#)
𝜇(! = ∑$ %!
, 𝜇(" = ∑$
!"# & !"#(!*%
&! )

% %
∑$ %! ' # *%
!"# & +# ' # *%
+# ∑$
!"# !*%
&! ' # *%
+& ' # *%
+&
Σ1! = ∑$ %!
, Σ1 " = ∑$
!"# & !"#(!*%
&! )
∑-#,! 𝛾 (#
𝑤
+=
𝑀

4. Iterate steps 2 & 3 until convergence

8
Expectation–Maximization (EM)
𝛾7 = 𝑃 𝑋 𝑚 ∈ component 1

𝑃 comp. 1 𝑃 𝑋 𝑚 |comp. 1
= apply Bayes rule
𝑃 𝑋 𝑚

𝑃 comp. 1 𝑃 𝑋 𝑚 |comp. 1
=
𝑃 comp. 1 𝑃 𝑋 𝑚 |comp. 1 + 𝑃 comp. 2 𝑃 𝑋 𝑚 |comp. 2

8 9 :(7),;! ,<!
≡
8 9 :(7),;! ,<! = >?8 9 :(7),;" ,<"

9
Issues with GMM
• Initialization:
– EM is an iterative algorithm which is very
sensitive to initial conditions:
– Start from trash → end up with trash
– Usually, we use the K-Means to get a good
initialization

• Number of Gaussian Components:


– Try different number of Gaussian components
and choose the best based on validation set.

10
Acknowledgment
• These slides have been created relying on
lecture notes of Amir Atiya and Mohand
Saïd Allili

11

You might also like