EM coding

Xingjie Shi

Dec 7, 23

Example
In this example, we will assume our mixture components are fully specified Gaussian distributions (i.e the
means and variances are known), and we are interested in finding the maximum likelihood estimates of the
πk ’s.
Assume we have K = 2 components, so that:

Xi |Zi = 0 ∼ N (5, 1.5) (1)

Xi |Zi = 1 ∼ N (10, 2) (2)
(3)

The true mixture proportions will be P (Zi = 0) = 0.25 and P (Zi = 1) = 0.75. First we simulate data from
this mixture model:
# mixture components
mu.true = c(5, 10)
sigma.true = c(1.5, 2)

# determine Z_i
Z = rbinom(500, 1, 0.75)
# sample from mixture model

X <- rnorm(10000, mean=mu.true[Z+1], sd=sigma.true[Z+1])

hist(X,breaks=15)

1
 Histogram of X
1500
1000
Frequency

500
0

0 5 10 15

X
Now we write a function to compute the log-likelihood for the incomplete data, assuming the parameters are
known. This will be used to determine convergence:
 
Xn X2
ℓ(θ) = log  πk N (xi ; µk , σk2 )
 
| {z }
i=1 k=1
L[i,k]

compute.log.lik <- function(L, w) {

L[,1] = L[,1]*w[1]
L[,2] = L[,2]*w[2]
return(sum(log(rowSums(L))))
}

Since the mixture components are fully specified, for each sample Xi we can compute the likelihood
P (Xi |Zi = 0) and P (Xi |Zi = 1). We store these values in the columns of L:
l <- function(X, mu, sigma) {
L = matrix(NA, nrow=length(X), ncol= 2)
L[, 1] = dnorm(X, mean=mu[1], sd = sigma[1])
L[, 2] = dnorm(X, mean=mu[2], sd = sigma[2])
return (L)
}

Finally, we implement the E and M step in the EM.iter function below. The mixture.EM function is the
driver which checks for convergence by computing the log-likelihoods at each step.
mixture.EM <- function(X, w.init, mu.init, sigma.init) {

w.curr <- w.init

mu.curr <- mu.init
sigma.curr <- sigma.init

2
 # store log-likehoods for each iteration
log_liks <- c()
L <- l(X, mu.curr, sigma.curr)
ll <- compute.log.lik(L, w.curr)
log_liks <- c(log_liks, ll)
delta.ll <- 1e5

while(delta.ll > 1e-5) {

oneIter <- EM.iter(X, w.curr, L)
w.curr <- oneIter$w.next
mu.curr <- oneIter$mu.next
sigma.curr <- oneIter$sigma.next
L <- l(X, mu.curr, sigma.curr)
ll <- compute.log.lik(L, w.curr)
log_liks <- c(log_liks, ll)
delta.ll <- log_liks[length(log_liks)] - log_liks[length(log_liks)-1]
}
return(list(w.curr, mu.curr, sigma.curr, log_liks))
}

EM.iter <- function(X, w.curr, L, ...) {

K <- ncol(L)
# E-step: compute E_{Z|X,w0}[I(Z_i = k)]
z_ik <- L
for(i in 1:K) {
z_ik[,i] <- w.curr[i]*z_ik[,i]
}
gam <- z_ik / rowSums(z_ik)
Nk <- colSums(gam)

# M-step
w.next <- Nk/sum(Nk)
mu.next <- colSums(gam*X)/Nk
sigma.next <- numeric(K)
for(i in 1:K) {
sigma.next[i] <- sqrt(sum(gam[,i] * (X - mu.next[i])ˆ2)/Nk[i])
}
list (w.next = w.next,
mu.next = mu.next,
sigma.next = sigma.next)
}
#perform EM
ee <- mixture.EM(X, w.init=c(0.5,0.5), mu.init=c(4,12), sigma.init = c(2,2))
print(paste("proportion = (", round(ee[[1]][1],2), ",", round(ee[[1]][2],2), ")", sep=""))

## [1] "proportion = (0.23,0.77)"

print(paste("mu = (", round(ee[[2]][1],2), ",", round(ee[[2]][2],2), ")", sep=""))

## [1] "mu = (4.98,9.94)"

print(paste("sigma = (", round(ee[[3]][1],2), ",", round(ee[[3]][2],2), ")", sep=""))

## [1] "sigma = (1.55,2.02)"

3
Finally, we inspect the evolution of the log-likelihood and note that it is strictly increases:
plot(ee[[4]], ylab='marginal log-likelihood', xlab='iteration')
marginal log−likelihood

−26000
−28000

0 50 100 150

iteration

HW
Please provide an EM algorithm for GMM with arbitrary number of components and dimensions.