Note Set 5: Hidden Markov Models

Probabilistic Learning: Theory and Algorithms, CS 274A, Winter 2016

1 Hidden Markov Models (HMMs)

1.1 Introduction

Consider observed data vectors xt that are d-dimensional and where the subscript t indicates discrete time
or position in a sequence, t = 1, . . . , T . We can assume for notational simplicity that the xt vector is
real-valued, but in general xt could be discrete or could be a mixture of discrete and real-valued.
Our data consists of D = {x1 , . . . , xT }. Unlike the IID assumption we have made in the past we would
now like to model the sequential dependence among the xt s. One approach is to use a hidden Markov
model where we assume that the xt s are noisy stochastic functions of an unobserved (hidden) Markov chain
denoted by zt , where zt is a discrete random variable taking one of K possible values, zt {1, . . . , K}. zt
is often referred to as the state variable.
The generative model for a hidden Markov model is simple: at each time step t, a data vector xt is
generated conditioned on the state zt , and the Markov chain then transitions to a new state zt+1 to generate
xt+1 , and so on. As with standard Markov chains there is an initial distribution over the K states to
initialize the chain.
There are two key assumptions in a hidden Markov model:

1. Observations xt are conditionally independent of all other variables given zt , so the observation at
time t depends only on the current state zt .

2. The zt s form a (first order) Markov chain, i.e., p(zt |zt1 , . . . , z1 ) = p(zt |zt1 ), t = 2, . . . , T . The
chain is also typically assumed to be homogeneous in that the transition probabilities do not depend
on t.

The earliest application of hidden Markov models was in speech recognition in the 1970s, but they have
since been used for various other problems in bioinformatics, language modeling, economics, and climate
modeling.
We will use the shorthand notation x[1,T ] for x1 , . . . , xT , and z[1,T ] for z1 , . . . , zT . Our observed data is
D = {x1 , . . . , xT }. From our graphical model we have
T
Y
p(x[1,T ] , z[1,T ] ) = p(xt |zt )p(zt |zt1 )
t=1

1
Note Set 5: Hidden Markov Models: CS 274A, Probabilistic Learning: Fall 2016 2

(where p(z1 |z0 ) = , the initial distribution on states.)

We have two sets of parameters:

Transition matrix, K K matrix A, with aij = p(zt = j|zt1 = i), 1 i K.

K emission distributions/densities p(xt |zt = j), j = 1 . . . K, e.g., multivariate Gaussian for real-
valued xt , and usually assumed to be homogeneous (i.e., does not depend on t). If xt is very high-
dimensional it is common to assume that the components of x are conditionally independent given
zt .

For simplicity we will assume that the initial distribution on states is known, e.g., set to the uniform
distribution, but if we had multiple sequences this could also be learned from the data. We will let indicate
the full set of parameters, i.e., the transition matrix parameters in A and the parameters of K state-dependent
emission distributions p(xt |zt = j), j = 1 . . . K.
Note the similarity of the HMM to a finite mixture model with K components. In particular, the HMM
can be viewed as adding Markov dependence to the unobserved component indicator variable z in a mixture
model.

1.2 Efficient Computation of HMM Likelihood

Below we show how to compute the likelihood L() (where both the K emission density parameters and
the transition matrix A are unknown)?

L() = p(D|)
= p(x[1,T ] |)
X
= p(x[1,T ] , z[1,T ] |)
z[1,T ]

But this sum is intractable to compute directly, since it has complexity O(K T ). However, we can use the
conditional independence assumptions (or equivalently the graphical model structure in the HMM) to carry
out this computation efficiently.
Let t (j) = p(zt = j, x[1,t] ), j = 1, . . . , K (implicitly conditioning on ). This is the joint probability
of (a) the unobserved state at time t being in state j and (b) all of the observed xt s up to and including time
Note Set 5: Hidden Markov Models: CS 274A, Probabilistic Learning: Fall 2016 3

t.

t (j) = p(zt = j, x[1,t] )

K
X
= p(zt = j, zt1 = i, x[1,t] )
i=1
X
= p(xt |zt = j, zt1 = i, x[1,t1] ) p(zt = j|zt1 = i, x[1,t1 ]) p(zt1 = i, x[1,t1] )
i
K
X
= p(xt |zt = j) p(zt = j|zt1 = i) p(zt1 = i, x[1,t1] )
i=1
XK
= p(xt |zt = j) aij t1 (i)
i=1

The first term is the evidence from the observation xt at time t, the second term is the transition probability,
and the final term is just t1 (i). This yields a simple recurrence relation for the s.
We can compute the t (j)s recursively, where t (j) = p(zt = j, x[1,t] ). We can do this in a single
forward pass, from t = 1 up to t = T , initializing the recursion with 0 (j) = (j). This is the forward part
of the well-known forward-backward algorithm for HMMs. Then given T (j), j = 1, . . . , K, the likelihood
P
can be computed as L() = j T (j), by the LTP.
If we know t1 (i), i = 1, . . . , K, we can compute t (j) in time O(K 2 + Kf (d)). The K 2 is because
we have to compute the probability for all i, j pairs, and the function f reflects the complexity of computing
the likelihood of the data vector xt for each possible state, e.g., f (d) = O(d2 ) for a Gaussian emission
density. The overall complexity of computing all of the s is O(T K 2 + T Kf (d)).

1.3 Efficient Computation of State Probabilities

In developing an EM algorithm for HMMs we will want to compute the probability of each possible state at
each time t given all of the observed data, i.e., p(zt = j|x[1,T ] ) (using all of the data, both before and after
t). We factor it as follows.

p(zt = j|x[1,T ] ) p(zt = j, x[1,t] , x[t+1,T ] )

= p(x[t+1,T ] |zt = j, x[1,t] ) p(zt = j, x[1,t] )
= p(x[t+1,T ] |zt = j) p(zt = j, x[1,t] )
= p(x[t+1,T ] |zt = j) t (j)

Note that given zt = j, the x[1,t] values give us no additional information about x[t+1,T ] (which is how we
get from the 2nd to the 3rd line above).
Define t (j) = p(x[t+1,T ] |zt = j), t = 1, . . . , T, j = 1, . . . , K. Then, from above, we have

p(zt = j|x[1,T ] ) t (j)t (j)

Note Set 5: Hidden Markov Models: CS 274A, Probabilistic Learning: Fall 2016 4

Using the same type of recursive decomposition as we used for t (j), the t (j)s can be computed in time
O(T K 2 + T Kf (d)), working backwards from t = T to t = 1.
Thus, to compute p(zt = j|x[1,T ] ), t = 1, . . . , T, j = 1, . . . , K:

We first recursively compute the t (j)s (forward step).

Next we recursively compute the t (j)s (backward step).

Finally, we can compute p(zt = j|x[1,T ] ) as

t (j)t (j)
wt (j) = p(zt = j|x[1,T ] ) = P
k t (k)t (k)
where the denominator is the normalization term. This yields a set of T K probabilities wt (j),
playing the same role in EM as the membership weights for mixture models.

2 EM for learning HMM Parameters

The EM algorithm for HMMs follows the same general idea as the EM algorithm for finite mixtures.
In the E-step we compute the probabilities wt (j) (or membership weights) of the unobserved states, for
each state j and each time t, conditioned on all of the data x[1,T ] and conditioned on the current parameters
.
In the M-step we compute point estimates of the parameters given the membership weights from the
E-step. There are two different sets of parameters: (1) the emission density parameters p(xt |zt = j), and
(2) the transition parameters aij , 1 i, j, K.
The estimation of the emission density parameters proceeds in exactly the same manner as for the finite
mixture case. For example, if the emission densities are Gaussian, then the membership weights are used to
generate fractional counts for estimating the mean and covariance for each of the K emission densities.
For the transition probabilities we proceed as follows. We first compute E[Nj ], the expected number of
times in state j, which is Tt=1 wt (j).
P

Next, we need to compute E[Nij ], the expected number of times we transition from state i to state j.
T
X 1
E[Nij ] = p(zt = i, zt+1 = j|x[1,T ] )
t=1
T
X 1
p(zt = i, zt=1 = j, x[1,T ] )
t=1

Letting t (i, j) = p(zt = i, zt=1 = j, x[1,T ] ), we have

t (i, j) = p(zt = i, zt=1 = j, x[1,T ] )
= p(x[t+2,T ] |zt+1 = j) p(xt+1 |zt+1 = j) p(zt+1 = j|zt = i) p(zt = i, x[1,t] )
= t+1 (j) p(xt+1 |zt+1 = j) aij t (i).
Note Set 5: Hidden Markov Models: CS 274A, Probabilistic Learning: Fall 2016 5

In going from the first to the second line we have used various conditional independence properties that
exist in the model. The final line consists of quantities that can easily be computed, e.g., they be computed
directly from the model (p(xt+1 |zt+1 = j)), or are known parameters (aij ), or have been computed during
the forward-backward computations of the E-step (the s and s).
We then normalize the s to get the conditional probabilities we need, i.e.,

t (i, j)
p(zt = i, zt+1 = j|x[1,T ] ) = P P
k1 k2 t (k1 , k2 )

from which we can compute E[Nij ] above.

The M step for the transition probabilities is now very simple:

E[Nij ]
aij = , 1 i, j K
E[Nj ]