lecture5-maximum-likelihood

September 15, 2020

1 Lecture 5: Maximum Likelihood Learning

1.0.1 Applied Machine Learning

Volodymyr KuleshovCornell Tech

2 Why Does Supervised Learning Work?

Prevously, we saw one way of explaining why supervised learning works.

3 Part 1: Probabilistic Modeling

In this lecture, we are going to look at why supervised learning works from a new, probabilistic
perspective.
First, we are going to start by defining the probabilistic approach to machine learning and set up
some notation.

4 Review: Machine Learning Models

A machine learning model is a function

f :X →Y

that maps inputs x ∈ X to targets y ∈ Y.

Often, models have parameters θ ∈ Θ living in a set Θ. We will then write the model as

fθ : X → Y

to denote that it’s parametrized by θ.

1
 5 Review: Data Distribution

We will assume that the dataset is governed by a probability distribution P, which we will call the
data distribution. We will denote this as

x, y ∼ P.

The training set D = {(x(i) , y (i) ) | i = 1, 2, ..., n} consists of independent and identicaly distributed
(IID) samples from P.

6 Probabilistic Models

A probabilistic model is a probability distribution

P (x, y) : X × Y → [0, 1].

This model can approximate the data distribution P(x, y).

Probabilistic models also have parameters θ ∈ Θ, which we denote as

Pθ (x, y) : X × Y → [0, 1].

If we know Pθ (x, y), we can use the conditional Pθ (y|x) for prediction.

7 Probabilistic Models: Example

Consider a simple version of our example with predicting diabetes from BMI. * For the target
Y = {0, 1}, we discretize the diabetes risk score into low risk (y = 0) and high risk (y = 1). * For
the input X = {0, 1, 2}, we also discretize the BMI into low (x = 0), medium (x = 1), and high
(x = 2).
Then the following is a simple probabilistic model.
[18]: import pandas as pd

df_model = pd.DataFrame.from_records([
['low', 'low', 0.20], ['medium', 'low', 0.1], ['high', 'low', 0.2],
['low', 'high', 0.05], ['medium', 'high', 0.1], ['high', 'high', 0.35],
], columns=['BMI $x$', 'Risk $y$', 'P'])
df_model

[18]: BMI $x$ Risk $y$ P

0 low low 0.20
1 medium low 0.10
2 high low 0.20
3 low high 0.05
4 medium high 0.10

2
 5 high high 0.35

Under this model, we can compute P (y|x) = P (x, y)/P (x) as follows.

[26]: df_px = df_model.groupby('BMI $x$').sum().rename(columns={'P' : 'Px'})

df_conditional_model = df_model.merge(df_px, on='BMI $x$', right_index=True)
df_conditional_model['$P(y|x)$'] = df_conditional_model['P'] /␣
,→df_conditional_model['Px']

df_conditional_model.iloc[:,[0,1,4]]

[26]: BMI $x$ Risk $y$ $P(y|x)$

0 low low 0.800000
3 low high 0.200000
1 medium low 0.500000
4 medium high 0.500000
2 high low 0.363636
5 high high 0.636364

8 Why Use Probabilistic Models?

The probabilistic approach to machine learning is powerful.

• We can fit models that capture predictive uncertainty.
• We can construct models in a more principled way by explicitly modeling the data distribution.
• It offers a new perspective on why supervised learning works.
# Part 2: Monte Carlo Estimation
Next, we are going to define Monte Carlo sampling, a mathematical tool that will be important in
this lecture and later in the course.

9 Notation: Random Variable

Suppose that we have a variable x ∈ X that is governed by a distribution P:

x ∼ P(x).

This x can be a sample from a data distribution, or any other random variable.

10 Notation: Expected Value

Recall that the expected value of a function g : X → R when the input x to g is sampled from P is
given by ∑
Ex∼P [g(x)] = g(x)P (x),
x

3
 where we assumed for simplicity that x is discrete.
In practice computing expected values is not always easy: * x can take on a very large number of
values and summing over all of them is not possible. * When x is continuous, the expected value
can be an integral with no closed form solution.
In practice, we often use approximate methods to compute expected values.

11 Monte Carlo Estimation

Monte Carlo estimation is a way to approximately compute exepected values

∑
Ex∼P [g(x)] = g(x)P (x).
x

1. We first generate T IID samples x1 , . . . , xT from P .

2. Then we estimate the expected value as:

1∑
T
ĝ(x1 , · · · , xT ) ≜ g(xt )
T
t=1

We call ĝ the Monte Carlo estimate of the expected value.

12 Monte Carlo Estimation: Example

Let’s say that we throw five dice. What is the expected number of twos?
• Let x = (x1 , x2 , . . . , x5 ) be a dice roll where xj ∈ {1, 2, . . . , 6} is the outcome of the j-th die.
• Let g(x) denote the number of twos in the roll of dice x.
∑
The expected value Ex∼P [g(x)] = x g(x)P (x) is the expected number of twos. We can calculate
it as follows
[57]: import numpy as np

# sample 10,000 rolls of five dice

dice_rolls = np.random.randint(0, 6, size=(5,10000))

# count the number of twos in each throw

TWO_VAL = 1 # twos are denoted by 1 because of zero-based indexing
num_twos = (dice_rolls==TWO_VAL).sum(axis=0).mean()

print('MC Estimate: %.4f' % num_twos)

MC Estimate: 0.8358
This makes sense, since the correct answer is 5/6 ≈ 0.83.

4
13 Properties of Monte Carlo Estimation

The Monte Carlo estimate ĝ has the following properties:

• It is an unbiased estimate of the true expectation:

EP [ĝ] = EP [g(x)]
• It converges to the true expectation as we average additional samples.

1∑
T
ĝ = g(xt ) → EP [g(x)] for T → ∞
T
t=1

• It’s variance decreases to zero as we collect more samples:

[ ]
1∑
T
varP [g(x)]
varP [ĝ] = varP g(xt ) =
T T
t=1

Thus, variance of the estimator can be reduced by increasing the number of samples.

14 Monte Carlo: Summary

• A lot of problems in ML require computing intractable expected values.

• Monte Carlo estimation is a simple approximate method that computes expected values ap-
proximately.
# Part 3: Maximum Likelihood
Maximum likelihood learning is a general way of training machine learning models. Many algo-
rithms we’ve seen so far implicitly use this principle.

15 Review: Data Distribution

We will assume that the dataset is governed by a probability distribution P, which we will call the
data distribution. We will denote this as

x, y ∼ Pdata .

The training set D = {(x(i) , y (i) ) | i = 1, 2, ..., n} consists of independent and identicaly distributed
(IID) samples from Pdata .

16 Review: Probabilistic Models

A probabilistic model is a probability distribution

Pθ (x, y) : X × Y → [0, 1].

5
This model can approximate the data distribution Pdata (x, y).
Probabilistic models may also have parameters θ ∈ Θ, which we denote as

Pθ (x, y) : X × Y → [0, 1].

If we know P (x, y), we can use the conditional P (y|x) for prediction.

17 Learning Probabilistic Models

We now have a probabilistic model and a data distribution. Thus, it is natural to try to learn learn
a good probability distribution Pθ (x, y) that approximates Pdata (x, y).
What are the characteristics of a good model Pθ (x, y)? * Predictive accuracy: correctly predicting
y from x. * Does this patient have diabetes or not?
• Understanding the relationship between x, y?
– What physiological features of the patient influence their diabetes risk?
• Density estimation: approximating Pdata (x, y) so that we can later answer any query.

18 Kullback-Leibler Divergence

In order to approximate Pdata with Pθ , we need a measure of distance between distributions.

A standard measure of similarity between distributions is the Kullback-Leibler (KL) divergence
between two distributions p and q, defined as
∑ p(x)
D(p∥q) = p(x) log .
x
q(x)

Observations:
• D(p ∥ q) ≥ 0 for all p, q, with equality if and only if p = q. Proof:
( )
q(x) q(x)
D(p∥q) = Ex∼p − log ≥ − log Ex∼p
p(x) p(x)
( )
∑ q(x)
= − log p(x) =0
x
p(x)

• The KL-divergence is asymmetric, i.e., D(p∥q) ̸= D(q∥p)

• It has roots in information theory.

6
19 Learning Models Using KL Divergence

We may now learn a probabilistic model Pθ (x, y) that approximates Pdata (x, y) via the KL diver-
gence:
( )
Pdata (x, y)
D(Pdata || Pθ ) = Ex,y∼Pdata log
Pθ (x, y)
∑ Pdata (x, y)
= Pdata (x, y) log
x,y
Pθ (x, y)

Note that D(Pdata || Pθ ) = 0 iff the two distributions are the same.

20 From KL Divergence to Log Likelihood

$
$ We can simplify the KL divergence objective somewhat:
( )
Pdata (x, y)
D(Pdata || Pθ ) = Ex,y∼Pdata log
Pθ (x, y)
= Ex,y∼Pdata log Pdata (x, y) − Ex,y∼Pdata log Pθ (x, y)

The first term does not depend on Pθ : minimizing KL divergence is equivalent to maximizing the
expected log-likelihood.

arg min D(Pdata || Pθ ) = arg min −Ex,y∼Pdata log Pθ (x, y)

Pθ Pθ

= arg max Ex,y∼Pdata log Pθ (x, y)

Pθ

We have now defined a learning objective equivalent to optimize the KL divergence:

arg max Ex,y∼Pdata log Pθ (x, y)

Pθ

• This asks that Pθ assign high probability to instances sampled from Pdata , so as to reflect the
true distribution.
• Because of log, samples x, y where Pθ (x, y) ≈ 0 weigh heavily in the objective.
Problem: In general we do not know Pdata , hence expected value is intractable.

21 Maximum Likelihood Estimation

$
$ Applying, Monte Carlo estimation, we may approximate the expected log-likelihood

Ex,y∼Pdata log Pθ (x, y)

7
 with the empirical log-likelihood:
1 ∑
ED∼log Pθ (x,y) = log Pθ (x, y)
|D|
x,y∈D

Maximum likelihood learning is then:

1 ∑
max log Pθ (x, y).
Pθ |D|
x,y∈D

22 Example: Flipping a Random Coin

$ $ Consider a simple example in which we repeatedly toss a biased coin and record the outcomes.
• There are two possible outcomes: heads (H) and tails (T ). A training dataset consists of
tosses of the biased coin, e.g., D = {H, H, T, H, T }
• Assumption: true probability distribution is Pdata (x), x ∈ {H, T }
• Our task is to model the probability of heads/tails. Our class of models M are Bernoulli
distributions over x ∈ {H, T }.

23 Example: Flipping a Random Coin

How should we choose Pθ (x) from M if 3 out of 5 tosses are heads in D? Let’s apply maximum
likelihood learning.
• Our model is Pθ (x = H) = θ and Pθ (x = T ) = 1 − θ
• Our data is: D = {H, H, T, H,∏T }
• The likelihood of the data is i Pθ (xi ) = θ · θ · (1 − θ) · θ · (1 − θ).
We optimize for θ which makes D most likely. What is the solution in this case?
[5]: %matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

# our dataset is {H, H, T, H, T}; if theta = P(x=H), we get:

coin_likelihood = lambda theta: theta*theta*(1-theta)*theta*(1-theta)

theta_vals = np.linspace(0,1)
plt.plot(theta_vals, coin_likelihood(theta_vals))

[5]: [<matplotlib.lines.Line2D at 0x121769a20>]

8
24 Example: Flipping a Random Coin

Our log-likelihood function is

L(θ) = θ# heads · (1 − θ)# tails

log L(θ) = log(θ# heads · (1 − θ)# tails )
= # heads · log(θ) + # tails · log(1 − θ)

The MLE estimate is the θ∗ ∈ [0, 1] such that log L(θ∗ ) is maximum.
Differentiating the log-likelihood function with respect to θ and setting the derivative to zero, we
obtain
# heads
θ∗ =
# heads + # tails

When exact solutions are not available, we can optimize the log likelihood numerically, e.g. using
gradient descent.
We will see examples of this later.

25 Conditional Maximum Likelihood

Sometimes, we may be interested in only fitting a conditional model P (y|x). For example, we may
be only interested in predicting y from x rather than learning the joint structure of x, y.

9
We can extend the principle of maximum likelihood learning to this setting as well. In this case,
we are interested in minimizing

min Ex∼Pdata [D(Pdata (y|x) || Pθ (y|x))] ,

θ

the expected KL divergence between Pdata (y|x) and Pθ (y|x) over all the inputs x.
With a bit of math, we can show that the maximum likelihood objective becomes

max Ex,y∼Pdata log Pθ (y|x).

θ

This is the principle of conditional maximum likelihood

# Part 4: Extensions of Maximum Likelihood
Maximum likelihood learning is one approach for training probabilistic machine learning models.
An evern more general approach comes from Bayesian statistics. We briefly overview the Bayesian
approach in this lesson.

26 Review: Maximum Likelihood Learning

Recall that in maximum likelihood learning, we are optimizing the following objective:

θMLE = arg max Ex,y∼Pdata log P (x, y; θ).

θ

27 The Frequentist Approach

So far, we viewed the parameter θMLE as a fixed but unknown quantity that we want to determine.

θMLE = arg max Ex,y∼Pdata log P (x, y; θ).

θ

This view is an example of the frequentist approach in statistics: there exists some true value of
θMLE and our job is to devise statistical procedure to estimate this value.

28 The Bayesian Approach

In Bayesian statistics, θ is a random variable whose value happens to be unknown.

We formulate two models: * A likelihood model P (x, y|θ) that defines the probability of x, y for
any fixed value of θ. * A prior P (θ) that specifies us existing belief about the distribution of the
random variable θ.
Together, these two models define the joint distribution

P (x, y, θ) = P (x, y | θ)P (θ)

in which both the x, y and the parameters θ are random variables.

10
29 Bayesian Inference and Learning

How do we estimate the parameter θ that is consistent with a given dataset D =

{(x(1) , y (1) ), (x(2) , y (2) ), . . . , (x(n) , y (n) )}?
Since the variable θ is a random value, in the Bayesian approach we are interested in the posterior
probability P (θ | D) of θ given the dataset D.
How do we obtain P (θ | D)? This value is computed using Bayes’ rule:

P (D | θ)P (θ)
P (θ | D) =
P (D)
P (D | θ)P (θ)
=∫ ,
θ P (D, θ)P (θ)dθ
∏n
where P (D | θ) = i=1 P (x
(i) , y (i) | θ).

30 Bayesian Predictions

Suppose we now want to predict the value of y from x. Unlike in the frequentist setting, we no
longer have a single estimate θ of the model params, but instead we have a distribution.
The Bayesian approach to predicting y given an input x and a training dataset D consists of taking
the prediction of all the possible models
∫
P (y|x, D) = P (y | x, θ)P (θ | D)dθ.
θ

This is called the posterior predictive distribution. Note how each P (y | x, θ) is weighted by the
probability of θ given D.

31 The Pros and Cons of the Bayesian Approach

The Bayesian approach is very powerful. Some of its advantages include: * Principled estimates
of uncertainty, both in the prediction and in the paramters of the model. * Ability to incorporate
prior knowledge via the prior. * Providing a general framework for reasoning about probabilistic
models.
The main disadvantage is by far the computational complexity. Averaging over all possible model
weights is typically intracatble. There exists an entire field of machine learning that studies how
to approximate it.

11
32 Maximum A Posteriori Learning

Instead of trying to use the posterior distribution of P (θ|D), a common approach is to approximate
this distribution by its most likely value:

θMAP = arg max log P (θ|D)

θ
= arg max (log P (D | θ) + log P (θ) − log P (D))
θ
( )
∏n
= arg max log P (x(i) , y (i) | θ) + log P (θ) ,
θ
i=1

where in the second line we used Bayes’ theorem and in the third line we used the fact that P (D)
does not depend on θ.
Thus, we have the following objective:
( )
∏
n
arg max log (i)
P (x , y (i)
| θ) + log P (θ) .
θ
i=1

The θMAP is known as the maximum a posteriori estimate. Note that we used the same formula
as we used for maximum likelihood, except that we have added the prior term log P (θ).

33 Example: Flipping a Random Coin

How should we choose P (x | θ) from M if 60 out of 100 tosses are heads in D? Let’s apply
maximum likelihood learning.
• Our model is P (x = H | θ) = θ and P (x = T | θ) = 1 − θ
• Our data is: D = {H, H, T, H,∏T }
• The likelihood of the data is i P (xi | θ) = θ · θ · (1 − θ) · θ · (1 − θ).
Let’s now make this a MAP problem. Let’s assume the prior follows the Beta distribution:
1
P (θ) = θα (1 − θ)β ,
B(α + 1, β + 1)

where α, β > 0 are parameters and B is the Beta function.

The joint probability on D = {H, H, T, H, T } is then
∏ θα (1 − θ)β
P (xi | θ)P (θ) = θ · θ · (1 − θ) · θ · (1 − θ)
B(α + 1, β + 1)
i

Let’s derive an analytic solution. Our objective function is

L(θ) ∝ θ# heads · (1 − θ)# tails · θα · (1 − θ)β

log L(θ) = log(θ# heads · (1 − θ)# tails · θα · (1 − θ)β ) + const.
= (# heads + α) · log(θ) + (# tails + β) · log(1 − θ)

12
 Differentiating the log-likelihood function with respect to θ and setting the derivative to zero, we
obtain
# heads + α
θ∗ =
# heads + # tails + α + β

Thus, we see that adding a Beta prior with parameters α, β allows to encode having seen α “virtual
heads” and β “virtual tails”.
This is an example of how we can add prior knowledge into the model.
[59]: %matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

# our dataset is {H, H, T, H, T}; if theta = P(x=H), we get:

alpha, beta = 1, 1
coin_likelihood = lambda theta:␣
,→theta*theta*(1-theta)*theta*(1-theta)*(theta**alpha)*((1-theta)**beta)

theta_vals = np.linspace(0,1)
plt.plot(theta_vals, coin_likelihood(theta_vals))

[59]: [<matplotlib.lines.Line2D at 0x122266b38>]

13