Logistic Regression

Lecture 05
School of Energy Science & Engineering
 Linear Models

Generalized Linear Models (GLM)

Logistic Regression Linear Models

School of Energy Science & Engineering

 Linear Regression
• Using data to predict something falls under the category of
“machine learning”

• Calculate R2 and determine

if weight and size are
correlated. Large values
imply a large effect
• Calculate a p-value to
Size determine if the R2 value is
statistically significant.
• Use the line to predict size
for give weight

Weight
School of Energy Science & Engineering
R2 compares a measure of a good fit, SS(fit)
to a measure of a bad fit, SS(mean)

SS(mean) - SS( fit)

R =2

SS(mean)
• Size = 0.7 x weight + 0.86

Size

Weight
Weight

School of Energy Science & Engineering

 Logistic Regression
• Logistic regression is similar to linear regression, except
• Logistic regression predicts whether something is True or
False, instead of predicting something continuous like size

Obese

1
y= -x
1+ e

Not Obese

School of Energy Science & Engineering weight

 Logistic Regression
• Instead of fitting a line to the data, logistic regression fits an “S”
Shaped “logistic function
• The curve tells you the probability that a mouse is obese based
on its weight

Obese 1

The curve
goes from
0 to 1

Not Obese 0

School of Energy Science & Engineering Weight

 Logistic Regression
• We use a continuous variable (like weight) to predict obesity
• Although logistic regression tells the probability that a mouse is
obese or not, its usually used for classification
• For example, if the probability a mouse is obese is >50%, then
we’ll classify it as obese, otherwise we’ll classify it as “not
obese”
Obese

Probability
a mouse is
Obese

Not Obese

School of Energy Science & Engineering weight

 Logistic Regression
• Logistic regression’s ability to provide probabilties and classify
new samples using continuous and discrete measurements
makes it a popular machine learning method
• One big difference between linear regression and logistic
regression is how the line is fit to the data
• With linear regression, we fit the line using “least squares”. In
other words, we find the line that minimizes the sum of the
squares of these residuals (SSR)
• We also use the residuals to calculate R2 and to compare
simple models to complicated models
• Logistic regression doesn’t have the same concept of a
“residual”, so it can’t use least squares and it can’t calculate R2.
• Instead it uses something called “maximum likelihood”.

School of Energy Science & Engineering

 We find the line that minimizes the sum
of the squares of these residuals

School of Energy Science & Engineering

 Logistic Regression
• pick a probability, scaled by weight, of observing an obese mouse and use
that to calculate the likelihood of observing a non-obese mouse that weighs
this much
• Then we calculate the likelihood of observing all remaining mouse
• Lastly we multiply all of those likelihoods together. That’s the likelihood of the
data given as “S” Shaped line
• Finally, the curve with the maximum likelihood is selected

Obese

Not Obese
School of Energy Science & Engineering weight
 Log(odds)
The Odds in favor of my team wining the game are 5 to 3:

We can write this as log(odds) = log(5/3)

Probability of wining: p = 5/8

æ 5 ö
æ p ö ç ÷ æ 5ö
log(odds) = log ç ÷ = log ç 8 ÷ = log ç ÷
è 1- p ø çç 1- 5 ÷÷ è 3ø
è 8ø

School of Energy Science & Engineering

The y-axis in logistic regression is transformed
from the “probability of obesity to the log(odds
of obesity
Obese 1

Probability
Of Obesity

Not Obese 0

weight

æ p ö
Log(odds of obesity) = log(odds) = log ç ÷
School of Energy Science & Engineering è 1- p ø
1

Log(odds of obesity)
0

weight æ 0.88 ö
log ç ÷=2
æ 0.5 ö è 1- 0.88 ø
log ç ÷=0
è 1- 0.5 ø æ 0.95 ö
æ 0.731 ö
log ç ÷=3
è 1- 0.95 ø
log ç ÷ =1
è 1- 0.731ø
School of Energy Science & Engineering
The coefficient for the line in logistic Regression

The new y-axis transform

the squiggly line into a
straight line

Log(odds of obesity)
Y = -3.48 + 1.83 x weight

School of Energy Science & Engineering

Can’t use least-squares to find the best fitting
line, instead use the maximum likelihood
• First, project the original data
points onto candidate line
• This gives each sample a
candidate log(odds) value
• Then transform the candidate
log(odds) to candidate probabilities
using this formula

e log(odds)
p=
1+ e log(odds)

School of Energy Science & Engineering

 æ p ö
log ç ÷ = log(odds)
è 1- p ø
An equation that takes probability
as input and outputs log(odds)

e log(odds)
p=
1+ e log(odds)
An equation that takes log(odds)
as input and outputs probability
School of Energy Science & Engineering
An equation that takes probability as input and
outputs log(odds)
æ p ö
log ç ÷ = log(odds)
è 1- p ø
p
=e log(odds)

1- p
p = (1- p) e log(odds)

p = elog(odds) - pelog(odds)
p + pe =e
log(odds) log(odds) e log(odds)
p=
(
p 1+ e log(odds)
=e )
log(odds) 1+ e log(odds)
An equation that takes log(odds)
School of Energy Science & Engineering as input and outputs probability
• Now we use the observed status (obese or not obese) to
calculate their likelihood given the shape of the squiggly line
• Calculate the likelihood of the obese mice, given the shape of
the squiggle
• The likelihood that this mouse is obese, given the shape of the
squiggle, is the same as the predicted probability
• The likelihood that these mice are obese are 0.49, 0.9, 0.91,
091, 0.92

The likelihood for all of the obese mice

is just the product of the individual
likelihoods
= 0.49 x 0.90 x 091 x 0.91 x 0.92

School of Energy Science & Engineering

The probability that these mice are obese are
both 0.01, so the probability and likelihood that
they are not obese is (1 - 0.01)

The likelihood for the mice that are not obese:

(1 - 0.9), (1 – 0.03), (1 – 0.01), (1 – 0.01)
likelihood of data given the squiggle
= 0.49 x 0.9 x 0.91 x 0.91 x 0.92 x
(1 - 0.9) x (1 – 0.03) x (1 – 0.01) x (1 – 0.01)
Log(likelihood of data given the squiggle)
= log(0.49) + log(0.9) + log(0.91) x log(0.91) + log(0.92) +
log(1 - 0.9) + log(1 – 0.03) + log(1 – 0.01) + log(1 – 0.01)

Log(likelihood of data given the squiggle) = -3.77

School of Energy Science & Engineering
The algorithm that finds the line
with the maximum likelihood is
pretty smart – each time it rotates
the line, it does so in a way that
increases the log-likelihood. Thus
the algorithm can find the optimal
fit after a few rotations

School of Energy Science & Engineering

R2 compares a measure of a good fit, SS(fit)
to a measure of a bad fit, SS(mean)

SS(mean) - SS( fit)

R =2

SS(mean)

Size

Weight Weight

School of Energy Science & Engineering

 Like linear regression, we need to find a
measure of a good fit to compare with bad fit

Unfortunately, the residuals for

Logistic Regression are all infinite,
so we can’t use them

School of Energy Science & Engineering

 Project the data onto the best fitting line and
then translate the log(odds) to probabilities

e log(odds)
p=
1+ e log(odds)

School of Energy Science & Engineering

 Lastly, calculate the log-likelihood of the
data given the best fitting squiggle

Log(likelihood of data given the squiggle)

= log(0.49) + log(0.9) + log(0.91) x log(0.91) + log(0.92) +
log(1 - 0.9) + log(1 – 0.03) + log(1 – 0.01) + log(1 – 0.01) = -3.77

We can call this LL(fit), for the log-likelihood of the fitted line,
and use it as substitute for SS(fit)
LL(fit)= -3.77
School of Energy Science & Engineering
 We need a measure of a poorly fitted line
that is analogous to SS(mean)

SS(mean) - SS( fit)

R =
2

SS(mean)

???- LL( fit)

R =
2

???

School of Energy Science & Engineering

 Log(odds of obesity)

æ no of obese mice ö
= log ç ÷
è total no of mice not obese ø

æ5ö
= log ç ÷ = 0.22
è4ø

School of Energy Science & Engineering

Translate the log(odds) back to probabilities

ee log(odds)
0.22
p
p== = 0.56
1+ee
1+
log(odds)
0.22

Weight

School of Energy Science & Engineering

 Calculate the log-likelihood of the data
given the overall probability of obesity

Weight

Log(likelihood of data given overall probability of obesity)

= log(0.55) + log(0.55) + log(0.55) x log(0.55) + log(0.55) +
log(1 – 0.55) + log(1 – 0.55) + log(1 – 0.55) + log(1 – 0.55)
= -6.18
LL(overall probability)= -6.18
School of Energy Science & Engineering
 Weight
LL(overall probability), LL(fit), hopefully
a measure of a bad fit a measure of a good fit
LL(overall probability) - LL( fit)
R =
2

LL(overall probability)
-6.18 - -3.77
R =
2
= 0.39
School of Energy Science & Engineering
-6.18