Logistic Regression

Prof. Andy Field

 Aims
• When and Why do we Use Logistic
Regression?
– Binary
– Multinomial
• Theory Behind Logistic Regression
– Assessing the Model
– Assessing predictors
– Things that can go Wrong
• Interpreting Logistic Regression
Slide 2
 When And Why

• To predict an outcome variable that is

categorical from one or more categorical or
continuous predictor variables.
• Used because having a categorical outcome
variable violates the assumption of linearity
in normal regression.

Slide 3
 With One Predictor
P(Y ) =1+e- ( b10 +b1X1i )
• Outcome
– We predict the probability of the outcome
occurring
• b0 and b0
– Can be thought of in much the same way as
multiple regression
– Note the normal regression equation forms part
of the logistic regression equation

Slide 4
 With Several Predictor
P(Y ) =1+e- ( b0 +b1X1i+b12 X2 i +...+bn Xni )
• Outcome
– We still predict the probability of the outcome
occurring
• Differences
– Note the multiple regression equation forms part
of the logistic regression equation
– This part of the equation expands to
accommodate additional predictors

Slide 5
 Assessing the Model: the log-
likelihood statistic
N
log  likelihood   Y ln PY     1  Y  ln 1  PY   
i1
i i i i

• The Log-likelihood statistic

– Analogous to the residual sum of squares in
multiple regression
– It is an indicator of how much unexplained
information there is after the model has been
fitted.
– Large values indicate poorly fitting statistical
models.
 Assessing the Model: the
deviance statistic
• The deviance is very closely related to the log-
likelihood: it’s given by
–Deviance = −2 × log-likelihood
• It’s possible to calculate a log-likelihood or deviance
for different models and to compare these models by
looking at the difference between their deviances.
 Assessing the model: R and R 2

• This R-statistic
– is the partial correlation between the outcome
variable and each of the predictor variables.
– It can vary between −1 and 1.
• The R-statistic is given by:

• Or:
 Assessing Predictors: The Wald
Statistic
b
Wald  SE b

• Similar to t-statistic in Regression.

• Tests the null hypothesis that b = 0.
• Is biased when b is large.
• Better to look at Likelihood-ratio statistics.

Slide 9
 The odds ratio: exp(B)
Odds after a unit change in the predictor
Odds ratio =
Original odds

• Indicates the change in odds resulting from

a unit change in the predictor.
– OR > 1: Predictor , Probability of outcome
occurring .
– OR < 1: Predictor , Probability of outcome
occurring .

Slide 10
 Methods of Regression
• Forced Entry: All variables entered
simultaneously.
• Hierarchical: Variables entered in blocks.
– Blocks should be based on past research, or theory
being tested. Good Method.
• Stepwise: Variables entered on the basis of
statistical criteria (i.e. relative contribution to
predicting outcome).
– Should be used only for exploratory analysis.

Slide 11
 Model building and Parsimony
• When building a model we should strive
for parsimony.
– predictors should not be included unless they
have explanatory benefit.
• First fit the model that includes all
potential predictors, and then
systematically remove any that don’t seem
to contribute to the model.
 Things That Can go Wrong
• Assumptions from Linear Regression:
– Linearity
– Independence of Errors
– Multicollinearity
• Unique Problems
– Incomplete Information
– Complete Separation
– Overdispersion
 Incomplete Information From the
Predictors
• Categorical Predictors:
– Predicting cancer from smoking and eating tomatoes.
– We don’t know what happens when nonsmokers eat tomatoes
because we have no data in this cell of the design.
• Continuous variables
– Will your sample contain a to include an 80 year old, highly
anxious, Buddhist left-handed lesbian?
 Complete Separation
• When the outcome variable can be perfectly
predicted.
– E.g. predicting whether someone is a burglar or your
teenage son or your cat based on weight.
– Weight is a perfect predictor of cat/burglar unless you
have a very fat cat indeed!

1.0 1.0

0.8 0.8
Probability of Outcome

Probability of Outcome

0.6 0.6

0.4 0.4

0.2 0.2

0.0 0.0

20 30 40 50 60 70 80 90 0 20 40 60 80

Weight (KG) Weight (KG)

 Overdispersion
• Overdispersion is where the variance is
larger than expected from the model.
• This can be caused by violating the
assumption of independence.
• This problem makes the standard errors
too small!
 An Example
• Predictors of a treatment intervention.
• Participants
– 113 adults with a medical problem
• Outcome:
– Cured (1) or not cured (0).
• Predictors:
– Intervention: intervention or no treatment.
– Duration: the number of days before treatment that
the patient had the problem.

Slide 17
Output: Initial Model
Output: Initial analysis
The output is split into two blocks: block 0 describes the model before Intervention is
included, and block 1 describes the model after Intervention is included. As such, block 1

Output: Block 0
is the main bit in which we’re interested. The bit of the block 0 output that does come in
useful is in Output 19.3, and will be there only if you selected Iteration history in Figure
19.10. This table tells us the initial 2LL, which is 154.084. We’ll use this value later so
don’t forget it.

OUT

19.6.2. Model summary ➁

With Intervention included in the model a patient is now classified as being cured or not
based on whether they had an intervention or not (waiting list). This can be explained eas-
Output: Model Summary
Output: Model Summary
Model Summary
Classification Plot
 Summary
• The overall fit of the final model is shown by −2LL and its associated
chi-square statistic.
– If the significance of the chi-square statistic is less than .05, then
the model is a significant fit of the data.
• Check the table labelled Variables in the equation to see the regression
parameters for any predictors in the model.
• Look at the Wald statistic and its significance.
• Use the odds ratio, Exp(B), for interpretation.
– OR > 1, then as the predictor increases, the odds of the outcome
occurring increase.
– OR < 1, then as the predictor increases, the odds of the outcome
occurring decrease.
– The confidence interval of Exp(B) should not cross 1.
Reporting the Analysis
 Multinomial logistic regression
• Logistic regression to predict membership of more than two
categories.
• It (basically) works in the same way as binary logistic regression.
• The analysis breaks the outcome variable down into a series of
comparisons between two categories.
– E.g., if you have three outcome categories (A, B and C), then the
analysis will consist of two comparisons that you choose:
• Compare everything against your first category (e.g. A vs. B and A vs. C),
• Or your last category (e.g. A vs. C and B vs. C),
• Or a custom category (e.g. B vs. A and B vs. C).
• The important parts of the analysis and output are much the same
as we have just seen for binary logistic regression
 I may not be Fred Flintstone …
• How successful are chat-up lines?
• The chat-up lines used by 348 men and 672 women in a night-
club were recorded.
• Outcome:
– Whether the chat-up line resulted in one of the following three events:
• The person got no response or the recipient walked away,
• The person obtained the recipient’s phone number,
• The person left the night-club with the recipient.
• Predictors:
– The content of the chat-up lines were rated for:
• Funniness (0 = not funny at all, 10 = the funniest thing that I have ever heard)
• Sexuality (0 = no sexual content at all, 10 = very sexually direct)
• Moral vales (0 = the chat-up line does not reflect good characteristics, 10 = the
chat-up line is very indicative of good characteristics).
– Gender of recipient
Output I
Output II
Output III
 Interpretation I
• Good_Mate: Whether the chat-up line showed signs of good
moral fibre did not significantly predict whether you went
home with the date or got a slap in the face, b = 0.13, Wald
χ2(1) = 2.42, p = .120.
• Funny: Whether the chat-up line was funny significantly
predicted whether you went home with the date or no response,
b = 0.32, Wald χ2(1) = 6.46, p = .011.
• Gender: The gender of the person being chatted up
significantly predicted whether they went home with the person
or gave no response, b = −5.63, Wald χ2(1) = 17.93, p < .001.
• Sex: The sexual content of the chat-up line significantly
predicted whether you went home with the date or got a slap in
the face, b = 0.42, Wald χ2(1) = 11.68, p = .001.
 Interpretation II
• Funny × Gender: The success of funny chat-up
lines depended on whether they were delivered to a
man or a woman because in interaction these
variables predicted whether or not you went home
with the date, b = 1.17, Wald χ2(1) = 34.63, p < .001.
• Sex × Gender: The success of chat-up lines with
sexual content depended on whether they were
delivered to a man or a woman because in interaction
these variables predicted whether or not you went
home with the date, b = −0.48, Wald χ2(1) = 8.51, p =
.004.
Reporting the Results