1

Classification Methods

Chapter 04 (part 01)

 2
`

Logistic Regression
Outline

3
The Default Dataset

4
Why not Linear Regression?

5
Solution: Use Logistic Function

 Instead of trying to predict Y, let’s try to predict P(Y = 1).

1
0.9
0.8
0.7
Probability

0.6
0.5
0.4
0.3
0.2
0.1
0
-5 -4 -3 -2 -1 0 1 2 3 4 5
X

6
Generalized Linear Model

 The logistic function can be rebalanced to yield

is known as the odds and takes on a value between 0 and infinity.

 Taking a logarithm of both sides of the logistic odds equation yields an

equation for the log-odds or logit,

Logistic regression has a logit that is linear in terms of X.

7
Logistic Function on Default Data

 Now the probability of default is close to, but not less than zero for low
balances. And close to but not above 1 for high balances

8
Review of Bernoulli Trials

9
Interpreting the logistic regression coefficients

Odds ratio

• Adding 1 to x has the effect of adding to both sides of the

equation
• Exponentiating both sides, the odds are then multiplied by
• If , then a unit difference in corresponds to a multiplicative
change of in the odds
• Changing the odds from 1 to 1.22
• Changing p from 0.5 to 0.55

10
Interpreting the logistic regression coefficients

Intercept

 The intercept can only be interpreted assuming zero values for the
predictors

 Or, we can evaluate at the mean of the input variables

11
Are the coefficients significant?

 We still want to perform a hypothesis test to see whether we can be sure that
are 0 and 1 significantly different from zero.

 We use a Z test instead of a T test,

12
Example

 It doesn’t change the way we interpret the p-value.

 Here the p-value for balance is very small, and b1 is positive, so we are sure
that if the balance increase, then the probability of default will increase as well.

 The column labelled “Z-statistic” is the Wald test statistic. What conclusions
can we make here?

13
Making Prediction
 Suppose an individual has an average balance of $1000. What is their
probability of default?

 The predicted probability of default for an individual with a balance of $1000

is less than 1%.

 For a balance of $2000, the probability is (??)

14
Qualitative Predictors in Logistic Regression

 We can use a qualitative variable “Student” coded as

 (Student = 1, Non-student =0).

 b1 is positive: This indicates students tend to have higher default probabilities

than non-students
 It increases the odds of the probability by

15
Multiple Logistic Regression

 We can fit multiple logistic just like regular regression

 The log-odds equation for multiple logistic regression can be express as

 Maximum likelihood is also used to estimate in the case of multiple logistic

regression.

16
Multiple Logistic Regression- Default Data

 Predict Default using:

 Balance (quantitative)

 Income (quantitative)

 Student (qualitative)

17
An Apparent Contradiction!

18
Students (Orange) vs. Non-students (Blue)

• Students tend to have higher balances than non-students, so

their marginal default rate is higher than for non-students.
• But for each level of balance, students default less than non-
students.

19
To whom should credit be offered?

 A student is risker than non students if no information about the credit card
balance is available. However, that student is less risky than a non student
with the same credit card balance!

 For example, a student with a credit card balance of $1, 500 and an income of
$40, 000 has an estimated probability of default of

 A non-student with the same balance and income has an estimated probability
of default of

20
Confounding

 As in the linear regression, the results obtained using one predictor

may be quite different from the those obtained using multiple
predictors.

 Especially when there is correlation among the predictors

 The phenomenon is known as confounding.

 Confounder is the third variable that (but not our interest) distorts the
observed relationship between the independent variable and outcome.

 Collinearity is an extreme form of confounding, where variables

become ‘non-identifiable’.

21
Logistic regression with more than two classes

 Select a single multinomial logistic regression class to serve as the baseline;

without loss of generality, we select the Kth class for this role.

22
Softmax Coding

 In the softmax coding, rather than selecting a baseline class, we treat all K classes
symmetrically, and assume that for k = 1,…,K,

The log odds ratio between the kth and k′th classes equals

23