ECON4150 - Introductory Econometrics

Lecture 15: Binary dependent variables

Monique de Haan
([email protected])

Stock and Watson Chapter 11

Lecture Outline

The linear probability model

Nonlinear probability models
Probit
Logit

Brief introduction of maximum likelihood estimation

Interpretation of coefficients in logit and probit models

Introduction

So far the dependent variable (Y ) has been continuous:

average hourly earnings
traffic fatality rate

What if Y is binary?
Y = get into college, or not; X = parental income.
Y = person smokes, or not; X = cigarette tax rate, income.
Y = mortgage application is accepted, or not; X = race, income,

house characteristics, marital status ...

The linear probability model

Multiple regression model with continuous dependent variable

Yi = 0 + 1 X1i + + k Xki + ui
The coefficient j can be interpreted as the change in Y associated with
a unit change in Xj
We will now discuss the case with a binary dependent variable
We know that the expected value of a binary variable Y is
E [Y ] = 1 Pr (Y = 1) + 0 Pr (Y = 0) = Pr (Y = 1)
In the multiple regression model with a binary dependent variable we
have
E [Yi |X1i , , Xki ] = Pr (Yi = 1|X1i , , Xki )
It is therefore called the linear probability model.

Mortgage applications
Example:
Most individuals who want to buy a house apply for a mortgage at a
bank.
Not all mortgage applications are approved.
What determines whether or not a mortgage application is approved or
denied?
During this lecture we use a subset of the Boston HMDA data
(N = 2380)
a data set on mortgage applications collected by the Federal

Reserve Bank in Boston

Variable

Description

Mean

SD

deny
pi_ratio
black

= 1if mortgage application is denied

anticipated monthly loan payments / monthly income
= 1if applicant is black, = 0 if applicant is white

0.120
0.331
0.142

0.325
0.107
0.350

Mortgage applications
Tuesday February 18 15:46:21 2014

Page 1

Does the payment to income ratio affect whether

or not
a mortgage
___ ____
____
____ ____(R)
/__
/
____/
/
____/
application is denied?
___/
/
/___/
/
/___/
Statistics/Data Analysis
1 . regress deny pi_ratio, robust
Linear regression

deny
pi_ratio
_cons

Number of obs =
F( 1, 2378)
Prob > F
R-squared
Root MSE

Coef.
.6035349
-.0799096

Robust
Std. Err.
.0984826
.0319666

t
6.13
-2.50

P>|t|
0.000
0.012

=
=
=
=

2380
37.56
0.0000
0.0397
.31828

[95% Conf. Interval]

.4104144
-.1425949

.7966555
-.0172243

The estimated OLS coefficient on the payment to income ratio equals

b1 = 0.60.
The estimated coefficient is significantly different from 0 at a 1%
significance level.
How should we interpret b1 ?

The linear probability model

The conditional expectation equals the probability that Yi = 1 conditional
on X1i , , Xki :
E [Yi |X1i , , Xki ] = Pr (Yi = 1|X1i , , Xki ) = 0 + 1 X1i + k Xki
The population coefficient j equals the change in the probability that
Yi = 1 associated with a unit change in Xj .
Pr (Yi = 1|X1i , , Xki )
= j
Xj
In the mortgage application example:
b1 = 0.60
A change in the payment to income ratio by 1 is estimated to increase
the probability that the mortgage application is denied by 0.60.
A change in the payment to income ratio by 0.10 is estimated to increase
the probability that the application is denied by 6% (0.10*0.60*100).

The linear probability model

Assumptions are the same as for general multiple regression model:
1

E(ui |X1i , X2i , . . . , Xki ) = 0

(X1i , . . . , Xki , Yi ) are i.i.d.

Big outliers are unlikely

No perfect multicollinearity.

Advantages of the linear probability model:

Easy to estimate
Coefficient estimates are easy to interpret
Disadvantages of the linear probability model
Predicted probability can be above 1 or below 0!
Error terms are heteroskedastic

The linear probability model: heteroskedasticity

Yi = 0 + 1 X1i + + k Xki + ui
The variance of a Bernoulli random variable (CH 2 S&W):
Var (Y ) = Pr (Y = 1) (1 Pr (Y = 1))
We can use this to find the conditional variance of the error term
Var (ui |X1i , , Xki )

Var (Yi (0 + 1 X1i + k Xki )| X1i , , Xki )

Var (Yi | X1i , , Xki )

Pr (Yi = 1| X1i , , Xki ) (1 Pr (Yi = 1| X1i , , Xki ))

(0 + 1 X1i + + k Xki ) (1 0 1 X1i k Xki )

6=

u2

Solution: Always use heteroskedasticity robust standard errors when

estimating a linear probability model!

10

The linear probability model: shortcomings

In the linear probability model the predicted probability can be below 0 or
above 1!
Example: linear probability model, HMDA data
Mortgage denial v. ratio of debt payments to income
(P/I ratio) in a subset of the HMDA data set (n = 127)

11

Nonlinear probability models

Probabilities cannot be less than 0 or greater than 1

To address this problem we will consider nonlinear probability models
Pr (Yi = 1) = G (Z )
with

Z = 0 + 1 X1i + + k Xki
and

0 G (Z ) 1

We will consider 2 nonlinear functions

1

Probit

Logit

G(Z ) = (Z )
G (Z ) =

1
1 + eZ

12

Probit
Probit regression models the probability that Y = 1
Using the cumulative standard normal distribution function (Z )
evaluated at Z = 0 + 1 X1i + + k Xki
since (z) = Pr (Z z) we have that the predicted probabilities of the
probit model are between 0 and 1

Example
Suppose we have only 1 regressor and Z = 2 + 3X1
We want to know the probability that Y = 1 when X1 = 0.4
z = 2 + 3 0.4 = 0.8
Pr (Y = 1) = Pr (Z 0.8) = (0.8)

13

Probit
Page 791 in the book:

Pr(z
-0.8)
= .2119
Pr (Y = 1) = Pr (Z
0.8)
= (0.8)
= 0.2119

11-15

14

Logit
Logit regression models the probability that Y = 1
Using the cumulative standard logistic distribution function
F (Z ) =

1
1 + eZ

evaluated at Z = 0 + 1 X1i + + k Xki

since F (z) = Pr (Z z) we have that the predicted probabilities of the
probit model are between 0 and 1
Example
Suppose we have only 1 regressor and Z = 2 + 3X1
We want to know the probability that Y = 1 when X1 = 0.4
z = 2 + 3 0.4 = 0.8
Pr (Y = 1) = Pr (Z 0.8) = F (0.8)

15

Logit

.2

.25

Standard logistic density

.05

.1

.15

Area = Pr(Z <= -0.8)

-5

-4

-3

-2

-1

x
Pr (Y = 1) = Pr (Z 0.8) =

1
1+e0.8

= 0.31

16

Logit & probit

Standard Logistic CDF and Standard Normal CDF
1
.9
.8
.7
.6
.5
.4
.3
.2
.1
0
-5

-4

-3

-2

-1

x
logistic

normal

17

How to estimate logit and probit models

In lecture 11 we discussed regression models that are nonlinear in the

independent variables
these models can be estimated by OLS

Logit and Probit models are nonlinear in the coefficients 0 , 1 , , k

these models cant be estimated by OLS

The method used to estimate logit and probit models is Maximum

Likelihood Estimation (MLE).
The MLE are the values of (0 , 1 , , k ) that best describe the full
distribution of the data.

18

Maximum likelihood estimation

The likelihood function is the joint probability distribution of the data,
treated as a function of the unknown coefficients.
The maximum likelihood estimator (MLE) are the values of the
coefficients that maximize the likelihood function.
MLEs are the parameter values most likely to have produced the data.

Lets start with a special case: The MLE with no X

We have n i.i.d. observations Y1 , . . . , Yn on a binary dependent variable
Y is a Bernoulli random variable
There is only 1 unknown parameter to estimate:
The probability p that Y = 1,
which is also the mean of Y

19

Maximum likelihood estimation

Step 1: write down the likelihood function, the joint probability
distribution of the data

Yi is a Bernoulli random variable we therefore have

Pr (Yi = y ) = Pr (Yi = 1)y (1 Pr (Yi = 1))1y = py (1 p)1y
Pr (Yi = 1) = p 1 (1 p)0 = p
Pr (Yi = 0) = p 0 (1 p)1 = 1 p

Y1 , . . . , Yn are i.i.d, the joint probability distribution is therefore the

product of the individual distributions
Pr (Y1 = y1 , . . . .Yn = yn )

Pr (Y1 = y1 ) . . . Pr (Yn = yn )
 y



p 1 (1 p)1y1 . . . pyn (1 p)1yn

p(y1 +y2 +...+yn ) (1 p)n(y1 +y2 +...+yn )

20

Maximum likelihood estimation

We have the likelihood function:
fBernouilli (p; Y1 = y1 , . . . .Yn = yn ) = p

yi

(1 p)n

yi

Step 2: Maximize the likelihood function w.r.t p

Easier to maximize the logarithm of the likelihood function

!
!
n
n
X
X
ln (fBernouilli (p; Y1 = y1 , . . . .Yn = yn )) =
yi ln (p)+ n
yi ln (1 p)
i=1

i=1

Since the logarithm is a strictly increasing function, maximizing the

likelihood or the log likelihood will give the same estimator.

21

Maximum likelihood estimation

Taking the derivative w.r.t p gives
d
ln (fBernouilli (p; Y1 = y1 , . . . .Yn = yn )) =
dp

Setting to zero and rearranging gives

P
(1 p) ni=1 yi
=
Pn
Pn
=
i=1 yi p
i=1 yi
Pn
=
i=1 yi

yi

P
n ni=1 yi
1p

Pn

yi )

npp

Pn

yi

n
1X
yi = Y
n
i=1

i=1

p (n

Solving for p gives the MLE

bMLE =
p

Pn

i=1

np

i=1

22

MLE of the probit model

Step 1: write down the likelihood function

Pr (Y1 = y1 , . . . .Yn = yn )

=
=

Pr (Y1 = y1 ) . . . Pr (Yn = yn )
 y1



p1 (1 p1 )1y1 . . . pnyn (1 pn )1yn

so far it is very similar as the case without explanatory variables except

that pi depends on X1i , . . . , Xki
pi = (X1i , . . . , Xki ) = (0 + 1 X1i + + k Xki )
substituting for pi gives the likelihood function:
h

i
(0 + 1 X11 + + k Xk 1 )y1 (1 (0 + 1 X11 + + k Xk 1 ))1y1 . . .
h
i
(0 + 1 X1n + + k Xkn )yn (1 (0 + 1 X1n + + k Xkn ))1yn

23

MLE of the probit model

Also with obtaining the MLE of the probit model it is easier to take the
logarithm of the likelihood function
Step 2: Maximize the log likelihood function
ln [fprobit (0 , . . . , k ; Y1 , . . . , Yn | X1i , . . . , Xki , i = 1, . . . , n)]
Pn
=
i=1 Yi ln [ (0 + 1 X1i + + k Xki )]
Pn
+ i=1 (1 Yi )ln [1 (0 + 1 X1i + + k Xki )]
w.r.t 0 , . . . , 1
There is no simple formula for the probit MLE, the maximization must be
done using numerical algorithm on a computer.

24

MLE of the logit model

Step 1: write down the likelihood function
Pr (Y1 = y1 , . . . .Yn = yn )




y
p11 (1 p1 )1y1 . . . pnyn (1 pn )1yn

very similar to the Probit model but with a different function for pi
h
i
pi = 1/ 1 + e(0 +1 X1i +...+k Xki )

Step 2: Maximize the log likelihood function w.r.t 0 , . . . , 1

ln [flogit (0 , . . . , k ; Y1 , . . . , Yn | X1i , . . . , Xki , i = 1, . . . , n)]
 h
i
Pn
(0 +1 X1i +...+k Xki )
=
i=1 Yi ln 1/ 1 + e
+

Pn

i=1 (1


 h
i
Yi )ln 1 1/ 1 + e(0 +1 X1i +...+k Xki )

There is no simple formula for the logit MLE, the maximization must be
done using numerical algorithm on a computer.

25

___ ____ ____ ____ ____(R)

/__
/
____/
/
____/
___/
/
/___/
/
/___/
Statistics/Data Analysis

Probit: mortgage applications

1 . probit deny pi_ratio
Iteration
Iteration
Iteration
Iteration

0:
1:
2:
3:

log
log
log
log

likelihood
likelihood
likelihood
likelihood

=
=
=
=

-872.0853
-832.02975
-831.79239
-831.79234

Probit regression

Log likelihood =

deny
pi_ratio
_cons

Number of obs
1)
LR chi2(
Prob > chi2
Pseudo R2

-831.79234

Coef.
2.967907
-2.194159

Std. Err.
.3591054
.12899

z
8.26
-17.01

P>|z|
0.000
0.000

=
=
=
=

2380
80.59
0.0000
0.0462

[95% Conf. Interval]

2.264073
-2.446974

3.67174
-1.941343

The estimated MLE coefficient on the payment to income ratio equals

b1 = 2.97.
The estimated coefficient is positive and significantly different from 0 at
a 1% significance level.
How should we interpret b1 ?

26

Probit: mortgage applications

The estimate of 1 in the probit model CANNOT be interpreted as the change
in the probability that Yi = 1 associated with a unit change in X1 !!
In general the effect on Y of a change in X is the expected change in Y
resulting from the change in X
Since Y is binary the expected change in Y is the change in the
probability that Y = 1
In the probit model the predicted change the probability that the mortgage
application is denied when the payment to income ratio increases from
0.10 to 0.20:
4Pr\
(Yi = 1) = (2.19 + 2.97 0.20) (2.19 + 2.97 0.10) = 0.0495
0.30 to 0.40:
4Pr\
(Yi = 1) = (2.19 + 2.97 0.40) (2.19 + 2.97 0.30) = 0.0619

27

Probit: mortgage applications

Predicted values in the probit model:

The probit
model
satisfies
these
All predicted
probabilities
are between
0 andconditions:
1!
I.

Pr(Y = 1|X) to be increasing in X for 1>0, and

___ ____ ____ ____ ____(R)28

/__
/
____/
/
____/
___/
/
/___/
/
/___/
Statistics/Data Analysis

Logit: mortgage applications

1 . logit deny pi_ratio
Iteration
Iteration
Iteration
Iteration
Iteration

0:
1:
2:
3:
4:

log
log
log
log
log

likelihood
likelihood
likelihood
likelihood
likelihood

=
=
=
=
=

-872.0853
-830.96071
-830.09497
-830.09403
-830.09403

Logistic regression

Log likelihood =

deny
pi_ratio
_cons

Number of obs
1)
LR chi2(
Prob > chi2
Pseudo R2

-830.09403

Coef.
5.884498
-4.028432

Std. Err.
.7336006
.2685763

z
8.02
-15.00

P>|z|
0.000
0.000

=
=
=
=

2380
83.98
0.0000
0.0482

[95% Conf. Interval]

4.446667
-4.554832

7.322328
-3.502032

The estimated MLE coefficient on the payment to income ratio equals

b1 = 5.88.
The estimated coefficient is positive and significantly different from 0 at
a 1% significance level.
How should we interpret b1 ?

29

Logit: mortgage applications

Also in the Logit model:

The estimate of 1 CANNOT be interpreted as the change in the probability
that Yi = 1 associated with a unit change in X1 !!
In the logit model the predicted change the probability that the mortgage
application is denied when the payment to income ratio increases from
0.10 to 0.20:
 


4Pr\
(Yi = 1) = 1/1 + e(4.03+5.880.20) 1/1 + e(4.03+5.880.10) = 0.023
0.30 to 0.40:

 

4Pr\
(Yi = 1) = 1/1 + e(4.03+5.880.40) 1/1 + e(4.03+5.880.30) = 0.063

30

Logit: mortgage applications

The predicted probabilities from the probit and logit
models are very close in these HMDA regressions:

Copyright 2011 Pearson Addison-Wesley. All rights reserved.

11-26

31

Probit & Logit with multiple regressors

We can easily extend the Logit and Probit regression models, by

including additional regressors
Suppose we want to know whether white and black applications are
treated differentially
Is there a significant difference in the probability of denial between black
and white applicants conditional on the payment to income ratio?
To answer this question we need to include two regressors
P/I ratio
Black

32

1 . probit deny black pi_ratio

Probit with multiple regressors

Iteration
Iteration
Iteration
Iteration
Iteration

0:
1:
2:
3:
4:

log
log
log
log
log

likelihood
likelihood
likelihood
likelihood
likelihood

=
=
=
=
=

-872.0853
-800.88504
-797.1478
-797.13604
-797.13604

Probit regression

Log likelihood =

deny
black
pi_ratio
_cons

Number of obs
2)
LR chi2(
Prob > chi2
Pseudo R2

-797.13604

Coef.
.7081579
2.741637
-2.258738

Std. Err.
.0834327
.3595888
.129882

z
8.49
7.62
-17.39

P>|z|
0.000
0.000
0.000

=
=
=
=

2380
149.90
0.0000
0.0859

[95% Conf. Interval]

.5446328
2.036856
-2.513302

.8716831
3.446418
-2.004174

To say something about the size of the impact of race we need to

specify a value for the payment to income ratio
Predicted denial probability for a white application with a P/I-ratio of 0.3
is
(2.26 + 0.71 0 + 2.74 0.3) = 0.0749
Predicted denial probability for a black application with a P/I-ratio of 0.3
is
(2.26 + 0.71 1 + 2.74 0.3) = 0.2327
Difference is 15.8%

1 . logit deny black pi_ratio

Iteration
Iteration
Iteration
Iteration
Iteration

0:
1:
2:
3:
4:

log
log
log
log
log

33

likelihood
likelihood
likelihood
likelihood
likelihood

=
=
=
=
=

-872.0853
-806.3571
-795.72934
-795.69521
-795.69521

Logit with multiple regressors

Logistic regression

Log likelihood =

deny
black
pi_ratio
_cons

Number of obs
2)
LR chi2(
Prob > chi2
Pseudo R2

-795.69521

Coef.
1.272782
5.370362
-4.125558

Std. Err.
.1461983
.7283192
.2684161

z
8.71
7.37
-15.37

P>|z|
0.000
0.000
0.000

=
=
=
=

2380
152.78
0.0000
0.0876

[95% Conf. Interval]

.9862385
3.942883
-4.651644

1.559325
6.797841
-3.599472

To say something about the size of the impact of race we need to

specify a value for the payment to income ratio
Predicted denial probability for a white application with a P/I-ratio of 0.3
is
1/1 + e(4.13+5.370.30) = 0.075
Predicted denial probability for a black application with a P/I-ratio of 0.3
is
1/1 + e(4.13+5.370.30+1.27) = 0.224
Difference is 14.8%

34

LPM, Probit & Logit

Table 1: Mortgage denial regression using the Boston HMDA Data

Dependent variable: deny = 1 if mortgage application is denied, = 0 if accepted
regression model

LPM

Probit

Logit

black

0.177***
(0.025)

0.71***
(0.083)

1.27***
(0.15)

P/I ratio

0.559***
(0.089)

2.74***
(0.44)

5.37***
(0.96)

constant

-0.091***
(0.029)

-2.26***
(0.16)

-4.13***
(0.35)

dierence Pr(deny =1) between black

and white applicant when P/I ratio =0.3

17.7%

15.8%

14.8%

35

Threats to internal and external validity

Both for the Linear Probability as for the Probit & Logit models we have to
consider threats to
1

Internal validity
Is there omitted variable bias?
Is the functional form correct?
Probit model: is assumption of a Normal distribution correct?
Logit model: is assumption of a Logistic distribution correct?
Is there measurement error?
Is there sample selection bias?
is there a problem of simultaneous causality?

External validity
These data are from Boston in 1990-91.
Do you think the results also apply today, where you live?

Sunday March 23 14:37:45 2014

Page 1
36

___ ____ ____ ____ ____(R)

/__
/
____/
/
____/
___/
/
/___/
/
/___/
Statistics/Data Analysis

Distance to college & probability of obtaining a college degree

Linear regression

Number of obs =
F( 1, 3794)
Prob > F
R-squared
Root MSE

Robust
Sunday March 23 14:38:21 2014
Page 1
college
Coef.
Std. Err.
-.012471
.2910057

dist
_cons

.0031403
.0093045

t
-3.97
31.28

Probit regression

Log likelihood =

-2204.8977

-.0407873
-.5464198

-3.73
-19.38

.0109263
.028192

Logistic regression

Log likelihood =

college
dist
_cons

Coef.
-.0709896
-.8801555

[95% Conf. Interval]

P>|z|

Std. Err.
.0193593
.0476434

z
-3.67
-18.47

=
=
=
=

3796
14.48
0.0001
0.0033

[95% Conf. Interval]

___ ____-.0622025
____ ____-.0193721
____(R)
0.000
/__
/ -.6016752
____/
/ -.4911645
____/
0.000
___/
/
/___/
/
/___/
Statistics/Data Analysis
Number of obs
1)
LR chi2(
Prob > chi2
Pseudo R2

-2204.8006

3796
15.77
0.0001
0.0036
.44302

___ ____ ____ ____ ____(R)

0.000
-.0186278
-.0063142
/__
/
____/
/
____/
0.000
.2727633
.3092481
___/
/
/___/
/
/___/
Statistics/Data Analysis
Number of obs
1)
LR chi2(
Prob > chi2
Pseudo R2

Sunday
March 23 14:38:55
Page 1
college
Coef. 2014
Std. Err.
dist
_cons

P>|t|

=
=
=
=

P>|z|
0.000
0.000

=
=
=
=

3796
14.68
0.0001
0.0033

[95% Conf. Interval]

-.1089332
-.9735349

-.033046
-.786776

37

Distance to college & probability of obtaining a college degree

Pr( college degree=1| distance)

.35
.3
.25
.2
.15
.1
.05
0
0

5
10
Distance to college ( x 10 miles)
Linear Probability

Probit

The 3 different models produce very similar results.

15
Logit

38

Summary
If Yi is binary, then E(Yi |Xi ) = Pr (Yi = 1|Xi )
Three models:
1

linear probability model (linear multiple regression)

probit (cumulative standard normal distribution)

logit (cumulative standard logistic distribution)

LPM, probit, logit all produce predicted probabilities

Effect of 4X is a change in conditional probability that Y = 1
For logit and probit, this depends on the initial X
Probit and logit are estimated via maximum likelihood
Coefficients are normally distributed for large n
Large-n hypothesis testing, conf. intervals is as usual