Qualitative Response Regression Models

 In all the regression models that we have considered so

far, we have implicitly assumed that the regressand, the
dependent variable, or the response variable Y is
quantitative, whereas the explanatory variables are
either quantitative, qualitative (or dummy), or a
mixture.

 In this chapter we consider several models in which the

regressand itself is qualitative in nature.
 Although increasingly used in various areas of social
sciences and medical research, qualitative response
regression models pose interesting estimation and
interpretation challenges.
 Suppose we want to study the labor force participation (LFP)
decision of adult males. Since an adult is either in the labor force
or not, LFP is a yes or no decision. Hence, the response variable,
or regressand, can take only two values, say, 1 if the person is in
the labor force and 0 if he or she is not. In other words, the
regressand is a binary, or dichotomous, variable.

 As another example, consider U.S. presidential elections. Assume

that there are two political parties, Democratic and Republican.
The dependent variable here is vote choice between the two
political parties. Suppose we let Y = 1, if the vote is for a
Democratic candidate, and Y = 0, if the vote is for a Republican
candidate.

 We do not have to restrict our response variable to yes/no or

dichotomous categories only. Returning to our presidential elections
example, suppose there are three parties, Democratic, Republican, and
Independent. The response variable here is trichotomous. In general, we
can have a polychotomous (or multiple-category) response variable.
1. Labor force participation decision

2. Determinants of poverty

3. Factors affecting house ownership or land ownership

4. Willingness to pay and Willingness to accept

5. Factors affecting Customers satisfaction in CBE

6. Subjective wellbeing analysis

 What we plan to do is to first consider the dichotomous regressand
and then consider various extensions of the basic model. But before
we do that, it is important to note a fundamental difference between
a regression model where the regressand Y is quantitative and a
model where it is qualitative.

 In a model where Y is quantitative, our objective is to estimate its

expected, or mean, value given the values of the regressors. In terms
of Chapter 2, what we want is E(Yi | X1i , X2i , . . . , Xki), where the
X’s are regressors, both quantitative and qualitative.

 In models where Y is qualitative, our objective is to find the

probability of something happening, such as voting for a
Democratic candidate, or owning a house, or belonging to a union,
or participating in a sport etc. Hence, qualitative response
regression models are often known as probability models.
In the rest of this chapter, we seek answers to the following
questions:

1. How do we estimate qualitative response regression models? Can

we simply estimate them with the usual OLS procedures?
2. How do we interpret qualitative response regression models?
3. If a regressand is qualitative, how can we measure the goodness of
fit of such models? Is the conventionally computed R2 of any value
in such models?
 We start our study of qualitative response models by first
considering the binary response regression model. There are three
approaches developed to estimate a probability model for a binary
response variable:
1. The linear probability model (LPM)
2. The logit model
3. The probit model
 Because of its comparative simplicity, and because it can be
estimated by OLS, we will first consider the LPM, leaving the
other two models for subsequent sections.
 The Linear Probability Model (Lpm)

 To fix ideas, consider the following regression model:

 where X = family income and Y = 1 if the family owns a house and 0 if it does
not own a house. Model (15.2.1) looks like a typical linear regression model but
because the regressand is binary, or dichotomous, it is called a linear
probability model (LPM).

 This is because the conditional expectation of Yi given Xi , E(Yi | Xi ), can be

interpreted as the conditional probability that the event will occur given Xi ,
that is, Pr (Yi = 1 | Xi). Thus, in our example, E(Yi | Xi) gives the probability of
a family owning a house and whose income is the given amount Xi .
Application of the linear probability model
 Consider the following model
where X = family income and Y = 1 if the family owns a house and 0 if it does not own a
house
Stata command : reg y x

. reg house income

Source SS df MS Number of obs = 40

F(1, 38) = 156.63
Model 8.0274948 1 8.0274948 Prob > F = 0.0000
Residual 1.9475052 38 .051250137 R-squared = 0.8048
Adj R-squared = 0.7996
Total 9.975 39 .255769231 Root MSE = .22638

house Coef. Std. Err. t P>|t| [95% Conf. Interval]

income .102131 .0081605 12.52 0.000 .085611 .118651

_cons -.9456861 .1228415 -7.70 0.000 -1.194366 -.6970066
• The justification of the name LPM for models like (15.2.1) can be seen as
follows: Assuming E(ui) = 0, as usual (to obtain unbiased estimators), we
Obtain

 Now, if Pi = probability that Yi = 1 (that is, the event occurs), and (1 − Pi) =
probability that Yi = 0 (that is, that the event does not occur), the variable Yi
has the following (probability) distribution.
 that is, the conditional expectation of the model (15.2.1) can, in
fact, be interpreted as the conditional probability of Yi . In
general, the expectation of a Bernoulli random variable is the
probability that the random variable equals 1.
 In passing note that if there are n independent trials, each with a
probability p of success and probability (1 − p) of failure, and X of
these trials represent the number of successes, then X is said to
follow the binomial distribution. The mean of the binomial
distribution is np and its variance is np(1 − p). The term success is
defined in the context of the problem.
that is, the conditional expectation (or conditional probability) must lie
between 0 and 1. From the preceding discussion it would seem that
OLS can be easily extended to binary dependent variable regression
models So, perhaps there is nothing new here. Unfortunately, this is
not the case, for the LPM poses several problems, which are as follows:

1. Non-Normality of the Disturbances ui

 But the assumption of normality for ui is not tenable for the

LPMs because, like Yi, the disturbances ui also take only two
values: Obviously, ui cannot be assumed to be normally
distributed; they follow the Bernoulli distribution.
 2. Heteroscedastic Variances of the Disturbances

 Even if E(ui) = 0 and cov (ui , uj ) = 0 for i = j (i.e., no serial

correlation), it can no longer be maintained that in the LPM the
disturbances are homoscedastic.
 This is, however, not surprising. As statistical theory shows, or a
Bernoulli distribution the theoretical mean and variance are,
respectively, p and p(1 − p), where p is the probability of success
(i.e., something happening), showing that the variance is a
function of the mean. Hence the error variance is heteroscedastic.
3. Nonfulfillment of 0 ≤ E(Yi | X) ≤ 1

 Since E(Yi | X) in the linear probability models measures the

conditional probability of the event Y occurring given X, it
must necessarily lie between 0 and 1.
 Although this is true a priori, there is no guarantee that ˆYi ,
the estimators of E(Yi | Xi ), will necessarily fulfill this
restriction, and this is the real problem with the OLS estimation
of the LPM.
 There are two ways of finding out whether the estimated ˆYi lie
between 0 and 1. One is to estimate the LPM by the usual OLS
method and find out whether the estimated ˆYi lie between 0
 If some are less than 0 (that is, negative), ˆYi is assumed to be zero
for those cases; if they are greater than 1, they are assumed to be
1.
 The second procedure is to devise an estimating technique that
will guarantee that the estimated conditional probabilities ˆYi will
lie between 0 and 1.
 The logit and probit models discussed later will guarantee that the
estimated probabilities will indeed lie between the logical limits 0
and 1.
 
4. Questionable Value of R2 as a Measure of Goodness of
Fit
The conventional coefficient of determination is of limited
value to judge the goodness of fit of the model.
5. Functional Form
E()= + is linear: LPM assumes that E() increases linearly
with (Marginal effect of is constant), which may be
unrealistic.
E() is non-linearly related to in practice.
 ALTERNATIVES TO LPM
 As we have seen, the LPM is plagued by several
problems, such as
 Non normality of ui ,
 Heteroscedasticity of ui ,
 Possibility of ˆYi lying outside the 0–1 range, and
 There is generally less valuable R2 values.
 But even then the fundamental problem with the LPM is it
assumes that Pi = E(Y = 1 | X) increases linearly with X, that is,
the marginal or incremental effect of X remains constant

throughout.
 Thus, in our home ownership example we found that as X
increases by a unit ($1000), the probability of owning a house
increases by the same constant amount of 0.10. This is so whether
the income level is $8000, $10,000, $18,000, or $22,000. This seems
patently unrealistic. In reality one would expect that Pi is
nonlinearly related to Xi :
 At very low income a family will not own a house but at a
sufficiently high level of income, say, X*, it most likely will own a
house. Any increase in income beyond X* will have little effect on
the probability of owning a house. Thus, at both ends of the
income distribution, the probability of owning a house will be
virtually unaffected by a small increase in X.
 Therefore, what we need is a (probability) model that has these two
features: (1) As Xi increases, Pi = E(Y = 1 | X) increases but never
steps outside the 0–1 interval, and (2) the relationship between Pi and
Xi is nonlinear, that is, “one which approaches zero at slower and
slower rates as Xi gets small and approaches one at slower and
slower rates as Xi gets very large.’’ Geometrically, the model we want
would look something like Figure 15.2. Notice in this model that the
probability lies between 0 and 1 and that it varies nonlinearly with X.
 The reader will realize that the sigmoid, or S-shaped, curve in the
figure very much resembles the cumulative distribution function
(CDF) of a random variable.
 Therefore, one can easily use the CDF to model regressions where
the response variable is dichotomous, taking 0–1 values. The
practical question now is, which CDF? For although all CDFs are
S shaped, for each random variable there is a unique CDF.
 For historical as well as practical reasons, the CDFs commonly
chosen to represent the 0–1 response models are
 The logistic CDF: giving rise to the logit model
 The normal CDF: giving rise to the probit (or normit) model.
 
Alternative Models to LPM:
( Logit and Probit Models)
They are probability models with two characteristics.
1. ) 1

 As increases increases but never steps

outside the 0-1 interval.

2. Non-linear relationship between and .

 The cumulative distribution function (CDF) can be used to model the such
qualitative response models. CDF is the probability of a random variable when
it takes a value less than or equal to some specified numerical value: F(Y)
=F(Y= ) = P(Y ).
 THE LOGIT MODEL
 We will continue with our home ownership example to explain the
basic ideas underlying the logit model. Recall that in explaining
home ownership in relation to income, the LPM was
 Equation (15.5.3) represents what is known as the (cumulative)
logistic distribution function. It is easy to verify that as Zi ranges
from −∞ to +∞, Pi ranges between 0 and 1 and that Pi is
nonlinearly related to Zi (i.e., Xi), thus satisfying the two
requirements considered earlier.

 But it seems that in satisfying these requirements, we have created

an estimation problem because Pi is nonlinear not only in X but
also in the β’s as can be seen clearly from (15.5.2). This means that
we cannot use the familiar OLS procedure to estimate the
parameters.
 Now Pi/(1 − Pi) is simply the odds ratio in favor of owning a house
—the ratio of the probability that a family will own a house to the
probability that it will not own a house.
 
Estimation of the Logit model
If a family owns a house, = .
If a family does not own a house, = .

In this situation we may have to resort to the maximum likelihood

(ML) method to estimate the parameters since the standard OLS is
not applicable. In ML method, the objective is to obtain the values of
the unknown parameters in such a manner that the probability of
observing the given s is as high as possible
 
Interpretation of the Logit model
 Test is based on standard normal (Z) statistic because it uses
asymptotic standard errors.
 Overall significance of the model is based on likelihood ratio test
(LR statistic):
 Pseudo is a measure of goodness of fit: Pseudo-R-squared:  Many
different measures of psuedo-R-squared exist. They all attempt to
provide information similar to that provided by R-squared in OLS
regression; however, none of them can be interpreted exactly as R-
squared in OLS regression is interpreted
Application of logit

Example1: Consider the previous house ownership example

. logit house
house income, nolog

Logistic regression Number of obs = 40

LR chi2(1) = 22.36
Prob > chi2 = 0.0000
Log likelihood = -15.742775 Pseudo R2 = 0.4152

house Coef. Std.

Std. Err. z P>|z| [95% Conf. Interval]

income .5123856 .157076

.157076 3.26
3.26 0.001 .2045224 .8202489
_cons -8.287357 2.583799 -3.21 0.001 -13.35151 -3.223205
 Logit=house ownership=-8.28+0.51income

The income coefficient of 0.51 means, that if income increases by a

unit, the log of odds in favor of owning house will increase by 0.51
units, suggesting a positive relationship between income & house
ownership.
The intercept: when income is zero the log of odds in favor of owning
house is -8.28.

Example 2
Consider the logit2 data, where =grade = , =gpa , = score on an examination given
at the beginning of the term(diagnostic exam) and = methodology = .
Fit the following model:= =+++ +
. logit grade gpa score methodology, nolog

Logistic regression Number of obs = 32

LR chi2(3) = 15.40
Prob > chi2 = 0.0015
Log likelihood = -12.889633 Pseudo R2 = 0.3740

grade Coef. Std. Err. z P>|z| [95% Conf. Interval]

gpa 2.826113 1.262941 2.24 0.025 .3507938 5.301432

score .0951577 .1415542 0.67 0.501 -.1822835 .3725988
methodology 2.378688 1.064564 2.23 0.025 .29218 4.465195
_cons -13.02135 4.931325 -2.64 0.008 -22.68657 -3.35613
 The likelihood ratio chi-square of 15.40 with a p-value of 0.0015 tells us that our model
as a whole fits significantly better than an empty model (i.e., a model with no predictors).

 In the table we see the coefficients, their standard errors, the z-statistic, associated p-
values, and the 95% confidence interval of the coefficients.  Both gpa and methodology
 are statistically significant, while score is statistically insignificant

 The logistic regression coefficients give the change in the log odds of the outcome for a
one unit increase in the predictor variable.
 For every one unit change in gpa, the log odds in favor of scoring A increases by
2.826
 For a one unit increase in score, the log odds in favor of scoring A grade increases
by 0.095. however this variable is statistically insignificant.
 The indicator variables for methodology have a slightly different interpretation. If
new teaching method is used (versus the old teaching method), the log odds in favor
of scoring A increases by 2.37.
. mfx

Marginal effects after logit

y = Pr(grade) (predict)
= .25282025

variable dy/dx Std. Err. z P>|z| [ 95% C.I. ] X

gpa .5338589 .23704 2.25 0.024 .069273 .998445 3.11719

score .0179755 .02624 0.69 0.493 -.033448 .069399 21.9375
method~y* .4564984 .18105 2.52 0.012 .10164 .811357 .4375

(*) dy/dx is for discrete change of dummy variable from 0 to 1

 Example three
 A researcher is interested in how variables, such as GRE (Graduate Record Exam
scores), GPA (grade point average) and prestige of the undergraduate institution,
effect admission into graduate school. The response variable, admit/don’t admit,

is a binary variable.
 This data set has a binary response (outcome, dependent) variable called admit.
There are three predictor variables:  gre, gpa and rank. We will treat the
variables gre and gpa as continuous. The variable rank takes on the values 1
through 4. Institutions with a rank of 1 have the highest prestige, while those with
a rank of 4 have the lowest.

 Stata result
. logit admit gpa gre i.rank, nolog

Logistic regression Number of obs = 400

LR chi2(5) = 41.46
Prob > chi2 = 0.0000
Log likelihood = -229.25875 Pseudo R2 = 0.0829

admit Coef. Std. Err. z P>|z| [95% Conf. Interval]

gpa .8040377 .3318193 2.42 0.015 .1536838 1.454392

gre .0022644 .001094 2.07 0.038 .0001202 .0044086

rank
2 -.6754429 .3164897 -2.13 0.033 -1.295751 -.0551346
3 -1.340204 .3453064 -3.88 0.000 -2.016992 -.6634158
4 -1.551464 .4178316 -3.71 0.000 -2.370399 -.7325287

_cons -3.989979 1.139951 -3.50 0.000 -6.224242 -1.755717

. mfx

Marginal effects after regress

y = Fitted values (predict)
= .3175

variable dy/dx Std. Err. z P>|z| [ 95% C.I. ] X

gre .0005489 .00021 2.56 0.010 .000129 .000968 587.7

gpa .1542363 .06498 2.37 0.018 .026884 .281589 3.3899
. logit admit diagonstic quality gpa

Iteration 0: log likelihood = -249.98826

Iteration 1: log likelihood = -239.17277
Iteration 2: log likelihood = -239.06484
Iteration 3: log likelihood = -239.06481
Iteration 4: log likelihood = -239.06481

Logistic regression Number of obs = 400

LR chi2(3) = 21.85
Prob > chi2 = 0.0001
Log likelihood = -239.06481 Pseudo R2 = 0.0437

admit Coef. Std. Err. z P>|z| [95% Conf. Interval]

diagonstic .0024768 .0010702 2.31 0.021 .0003792 .0045744

quality .4372236 .2918532 1.50 0.134 -.1347983 1.009245
gpa .6675556 .3252593 2.05 0.040 .0300591 1.305052
_cons -4.600814 1.09638 -4.20 0.000 -6.749678 -2.451949

. mfx

Marginal effects after logit

y = Pr(admit) (predict)
= .3076378

variable dy/dx Std. Err. z P>|z| [ 95% C.I. ] X

diagon~c .0005276 .00023 2.33 0.020 .000083 .000972 587.7

quality* .0978326 .068 1.44 0.150 -.03545 .231115 .1625
gpa .1421872 .069 2.06 0.039 .006948 .277427 3.3899

(*) dy/dx is for discrete change of dummy variable from 0 to 1

 The likelihood ratio chi-square of 41.46 with a p-value of 0.0001 tells us that our model
as a whole fits significantly better than an empty model (i.e., a model with no predictors).

 In the table we see the coefficients, their standard errors, the z-statistic, associated p-
values, and the 95% confidence interval of the coefficients.  Both gre and gpa are
statistically significant, as are the three indicator variables for rank.

 The logistic regression coefficients give the change in the log odds of the outcome for a
one unit increase in the predictor variable.
 For every one unit change in gre, the log odds of admission (versus non-admission)
increases by 0.002.
 For a one unit increase in gpa, the log odds of being admitted to graduate school
increases by 0.804.
 The indicator variables for rank have a slightly different interpretation. For
example, having attended an undergraduate institution with rank of 2, versus an
institution with a rank of 1, decreases the log odds of admission by 0.675.
 Probit model
 As we have noted, to explain the behavior of a dichotomous dependent variable we
will have to use a suitably chosen CDF. The logit model uses the cumulative logistic
function, as shown in (15.5.2).
 But this is not the only CDF that one can use. In some applications, the normal CDF
has been found useful. The estimating model that emerges from the normal CDF is
popularly known as the probit model, although sometimes it is also known as the
normit model. In principle one could substitute the normal CDF in place of the
logistic CDF .
 In statistics, a probit model is a type of regression where the dependent variable can
take only two values, for example married or not married. The purpose of the model
is to estimate the probability that an observation with particular characteristics will
fall into a specific one of the categories; moreover, classifying observations based on
their predicted probabilities is a type of binary classification model.
 Example.

.. probit
probit admit
admit gpa
gpa gre
gre i.rank,
i.rank, nolog
nolog

Probit
Probit regression
regression Number
Number ofof obs
obs == 400
400
LR
LR chi2(5)
chi2(5) == 41.56
41.56
Prob
Prob >> chi2
chi2 == 0.0000
0.0000
Log
Log likelihood
likelihood == -229.20658
-229.20658 Pseudo R2
Pseudo R2 == 0.0831
0.0831

admit
admit Coef.
Coef. Std.
Std. Err.
Err. zz P>|z|
P>|z| [95%
[95% Conf.
Conf. Interval]
Interval]

gpa
gpa .4777302
.4777302 .1954625
.1954625 2.44
2.44 0.015
0.015 .0946308
.0946308 .8608297
.8608297
gre
gre .0013756
.0013756 .0006489
.0006489 2.12
2.12 0.034
0.034 .0001038
.0001038 .0026473
.0026473

rank
rank
22 -.4153992
-.4153992 .1953769
.1953769 -2.13
-2.13 0.033
0.033 -.7983308
-.7983308 -.0324675
-.0324675
33 -.812138
-.812138 .2085956
.2085956 -3.89
-3.89 0.000
0.000 -1.220978
-1.220978 -.4032981
-.4032981
44 -.935899
-.935899 .2456339
.2456339 -3.81
-3.81 0.000
0.000 -1.417333
-1.417333 -.4544654
-.4544654

_cons
_cons -2.386838
-2.386838 .6740879
.6740879 -3.54
-3.54 0.000
0.000 -3.708026
-3.708026 -1.065649
-1.065649
 Example.
 The likelihood ratio chi-square of 41.56 with a p-value of 0.0001 tells us that our
model as a whole is statistically significant, that is, it fits significantly better than a
model with no predictors.
 In the table we see the coefficients, their standard errors, the z-statistic, associated p-
values, and the 95% confidence interval of the coefficients. Both gre, gpa, and the
three indicator variables for rank are statistically significant. The probit regression
coefficients give the change in the z-score or probit index for a one unit change in the
predictor.

 For a one unit increase in gre, the z-score increases by 0.001.

 For each one unit increase in gpa, the z-score increases by 0.478.
 The indicator variables for rank have a slightly different interpretation. For example,
having attended an undergraduate institution of rank of 2, versus an institution with
a rank of 1 (the reference group), decreases the z-score by 0.415.
 
Logit vs Probit Model
Both models give qualitatively similar results.

The estimates of parameters of the two models are not directly

comparable.
Both give statistically sound results as ) 1

Both models show non-linear relationship b/n &.

Both modes arise from the cumulative distribution function

(CDF), having S-shaped curves. Both models produce
asymptotically consistent, efficient and normal estimates.
 
Both logit and probit models are estimated using maximum
likelihood (ML) method as a standard OLS is not applicable.
The objective of ML method is to obtain the values of unknown
parameters in such a manner that the probability of observing a
given Y is as high as possible.
Measure of fit:
the overall significance of both logit and probit models is based on
likelihood ratio test (LR statistic).
Pseudo is a measure of goodness of fit in both models.
 
Differences
Logit model arises from Logistic CDF while Probit Model from
Normal CDF
The logit model has slightly fatter tails: the conditional
probability approaches zero or one at a slower rate in Logit than
in Probit.
The probit curve approaches more quickly than the logistic
curve.
Logit model is relatively simple to analyze and interpret
compared to probit model.
Exercise

Considere the following model

Y=B1+B2X2+B3X3+ B4X4 + Ui
Dependent variable is poverty = {1, if non-poor
{0, otherwise

Explanatory variables { land size , age , and annual income}

The above model will be estimated by LPM, logit and probit

models. Having the results of each answere the preciding
questions.
reg poverty landsize age income

Source SS df MS Number of obs = 400

F(3, 396) = 7.51
Model 4.66401014 3 1.55467005 Prob > F = 0.0001
Residual 82.0134899 396 .207104772 R-squared = 0.0538
Adj R-squared = 0.0466
Total 86.6775 399 .217236842 Root MSE = .45509

poverty Coef. Std. Err. t P>|t| [95% Conf. Interval]

landsize .0005006 .0002157 2.32 0.021 .0000765 .0009246

age .1040503 .0642078 1.62 0.106 -.0221806 .2302811
income .1353018 .0658885 2.05 0.041 .0057668 .2648368
_cons -.4522542 .2134725 -2.12 0.035 -.8719353 -.0325732
Based on the result of LPM nswere the following
Questions

A.Interpret the coefficients (parameters) of the

model
B.Comment on the significance of each variables
C.Comment on the overall significance of the
model
D.Comment on the goodness of fit
Logistic regression Number of obs = 400
LR chi2(3) = 21.85
Prob > chi2 = 0.0001
Log likelihood = -239.06481 Pseudo R2 = 0.0437

poverty Coef. Std. Err. z P>|z| [95% Conf. Interval]

landsize .0024768 .0010702 2.31 0.021 .0003792 .0045744

age .4372236 .2918532 1.50 0.134 -.1347983 1.009245
income .6675556 .3252593 2.05 0.040 .0300591 1.305052
_cons -4.600814 1.09638 -4.20 0.000 -6.749678 -2.451949
Based on the logistic regression modelAnswere the following
Questions

A.Interpret the coefficients (parameters) of the model

B.Comment on the significance of each variables

C.Comment on the overall significance of the model

D.Comment on the goodness of fit

. probit poverty landsize age income

Iteration 0: log likelihood = -249.98826

Iteration 1: log likelihood = -238.97735
Iteration 2: log likelihood = -238.94339
Iteration 3: log likelihood = -238.94339

Probit regression Number of obs = 400

LR chi2(3) = 22.09
Prob > chi2 = 0.0001
Log likelihood = -238.94339 Pseudo R2 = 0.0442

poverty Coef. Std. Err. z P>|z| [95% Conf. Interval]

landsize .0015244 .0006382 2.39 0.017 .0002736 .0027752

age .2730334 .1795984 1.52 0.128 -.078973 .6250398
income .4009853 .1931077 2.08 0.038 .0225012 .7794694
_cons -2.797884 .6475363 -4.32 0.000 -4.067032 -1.528736
Based on the probit regression modelAnswere the following
Questions

A.Interpret the coefficients (parameters) of the model

B.Comment on the significance of each variables

C.Comment on the overall significance of the model

D.Comment on the goodness of fit

E.Compare the results of logit and probit Model