The document provides an overview of regression analysis using linear probability models, logit models, and probit models for limited dependent variables. It discusses how linear probability models can predict probabilities outside the valid 0-1 range, and introduces logit and probit models as alternatives that guarantee predictions between 0 and 1 through the use of logistic and normal cumulative distribution functions. The document uses examples and formulas to demonstrate how logit models calculate log odds and convert them to probability values.
The document provides an overview of regression analysis using linear probability models, logit models, and probit models for limited dependent variables. It discusses how linear probability models can predict probabilities outside the valid 0-1 range, and introduces logit and probit models as alternatives that guarantee predictions between 0 and 1 through the use of logistic and normal cumulative distribution functions. The document uses examples and formulas to demonstrate how logit models calculate log odds and convert them to probability values.
The document provides an overview of regression analysis using linear probability models, logit models, and probit models for limited dependent variables. It discusses how linear probability models can predict probabilities outside the valid 0-1 range, and introduces logit and probit models as alternatives that guarantee predictions between 0 and 1 through the use of logistic and normal cumulative distribution functions. The document uses examples and formulas to demonstrate how logit models calculate log odds and convert them to probability values.
The document provides an overview of regression analysis using linear probability models, logit models, and probit models for limited dependent variables. It discusses how linear probability models can predict probabilities outside the valid 0-1 range, and introduces logit and probit models as alternatives that guarantee predictions between 0 and 1 through the use of logistic and normal cumulative distribution functions. The document uses examples and formulas to demonstrate how logit models calculate log odds and convert them to probability values.
Lesson Goals ▸ Understand how to estimate qualitative response models.
▸ Advance your knowledge of
statistical inference for limited dependent variables.
Lecture Series
2 12/6/2022
Limited Dependent Variables
Sometimes, the dependent variable may be categorical.
Two categories – binary or dichotomous
Three categories – trichotomous Four or more categories - polychotomous
Binary Response Variable
The dependent (or response) variable is binary. Independent variable(s) may be quantitative, qualitative, or mixed as usual. Examples of binary variables: Yes or No responses (0 - No, 1 - Yes) Patient Satisfaction(0 - Not Satisfied vs. 1- Satisfied) Health Outcome (0 - Not Cured vs. 1- Cured) Votes (0 - Democrat vs. 1 - Republican)
3 12/6/2022
Binary Response Variable
Three approaches to measuring binary response models:
Linear Probability Model (LPM)
Logistic Regression (or Logit) Model
Probit (or Normit) Model
LINEAR PROBABILITY MODEL
4 12/6/2022
Linear Probability Model
LPM is estimated using Ordinary Least Squares (OLS).
Coded values, 0 and 1, of binary response
variables are treated as numbers, not categories.
10
Linear Probability Model
Consider the regression model: 𝒊 𝟎 𝟏 𝒊 where: 𝑿𝒊 = family income, 𝒀𝒊 = 𝟏 if the family owns a house and 𝟎 if it does not own a house. Conditional expectation - 𝑬(𝒀𝒊 | 𝑿𝒊 ) – is the conditional probability that the event will occur given 𝑿𝒊 , that is, 𝑷𝒓(𝒀𝒊 = 𝟏|𝑿𝒊 ). 𝑷𝒓(𝒀𝒊 = 𝟏|𝑿) = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 Hence, Linear Probability Model.
5 12/6/2022
11
Linear Probability Model
If 𝑷𝒊 is probability that 𝒀𝒊 = 𝟏, and (𝟏 − 𝑷𝒊 ) is probability that 𝒀𝒊 = 𝟎, then the variable 𝒀𝒊 will have the probability distribution as: 𝒀𝒊 Probability
Bernoulli 0 𝟏 − 𝑷𝒊 Probability Distribution 1 𝑷𝒊 Total 𝟏
12
Linear Probability Model - Problems
Recall: Rule of Probability ~ 𝒊 𝒊 .
Real problem with OLS estimation of LPM is
nonfulfillment of 𝟎 ≤ 𝑬(𝒀𝒊 |𝑿𝒊 ) ≤ 𝟏. That is, predicted probabilities can be negative, or greater than 1. Heteroscedasticity (can be solved using robust standard errors). Low R-squared value.
6 12/6/2022
13
Now, let’s practice
14
Linear Probability Model
𝐏𝐫(𝒀 > 𝟏)
𝐏𝐫(𝒀 < 𝟎)
7 12/6/2022
15
Linear Probability Model - Problems
Probability Problem (How do we get around this problem?) Estimate the LPM by the usual OLS method. If estimated Y has some values less than 0 [i.e. negative], Y is assumed Constrained LPM to be zero for those cases; if they are greater than 1, they are assumed to be 1.
Devise an estimating technique that will
Logit and Probit Models guarantee that the estimated conditional probabilities Y will lie between 0 and 1.
16
Now, let’s practice
8 12/6/2022
17
MODEL
18
Logistic Regression or Logit Model
Cumulative plot.ecdf(rnorm(500)) distribution functions (CDF) are sigmoid, or S-shaped curves.
Some CDFs are the
logistic CDF (logit) and standard normal CDF (probit).
9 12/6/2022
19
Logit Model match match frequency What can we Win Win 4 do with this Win Lose 6 count data? Win Win Lose Lose What is the probability that a Lose team wins the match? Lose Lose What are the odds of winning? Lose
20
Logit Model Probability is the number of events divided by total outcome.
match frequency 𝟒 𝒑𝒓𝒐𝒃 = = 𝟎. 𝟒𝟎 Win 4 𝟏𝟎 Lose 6
Odds are the chances of something happening to
something not happening. 𝟒 𝒐𝒅𝒅𝒔 = = 𝟎. 𝟔𝟕 𝟔
10 12/6/2022
21
Logit Model Probability and odds can be expressed in terms of each other. 𝒐𝒅𝒅𝒔 𝒑𝒓𝒐𝒃 𝒑𝒓𝒐𝒃 = 𝒐𝒅𝒅𝒔 = 𝟏 + 𝒐𝒅𝒅𝒔 𝟏 − 𝒑𝒓𝒐𝒃
𝑳 = 𝒁𝒊 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 where: 𝑳, is the log of the odds ratio, is not only linear in 𝑿𝒊 but also the parameters. L is called the Logit, hence the name Logit model.
32
Logit Model Key Takeaways: The logits are not bounded by 0 and 1, but the probabilities will (so we need to calculate probabilities for the logit model – MARGINAL EFFECTS). Although L is linear in X, but the probabilities are not. You can include many predictors.
16 12/6/2022
33
Logit Model Key Takeaways: If L, the logit, is positive, it means that when the value of the regressor(s) increases, the odds that the regressand equals 1 (meaning some event of interest happens) increases. If L is negative, the odds that the regressand equals 1 decreases as the value of X increases.
34
MODEL
17 12/6/2022
35
Probit Model The probit model uses a standardized normal CDF. If a variable 𝑿 follows the normal distribution with mean 𝝁 and standard deviation 𝝈, then the PDF is: 𝟏 𝟐 𝟐 𝒇 𝑿 = 𝒆 𝑿 𝝁 /𝟐𝝈 𝟐𝝈𝟐 𝝅
Its CDF becomes:
𝑿𝟎 𝟏 𝑿 𝝁 𝟐 /𝟐𝝈𝟐 𝑭 𝑿 = 𝒆 𝟐𝝈𝟐 𝝅
36
Probit Model The probit model becomes: 𝑷𝒓 𝒀𝒊 = 𝟏 𝑿𝒊 = 𝑭(𝜷𝟎 + 𝜷𝟏 𝑿𝒊 )
Thus, if 𝒁 = 𝜷𝟎 + 𝜷𝟏 𝑿𝒊 , then the CDF is:
𝜷𝟎 𝜷𝟏 𝑿𝒊 𝟏 𝒁𝟐 /𝟐 𝑭 𝑰 = 𝒆 𝒅𝒁 𝟐𝝅
18 12/6/2022
37
Logit and Probit Models
Key Notes: Both models estimate the log odds, but logit follows a logistic CDF whiles probit follows a standard normal CDF. To express coefficients in probabilities, we calculate the marginal effects. Both models have predicted probabilities lying between 0 and 1 inclusive.
38
Logit and Probit Models
Key Notes: Both models are estimated using an iterative algorithm to calculate the log odds of the regression coefficients.
Maximum Likelihood Estimation
19 12/6/2022
39
Logit and Probit Models - R
Logit and Probit estimates in log odds:
stat::glm()
Logit and Probit estimates in probabilities:
mfx:logitmfx() mfx::probitmfx()
40
Logit and Probit Models - R
Goodness of Fit: 𝒅𝒆𝒗𝒊𝒂𝒏𝒄𝒆 𝑷𝒔𝒆𝒖𝒅𝒐 𝑹𝟐 = 𝟏 − 𝒏𝒖𝒍𝒍. 𝒅𝒆𝒗𝒊𝒂𝒏𝒄𝒆 deviance and null.deviance are attributes of both models.
𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒐𝒏𝒔
𝑪𝒐𝒖𝒏𝒕 𝑹𝟐 = 𝑻𝒐𝒕𝒂𝒍 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒐𝒃𝒔𝒆𝒓𝒗𝒂𝒕𝒊𝒐𝒏𝒔
20 12/6/2022
41
Now, let’s practice
42
THANKS! Any questions?
Reach me anytime! Email [email protected] LinkedIn https://www.linkedin.com/in/appiah-elijah-383231123/
