The linear probability model, logit and probit

Let y be 1 if the household owns a motorcycle and 0 if not. Since y takes only the values 0 and 1, E(y | x) is the probability that y = 1, and regressing y on x by OLS gives the linear probability model (LPM): P(y = 1 | x) = x′β. It is easy to estimate and its coefficients read directly: a coefficient of 0.04 on income means an extra ₹1 thousand raises the probability by 4 percentage points. But as a model of a probability it has built-in flaws:

• Predictions outside [0, 1]. A line eventually leaves the band. In the survey, the LPM of ownership on income, town and household size predicts a probability below 0 or above 1 for 93 of the 600 households, as low as −0.3 and as high as 1.06.
• Heteroscedastic errors. y is 0 or 1, so the error can take only two values and its variance is p(1 − p), which depends on x. The usual standard errors are therefore wrong (lesson 9), and the errors cannot be normal.
• A constant effect. A one-unit rise in x changes the probability by the same amount everywhere. Near 0 or 1 that cannot hold: it must run out of room.
• R² is not a good measure of fit. y cannot follow a straight line, so even a perfect model has an R² well below 1.

The LPM is still used, often, for the average effect near the middle of the data, with robust standard errors. But for predicted probabilities and for outcomes that are rare or common, it is replaced by a model that keeps probabilities inside the band.

Keep a linear index z = x′β but pass it through an S-shaped function F that maps any real number into (0, 1): P(y = 1 | x) = F(x′β). Two choices are standard:

Move the marker along the income axis. The red line leaves the band at both ends, while the logit and probit curves bend to stay inside it, and are close to each other everywhere. Notice also how the slope of the S-curve, the effect on the probability of one more rupee of income, changes with income.

Odds and log-odds. For the logit, p ÷ (1 − p) is the odds of the event (p = 0.8 means odds of 4 to 1), and the model says ln[p ÷ (1 − p)] = x′β: the log-odds is linear in x. A coefficient β therefore means: one more unit of x multiplies the odds by e^β (the odds ratio), whatever the starting value. The probit has no such simple reading, and its coefficients are about 1.6 to 1.8 times smaller than the logit’s (a difference of scale between the logistic and normal distributions, not a difference in fit).

OLS minimises squared residuals; it cannot be the right criterion for a 0/1 outcome. Logit and probit are estimated by maximum likelihood: choose the coefficients that make the observed pattern of 0s and 1s most probable. The likelihood multiplies p for each household that owns a motorcycle and 1 − p for each that does not.

Four things follow that the exams ask:

• There is no formula for the estimates: the equations are non-linear, so software iterates. (With a single 0/1 regressor there is one, the log of the ratio of the odds in the two groups: the puzzle.)
• Tests use the likelihood. The overall test that all the slope coefficients are zero is the likelihood-ratio test: LR = 2(ln L − ln L₀), where L₀ is the likelihood of a model with only a constant, chi-square with one degree of freedom per slope. It plays the part of the F test in OLS. Individual coefficients use z (normal) statistics, not t.
• R² has no direct analogue. McFadden’s pseudo R² = 1 − ln L ÷ ln L₀ compares the log-likelihoods. The count R² is the share of observations whose outcome the model predicts correctly with a cutoff of one half. Beware the count R²: a model that predicts “no” for everyone gets the share of 0s, here 0.672.
• Large samples. The standard errors and tests are justified for large n, not exact in small samples.

In the survey the households own a motorcycle when a logistic shock falls below an index we wrote: −5 + 0.3 × income + 0.8 × town − 0.25 × household size. 33% of the 600 households own one. The three models give:

The logit recovers the truth within about one standard error. The probit coefficients are smaller by a factor of about 1.76, as expected, and the LPM coefficients are on yet another scale (probability points per unit). So the coefficients of different models cannot be compared directly; compare marginal effects:

A marginal effect for a logit is β × p(1 − p) at the chosen point (for the probit, β × φ(x′β)), so it is largest where p is near one half and shrinks towards the ends. It is reported either at the means of the regressors or, better, averaged over the sample (the “average marginal effect”). The three models agree: an extra ₹1 thousand of income raises the probability by about 3.6 to 3.8 percentage points. For a dummy such as town, the effect is better computed as the difference between the probability with the dummy at 1 and at 0.

The fit: the logit has a log-likelihood of −229.9 against −379.8 for the constant-only model, so LR = 299.8 with 3 degrees of freedom (p < 0.0001), McFadden’s pseudo R² = 0.395, and the count R² is 0.825 against 0.672 for always predicting “no”.

The probit calculation, the logit formula, the statements about the linear probability model and the test for all the slopes are in the sets above. Explanations of the dummy-variable regressors in these models are in lesson 5.