Econometrics for economics students

Each coefficient in a multiple regression is the change in the expected value of the dependent variable for a one-unit rise in that regressor, with every other regressor in the model unchanged. Nothing is literally held fixed in survey data. The regression achieves the same thing by first removing the influence of the other regressors from both the dependent variable and the regressor of interest, and then taking the ordinary slope of what is left (the Frisch-Waugh-Lovell result).

Simple regression has one explanatory variable, so its slope absorbs the effect of everything correlated with it. Multiple regression has several, and each coefficient is the effect of that variable with the others held fixed. The two slopes on the same variable differ whenever the added variables are related both to it and to the dependent variable.

Six are usually listed: the model is linear in its parameters; the regressors are not perfectly collinear; the error has a zero mean given the regressors; the error has constant variance (homoscedasticity); errors are uncorrelated across observations; and, for exact small-sample tests, the errors are normal. The first three give unbiased estimates, the first five give the standard variance formula and the Gauss-Markov theorem, and normality gives exact t and F distributions.

Under the classical assumptions other than normality, the ordinary least squares estimator has the smallest variance among all estimators that are linear in the dependent variable and unbiased. BLUE stands for best linear unbiased estimator. It does not require normal errors, and it does not claim that OLS beats every possible estimator, only the linear unbiased ones.

Say how much the dependent variable is expected to change for a one-unit rise in the regressor, holding the other regressors fixed, in the units of both variables. For a dummy variable, the coefficient is the difference between the group coded 1 and the group coded 0 at the same values of the other variables. The intercept is the predicted value when every regressor is zero, which is often outside the data.

The t statistic for a coefficient is the coefficient divided by its standard error, with n − k − 1 degrees of freedom for k slopes. The overall F is (R² ÷ k) ÷ ((1 − R²) ÷ (n − k − 1)), equivalently the explained sum of squares over k divided by the residual sum of squares over n − k − 1. For q linear restrictions, F is the rise in the residual sum of squares from imposing them, divided by q, over the unrestricted residual mean square.

Least squares can always set a new coefficient to zero and recover the old fit, so the residual sum of squares cannot rise and R² cannot fall. Adjusted R², 1 − (1 − R²)(n − 1) ÷ (n − k − 1), penalises each extra parameter, and it can be negative. It rises when a variable is added exactly when that variable's t statistic exceeds 1 in absolute value.

The unrestricted regression estimates every coefficient freely. The restricted regression imposes the null hypothesis, for example by dropping variables whose coefficients are hypothesised to be zero. Imposing a restriction can only make the fit worse, and the F test asks whether the rise in the sum of squared residuals is larger than chance would produce.

If a variable has m categories and the regression has a constant, including a dummy for every category makes the dummies add up to the constant column, so the regressors are perfectly collinear and X′X cannot be inverted. Avoid it by dropping one dummy (that category becomes the base group, and each coefficient is a gap from it) or by dropping the constant (each coefficient is then that group's average).

In a log-log regression the slope is the elasticity: a 1% rise in x goes with a b% change in y. In a log-level regression (ln y on x), one more unit of x changes y by about 100b%. In a level-log regression (y on ln x), a 1% rise in x changes y by about b ÷ 100 units. In a level-level regression one more unit of x changes y by b units.

If y depends on x1 and x2 and x2 is left out, the slope on x1 estimates β1 plus β2 times δ, where β2 is the effect of the omitted variable on y and δ is the slope of the regression of x2 on x1. The sign of the bias is the product of the signs of β2 and δ. There is no bias if β2 is zero or if the omitted variable is unrelated to x1.

Random error in an explanatory variable biases its coefficient towards zero (attenuation) by the reliability ratio, the share of the recorded variable's variance that is true signal, and the bias does not disappear in large samples. Random error in the dependent variable does not bias the coefficients, it only makes them less precise.

Multicollinearity is high correlation among the regressors. With perfect collinearity the regression cannot be estimated. With high but not perfect collinearity the estimates are still unbiased and BLUE, but their variances are inflated by the variance inflation factor, 1 ÷ (1 − R²) from regressing each regressor on the others, so standard errors are large, t ratios small and coefficients unstable, even though R² can be high.

With heteroscedasticity the OLS estimators remain unbiased and consistent, but they are no longer efficient and the usual standard errors are wrong, so t and F tests are unreliable. It is detected with residual plots and the Park, Glejser, Goldfeld-Quandt, Breusch-Pagan and White tests, and corrected with weighted least squares (GLS) if the form of the variance is known or with robust standard errors.

The Durbin-Watson statistic, the sum of squared differences of successive residuals over the sum of squared residuals, is approximately 2(1 − ρ̂), so it lies between 0 and 4. Values near 2 suggest no first-order autocorrelation, values below the lower critical value dL suggest positive autocorrelation, and values above 4 − dL suggest negative autocorrelation. Between dL and dU, or between 4 − dU and 4 − dL, the test is inconclusive.

An instrumental variable z helps estimate the effect of a regressor x that is correlated with the error term. It must be relevant (correlated with x, which can be tested in the first stage) and exogenous (uncorrelated with the error, so that it affects y only through x, which can only be argued). The IV estimator is Cov(z, y) ÷ Cov(z, x); with a 0/1 instrument it is the difference in the mean of y between the two z groups divided by the difference in the mean of x.

In a simultaneous equations system, the order condition says an equation can be identified only if the number of exogenous variables excluded from it is at least the number of endogenous variables it includes, less one (K − k ≥ m − 1). It is necessary but not sufficient. The rank condition, which is necessary and sufficient, requires that the coefficients of the excluded variables in the other equations form a matrix of rank equal to the number of equations less one.

Both models allow each individual a persistent effect. Fixed effects treats the effects as parameters, removed by subtracting individual means or differencing, and is consistent even when the effects are correlated with the regressors. Random effects treats them as random draws assumed uncorrelated with the regressors; it is more efficient when that holds but biased when it does not. The Hausman test compares the two, and rejection favours fixed effects.

All three model a 0/1 outcome. The linear probability model is OLS on the 0/1 variable: simple, but it can predict probabilities outside 0 to 1 and has heteroscedastic errors. The logit and probit pass a linear index through the logistic or normal distribution function so that probabilities stay between 0 and 1, and are estimated by maximum likelihood. Their coefficients are on a different scale from each other and from the linear model, so models are compared through marginal effects.

A series has a unit root if it behaves like a random walk: shocks are permanent, its variance grows over time, and it must be differenced once to become stationary. The Dickey-Fuller and augmented Dickey-Fuller tests regress the change in the series on its lagged level, with the null hypothesis of a unit root, using non-standard critical values (about −1.95, −2.86 and −3.41 at 5% for no constant, a constant, and a constant and trend). The KPSS test reverses the hypotheses: its null is stationarity.

Two non-stationary I(1) series are cointegrated if some linear combination of them is stationary: they wander together and the gap between them keeps returning to a fixed level. It matters because regressing one unit-root series on another is normally spurious, but with cointegration the levels regression is meaningful and gives a long-run relationship. It is tested with the Engle-Granger two-step method and modelled with an error-correction model.

Variable x Granger-causes y if past values of x help predict y after past values of y have been taken into account, tested with an F test that all the lags of x are zero in the equation for y. It is a statement about prediction and timing within a model, not about economic causation.

Match the test to the symptom. Heteroscedasticity: Breusch-Pagan, White, Goldfeld-Quandt, Park or Glejser. Autocorrelation: Durbin-Watson, or Breusch-Godfrey when there is a lagged dependent variable. Multicollinearity: variance inflation factors, Klein's rule or Farrar-Glauber. Functional form: Ramsey's RESET. Structural change: Chow when the date is known, Bai-Perron when it is not. Endogeneity: the Hausman test. Unit roots: Dickey-Fuller (null: unit root) or KPSS (null: stationary). Cointegration: Engle-Granger. Normality: Jarque-Bera or Shapiro-Wilk.

The Chow test asks whether the same regression coefficients, intercept included, fit two groups or two periods. Fit the regression to everyone and keep its residual sum of squares, fit it to each group separately and keep both, then compute F = [(RSS pooled − RSS₁ − RSS₂) ÷ k] ÷ [(RSS₁ + RSS₂) ÷ (n − 2k)] with k and n − 2k degrees of freedom. The null is that the coefficients are equal. It needs the split in advance, and it rejects if only the intercept differs.

Use the Jarque-Bera test, which adds the squared departures of the residuals' skewness from 0 and kurtosis from 3, JB = n[S² ÷ 6 + (K − 3)² ÷ 24], a chi-square with 2 degrees of freedom (5% cutoff 5.99), or the Shapiro-Wilk test, which is better in small samples. Both have normality as the null. Run them on the residuals, not on the variables, and remember they have little power in small samples. In large samples the t and F tests hold approximately without normality.

In the order a standard textbook gives them: statement of the theory or hypothesis, specification of the mathematical model, specification of the statistical or econometric model (the same model with an error term), obtaining the data, estimation of the parameters, hypothesis testing, forecasting or prediction, and using the model for control or policy.