Panel data: fixed and random effects

Data collected at a point in time are cross-sectional (CUET asks it directly), and one state over many years is a time series. A panel is balanced if every individual is observed in every period, unbalanced otherwise. With N individuals and T periods the sample has n = NT observations. A short panel has many individuals and few periods (typical of surveys), a long panel few individuals and many periods (typical of country data).

The simplest approach stacks all the observations and runs OLS, ignoring that they come in groups. That is the pooled regression, y_it = β₀ + β₁x_it + v_it. But every observation of individual i shares an individual effect μ_i: something about that person, firm or country that persists, such as ability, management quality or institutions. It sits in the error: v_it = μ_i + ε_it. If μ_i is related to x, we have an omitted variable that is correlated with the regressor, and the bias of lesson 7 returns:

In the lab each individual in colour has its own level (μ), and individuals with a high level also have a high x. Looking across individuals, the pattern is steep; looking within one individual, the slope is the true 1. The dashed pooled line is much steeper than any individual’s own line. Press “draw new samples” and raise c to see the pooled and between estimators drift away from 1 while the within and first-difference estimators stay put.

If μ_i is constant over time, differences within an individual cannot contain it. Three equivalent ways to use that:

• The within (fixed-effects) estimator. Subtract each individual’s average from each variable and run OLS on the demeaned data. μ_i disappears.
• First differences. Subtract the previous period: Δy_it = βΔx_it + Δε_it. μ_i disappears. With T = 2 it is identical to the within estimator; with T > 2 they differ (in efficiency, and in how they treat serial correlation in ε).
• Least squares with dummy variables (LSDV). Add a dummy for each individual (N − 1 of them, with a constant). The slope is identical to the within estimator’s, and the dummies’ coefficients estimate the individual effects.

What fixed effects costs: degrees of freedom (n − N − k), and the ability to estimate the effect of any variable that does not change within an individual: sex, birthplace, a country’s geography. Those are absorbed by the individual effect. The within estimator answers “what happens when x changes for the same individual?” and is silent on differences between individuals.

The random-effects (RE) model treats μ_i as a random draw, assumed uncorrelated with x. The composite error v_it = μ_i + ε_it is then correlated within an individual (the same μ_i in each period), so OLS is unbiased but inefficient, and GLS fixes that. GLS is computed as OLS on quasi-demeaned data: subtract a fraction θ of the individual mean,

RE is more efficient than FE when its assumption holds, because it also uses the variation between individuals, and it can estimate the effect of time-invariant variables. But if μ_i is correlated with x, RE is biased (the lab shows it), while FE is not. Which to use? The Hausman test compares the two:

• H₀: μ_i is uncorrelated with x. Both FE and RE are consistent, and RE is efficient.
• H₁: μ_i is correlated with x. Only FE is consistent.
• Statistic: H = (b_FE − b_RE)′ [Var(b_FE) − Var(b_RE)]⁻¹ (b_FE − b_RE), chi-square with k degrees of freedom (the number of slope coefficients compared). A significant H rejects RE: use fixed effects.

A model written y_it = β₀ + β₁x_it + ε_i + u_it with a person-specific error ε_i and an idiosyncratic error u_it is the random-effects form (the individual term is a part of the error, not a set of coefficients).

World Bank, World Development Indicators: life expectancy and GDP per head for 12 economies (India, China, the United States, Japan, Korea, Brazil, Nigeria, Germany, Indonesia, Bangladesh, Ethiopia and the United Kingdom), 2000 to 2019: a balanced panel of 240 observations. Regress life expectancy on the log of GDP per head:

The estimated individual effects range from −21.6 (Nigeria) to 1 (China) years: economies differ a great deal in life expectancy at the same income, which is what pooling ignores. The within estimate is 8.13, the pooled 6.37. The Hausman test comparing fixed and random effects gives H = 1.45 with 1 degree of freedom (p = 0.23), so it does not reject random effects here; the two estimates (8.13 and 8.06) are close, and the random-effects weight θ = 0.96 is near 1, so RE here behaves almost like FE.

The first-difference slope (3.1) is well below the within slope. They answer slightly different questions: the within estimator uses all the variation over twenty years, including long slow trends in both GDP and life expectancy, whereas first differences use only year-to-year changes, which are small and noisy; one standard reason for such a gap is that differencing amplifies measurement error. These are real data, and no estimator here proves that GDP causes longer lives: fixed effects removes what is constant about an economy, not what changes with GDP and also lengthens life.

This lesson’s set is short: few papers ask about panels, and two further questions (on the form of a random-effects model, and on panel data for states) are waiting for a person to sign off their keys. The instrumental-variables alternative to fixed effects is in lesson 11.