Skip to content
We use only essential cookies to keep you signed in and remember your preferences. See our Cookie Policy.
TestStop
MocksPracticeResourcesIndiaBlogPricing
Sign in
Resources › Econometrics › 2. Matrix form and Gauss-Markov
Econometrics · Lesson 2 of 18

OLS in matrix form and the Gauss-Markov theorem

The first lesson chose coefficients by minimising squared residuals and described what holds for every fit. This one writes the whole thing in matrices, which is how every later result is proved and how the entrance exams state it. It asks the two questions behind everything in econometrics: is the estimator centred on the truth, and how spread out is it? And it says exactly what “best” means in the phrase best linear unbiased estimator.

← Econometrics moduleLesson 2 of 18 · about 55 minutes
1Multiple regression2Matrix form and Gauss-Markov3t and F tests4R² and adjusted R²5Dummies and interactions6Functional forms7Misspecification8Multicollinearity9Heteroscedasticity10Autocorrelation11Instrumental variables12Simultaneous equations13Panel data14Logit and probit15Unit roots16ARMA, VAR and cointegration17Test Atlas18Hard problems
A question to hold on to
Given the assumptions of the classical linear regression model, the ordinary least squares estimators possess optimum properties given in the Gauss-Markov theorem. Which of the following is not part of this theorem? (1) The slope estimator is a linear function of random variables. (2) The expected value of the slope estimator is equal to zero. (3) No linear unbiased estimator has lower variance than that of the slope estimator. (4) The slope estimator is an unbiased estimator of the true slope.
CUET 2023Solved at the end ↓
New here?What is the Ashanagar survey?The invented household survey behind this page's examples: 600 households, generated from a model whose true coefficients we publish.

The Ashanagar survey is invented. The statistics modules used one district of 60 households. Econometrics needs more households and more variables, so this is a new, larger survey of 600 households in the same imaginary district: income, spending, household size, whether the household is in the town or a village, the head’s schooling and earnings.

Because we wrote the data-generating process, we know the true answers, and every lab can show whether a regression recovers them. Household spending, in thousands of rupees a month, was generated as 1 + 0.45 × income + 0.9 × household size + 1.5 × town + an error with standard deviation 2.5. A real survey never tells you its true coefficients. This one does, so you can see how close an estimate comes.

It also holds a variable, the head’s ability, that a real survey could not observe. It is kept so that the lessons on omitted variables can show what happens when it is left out. Every figure here is illustrative, not official statistics.

Households
600
Share in the town
41%
Mean monthly income
₹15.4 thousand
Mean monthly spending
₹12.6 thousand
Mean schooling of the head
8.2 years

How the module uses it →

In this guide
Lesson · about 55 minutes
  1. Regression in matrices
  2. What the matrices carry
  3. The assumptions, in a list
  4. Unbiased, and its variance
  5. Best: the Gauss-Markov theorem
  6. What makes a coefficient precise
  7. The exam that dropped a question
  8. The opening question, solved
  9. The lesson on one screen
  10. Check yourself
  11. Past papers, by level
01
The idea

Regression in matrices

Stack the n observations. Put the dependent variable in a column y, with n rows. Put the regressors in a matrix X, with a column of ones for the intercept and then one column per regressor, so X has n rows and k + 1 columns. Collect the coefficients in a column β with k + 1 rows, and the errors in a column u with n rows. The whole model is one line:

The model, and the estimator
y=Xβ+u,b=(X′X)−1X′y
y is n × 1, X is n × (k + 1), β is (k + 1) × 1 and u is n × 1. The estimator exists only when X′X can be inverted.

The estimator is the normal equations X′X b = X′y, solved. It is nothing mysterious: X′X collects sums of squares and cross-products of the regressors, X′y collects their cross-products with y, and b is what you get by dividing one by the other, the matrix way. The stepper below does the whole calculation on five households, in exact fractions. Change any number, switch on a second regressor, and then press Try z = 2x to see what happens when one regressor is a multiple of another.

Five households: x is the head’s years of schooling, y is monthly earnings in thousands of rupees. Edit any number (whole numbers from -30 to 40).

HouseholdxyFittedResidual
18.60.4
211.5−0.5
314.40.6
417.3−1.3
520.20.8
  1. 1
    The matrices X and y

    X has a column of ones for the intercept, then one column per regressor, so it is 5 × 2. y is 5 × 1.

    16
    18
    110
    112
    114
    9
    11
    15
    16
    21
  2. 2
    X′X: sums and cross-products

    Entry (i, j) is the sum, over households, of column i times column j. The corner is n = 5, and the next entry in the first row is Σx = 50.

    550
    50540
  3. 3
    X′y
    72
    778
  4. 4
    The inverse (X′X)⁻¹

    Exact fractions. Multiply it by X′X and you get the identity.

    27/10−1/4
    −1/41/40
  5. 5
    The estimates b = (X′X)⁻¹X′y
    −1/10
    29/20

    So the fitted line is ŷ = −0.1 + 1.45 x.

  6. 6
    The check: the residuals are orthogonal to every column of X

    Σ(residual × 1) = 0; Σ(residual × x) = 0. That is what the normal equations X′(y − Xb) = 0 say.

02
What the matrices carry

What the matrices carry

X′X and invertibility. The corner entry of X′X is n, the first row holds the sums of each regressor, and the rest are sums of squares and cross-products. X′X has an inverse exactly when no column of X is an exact combination of the others. That rules out two regressors that are multiples of each other, a dummy for every category together with an intercept, and, in a simple regression, an x that takes the same value in every household (CUET 2023: the values of the explanatory variable must not all be the same). The assumption is called no perfect collinearity, or full column rank.

The hat matrix and the residual maker. Two matrices built from X do all the work in later proofs:

Hat matrix H and residual maker M
y^​=Hy,H=X(X′X)−1X′,e=My,M=I−H
H puts the hat on y. M turns y into residuals. Both are symmetric and idempotent: HH = H and MM = M.

Idempotent means that applying the operation twice changes nothing: once y has been projected on the regressors, projecting again leaves it where it is. Two traces matter. The trace of H is the number of estimated parameters, k + 1, and the trace of M is n − k − 1, the residual degrees of freedom. For the five households in the stepper the diagonal entries of H are 0.6, 0.3, 0.2, 0.3, 0.6, which add to 2 = k + 1 = 2. A diagonal entry is a household’s leverage: how much its own y pulls its own fitted value. Households far from the mean of x have the highest leverage.

Without an intercept. Drop the column of ones and the normal equation Σe = 0 disappears with it. Regress spending on income alone, through the origin, in the survey: the slope is Σxy ÷ Σx² = 0.7741 and the residuals add to 407.6, not zero. The usual R² is defined with the centred total variation, which no longer splits cleanly, so software reports an uncentred R² of 0.939, compared with 0.606 for the same regression with an intercept. The two cannot be compared, and the uncentred figure can be large even when x and y are barely related. The mean square error is SSE ÷ (n − 1), since only one parameter was estimated.

Watch out for

The mistakes examiners are fishing for

  • Calling the formula (X′X)⁻¹X′y valid for any data. It needs X′X invertible, which fails under perfect collinearity.
  • Comparing R² from a regression with and without an intercept. They are different statistics.
03
The classical model

The assumptions, in a list

The classical linear regression model has six assumptions. Each result in econometrics needs some of them and not others, and exam questions are built to see whether you know which.

AssumptionIn wordsWhat it buys, and what goes wrong without it
1. Linear in parametersy = Xβ + u. The model is linear in the coefficients. It need not be linear in the variables: ln y, x² and 1/x are all allowed.The estimator is a linear function of y. Without it, least squares is not the right method.
2. No perfect collinearityThe columns of X are linearly independent, and n exceeds the number of parameters. X is not all one value.X′X can be inverted, so there is one unique b. Without it, b does not exist.
3. Zero conditional meanE(u | X) = 0. The error averages zero whatever the regressors, so it is uncorrelated with them.Unbiasedness (and consistency). Without it, b is biased: omitted variables, measurement error and simultaneity all break it.
4. Constant varianceVar(u | X) = σ² for every observation: homoscedasticity.The variance formula σ²(X′X)⁻¹ and the Gauss-Markov result. Without it, b is still unbiased, but its standard errors are wrong and it is not best.
5. No autocorrelationCov(uᵢ, uⱼ) = 0 for i ≠ j: the errors are uncorrelated across observations.Again the variance formula and BLUE. Without it, b is unbiased, but standard errors are wrong and it is not best.
6. Normal errorsu is normally distributed.Exact t and F distributions in small samples. Not needed for unbiasedness, for the variance, or for BLUE. In large samples the central limit theorem makes the tests approximately right anyway.

Two traps recur. First, assumption 3 is about the error and the regressors, not about y: Cov(y, u) is not zero. Since y = Xβ + u, the covariance of y with u is the variance of u, which is positive (JAM 2025). Second, “linear” means linear in the parameters. A regression of y on x and x² is a linear regression; one of y on x^β is not (JNU 2021, JAM 2025).

Pause and predict
Which assumption, when it fails, makes the OLS estimator biased?
04
Derive it

Unbiased, and its variance

The two properties that everything rests on follow in a few lines from writing the estimator as the truth plus a term in the errors. Open the steps one at a time.

Derive it

OLS is unbiased and its variance is σ²(X′X)⁻¹

We want to show
If E(u | X) = 0 and Var(u | X) = σ²I, then E(b)=β and Var(b)=σ2(X′X)−1.
  1. 1
    Write the estimator as the truth plus noise.

    Substitute y = Xβ + u into b = (X′X)⁻¹X′y:

    b=(X′X)−1X′(Xβ+u)=I(X′X)−1X′X​​β+(X′X)−1X′u=β+(X′X)−1X′u

    The estimate is the true β plus a term built from the errors. All the sampling variation is in u.

  2. 2
    Unbiased.

    Take the expectation given X. The matrix (X′X)⁻¹X′ is fixed once X is given, so it passes through:

    E(b∣X)=β+(X′X)−1X′E(u∣X)=β+0=β

    Only E(u | X) = 0 was used, with X′X invertible. Nothing about the errors’ variance. This is why OLS stays unbiased when the errors have unequal variances or are correlated with each other.

  3. 3
    Variance.

    The deviation b − β is (X′X)⁻¹X′u. The variance of a matrix times u is the matrix, times Var(u), times the matrix transposed:

    Var(b∣X)=(X′X)−1X′(σ2I)X(X′X)−1=σ2(X′X)−1X′X(X′X)−1=σ2(X′X)−1

    The step σ²I is where constant variance and uncorrelated errors are used. If either fails, the formula is wrong, and so are the standard errors built on it.

  4. 4
    Read it off the matrix.

    The variance of one coefficient is σ² times the matching diagonal entry of (X′X)⁻¹. In the survey regression the entry for income is 0.000055. The true error standard deviation is 2.5, so the true standard error of the income coefficient is √(6.25 × 0.000055) = 0.0185. The regression, which does not know σ, estimates it with s = 2.564and reports 0.019. Close, as it should be.

Step 1 of 4
So
b = β + (X′X)⁻¹X′u, so OLS is unbiased whenever the errors average zero given X, and its variance is σ²(X′X)⁻¹ whenever they also have constant variance and are uncorrelated. The first needs fewer assumptions than the second.
05
Interactive

Best: the Gauss-Markov theorem

Unbiasedness alone does not make an estimator good: the slope through just the first and last observation is also unbiased, and it is a poor use of the data. The question is which unbiased estimator is least spread out. The Gauss-Markov theorem answers for a whole family:

The Gauss-Markov theorem
Under assumptions 1 to 5, for any linear unbiased estimator b~:Var(b~)−Var(b) is positive semidefinite
BLUE: Best (smallest variance) among Linear Unbiased Estimators. Linear means a weighted sum of the y values with weights that depend only on X.

The proof is short. Write any other linear unbiased estimator as b̃ = [(X′X)⁻¹X′ + D]y for some matrix D. Unbiasedness forces DX = 0. Its variance is then σ²[(X′X)⁻¹ + DD′], and DD′ cannot be negative, so no choice of D can beat D = 0, which is OLS itself. Watch it happen: the lab draws the same data twice over and estimates the slope with least squares and with a rival, 2,000 times.

OLS: average slope1.5
Rival: average slope1.501
OLS: spread (sd)0.22
Rival: spread (sd)0.259
OLS spread ÷ rival spread0.85
0.60.811.21.41.61.822.22.40.511.52Estimated slope
Least squaresRival: top half against bottom halfTrue slope 1.5
What it shows

Both estimators are unbiased: their averages, 1.5 and 1.501, sit on the true 1.5. Neither is wrong on average. But the rival (the gap between the mean y of the upper and lower halves of x, over the gap in mean x) is a more wasteful use of the same data, and its spread is 0.259 against 0.22 for least squares (theory: 0.255 and 0.221). Among all estimators that are linear in y and unbiased, least squares is the one with the smallest spread, and no rival you can invent beats it, whatever n and σ you choose. That is the Gauss-Markov theorem.

Watch out for

The mistakes examiners are fishing for

  • Thinking BLUE needs normal errors. It does not. Assumptions 1 to 5 are enough (JNU 2013 asks which condition is not necessary, and the answer is normality).
  • Hearing “best” as “best of all estimators”. The theorem compares only linear unbiased estimators. Some biased estimators have lower mean squared error, and with non-normal errors some nonlinear estimators have lower variance.
  • Forgetting that constant variance and no autocorrelation are in the theorem. Without them OLS is still unbiased but no longer best. Lessons 9 and 10 show the estimators that are.
Pause and predict
The errors in a regression are not normally distributed, but assumptions 1 to 5 hold. Is OLS still the best linear unbiased estimator?
06
Reading the variance

What makes a coefficient precise

The variance of one coefficient is σ² times the matching diagonal entry of (X′X)⁻¹. That entry has a clean form, which follows from the partialling-out result of lesson 1:

Variance of the j-th coefficient
Var(bj​)=∑i​(xij​−xˉj​)2(1−Rj2​)σ2​
R² here is from regressing x_j on the other regressors, with an intercept. The factor 1 ÷ (1 − R²) is the variance inflation factor.

Four levers move it: a smaller error variance σ² (a better-fitting model), a larger sample (the sum grows with n), more spread in xj (the sum grows), and a lower correlation of xj with the other regressors (Rj² nearer zero). The last is multicollinearity, the subject of lesson 8. Delhi School 2018 asks which of these raise the variance: smaller sample, less variation in the regressor, more variation in the error, higher correlation between regressors. All four do.

RegressorΣ(x − x̄)²R² on the othersVIFSE if uncorrelatedSE reported
Income26,183.90.3051.440.01580.019
Household size1,474.20.2261.290.06680.0759
Town dummy144.80.3371.510.21310.2617

Survey regression of spending on all three. The reported standard error is the standard error if the regressor were uncorrelated with the others, multiplied by √VIF. Here the regressors overlap only mildly, so the inflation is small. Lesson 8 shows what happens when it is not.

07
A question the exam withdrew

The exam that dropped a question

IIT JAM 2022 printed a question with n = 23 observations, three coefficients, X′Y = (0.3, 0.2, 0.1)′ and e′e = 0.7, and gave (X′X)⁻¹ as

The matrix as printed
(X′X)−1=​0.30.40.4​0.5−0.60.5​0.80.20.3​​
Asked for the t statistic for β₃ = 0.

This cannot be the inverse of any X′X. The matrix X′X is symmetric, because its (i, j) entry is the sum of column i times column j, which is the same as column j times column i. The inverse of a symmetric matrix is symmetric. The printed matrix is not: its (1, 2) entry is 0.5 and its (2, 1) entry is 0.4. The data are inconsistent, and no t statistic is defined. The bank’s record of the official key says the question was dropped and every candidate was given the marks.

If you ignored the problem and read the matrix as printed, the third row would give b₃ = 0.4 × 0.3 + 0.5 × 0.2 + 0.3 × 0.1 = 0.25, s² = 0.7 ÷ 20 = 0.035, and t = 0.25 ÷ √(0.035 × 0.3) = 2.44. That is not an accepted answer, and it is not offered in practice. The lesson is the check, which takes five seconds: X′X and its inverse are always symmetric, and a variance matrix has a non-negative diagonal.

08
Solving the opening question

The opening question, solved

CUET 2023 · moderate
Which of these is not part of the Gauss-Markov theorem? (2) The expected value of the slope estimator is equal to zero.
  1. 1
    Recall what the theorem says.
    Under the classical assumptions the OLS slope estimator is linear in y (statement 1), unbiased (statement 4), and has the smallest variance among linear unbiased estimators (statement 3).
  2. 2
    Test statement 2.
    Unbiased means E(b) = β, the true slope, not zero. The expected value equals zero only if the true slope is zero.
Answer
Statement 2 is not part of the theorem. It confuses “unbiased” (the expected estimate equals the true value) with “the expected estimate is zero”. The other three are the three words in BLUE: linear, unbiased, best.
In economics

Almost every regression in applied economics rests on assumption 3, and almost every argument in a seminar is about whether it holds. Wages on schooling, spending on income, growth on investment: in each, the question is whether something left in the error is related to the regressor. The estimator is only as good as that one claim. Everything from lesson 7 onward is about what to do when it fails.

09
Revise

The lesson on one screen

The lesson on one screen
Model and estimator
y = Xβ + u. b = (X′X)⁻¹X′y, the solution of the normal equations. Needs X′X invertible: no perfect collinearity, and x not all the same.
H and M
ŷ = Hy, e = My, H = X(X′X)⁻¹X′, M = I − H. Symmetric and idempotent. trace H = k + 1. trace M = n − k − 1.
Assumptions
1 linear in parameters. 2 no perfect collinearity. 3 E(u|X) = 0. 4 constant variance. 5 no autocorrelation. 6 normality.
Unbiased
b = β + (X′X)⁻¹X′u, so E(b) = β using only assumption 3 (and 1 and 2).
Variance
Var(b) = σ²(X′X)⁻¹, using assumptions 4 and 5. Var(b_j) = σ² ÷ [Σ(x_j − x̄_j)²(1 − R_j²)].
Gauss-Markov
Under 1 to 5, OLS is BLUE. Normality is not needed. Without 4 or 5, OLS is unbiased but no longer best.
No intercept
Σe need not be 0. R² is uncentred. The mean square error divides by n − 1.
Checks
X′X and (X′X)⁻¹ are symmetric. Cov(y, u) = σ², not zero. Linear in parameters, not necessarily in variables.
10
Practice

Check yourself

0 of 13 correct so far

Compute

Dimensions
A regression has an intercept and three regressors, estimated on 50 observations. How many rows does X′X have?
The trace of M
A regression with an intercept and two regressors uses 30 observations. What is the trace of the residual maker M?
A standard error from the matrix
σ² is estimated as 4, and the diagonal entry of (X′X)⁻¹ for the coefficient of interest is 0.01. Find its standard error.
Through the origin
Regress y on x with no intercept, using (x, y) = (1, 2), (2, 3) and (3, 7). Find the slope.
Solve the normal equations
X′X = [[4, 10], [10, 30]] and X′y = (20, 55)′. Find the slope, the second element of b = (X′X)⁻¹X′y.
The error variance estimate
A regression with an intercept and two regressors has e′e = 54 on 30 observations. Find the estimate of σ².
Inflated by correlation
A coefficient has standard error 0.05 when its regressor is uncorrelated with the others. The regressor’s R² on the other regressors is in fact 0.75. Find the standard error.
A bigger sample
The sample size doubles, with the same spread of x and the same σ. By what factor does the standard error of the slope change? (Give the new over the old.)

Quick questions

1Which of these is NOT needed for OLS to be the best linear unbiased estimator?
2With k regressors and an intercept, the matrix X′X has dimensions
3Which of these makes the OLS estimator biased?
4The OLS estimator (X′X)⁻¹X′y cannot be computed when
5The Gauss-Markov theorem says OLS has the smallest variance among
11
Real questions

Past papers, by level

Foundation

Definitions and one-step calculations: a t statistic, an R², which assumption is which. Most CUET and JNU questions sit here.

4 questionsPractise →

Standard

Restricted against unrestricted regressions, reading output, and statement questions, up to GATE and IIT JAM level.

4 questionsPractise →

Delhi School level

This topic’s hardest questions, from the DSE entrance, are worked through in Lesson 18: The algebra of least squares.

4 questionsLesson 18 →
All 8 past-paper questions for this lesson, easiest first: practise the full set →

✦These are questions from the TestStop bank. A free account answers 10 a day, each with its worked solution; All-Access opens every set with no daily limit.

Where the harder ones are

The Delhi School’s questions on these ideas (the variance of one coefficient, regression through the origin) are collected in the hard-problems lesson at the end of the module. Until it is published, they are not served in practice.

← Lesson 1Multiple regression: what holding other things constant meansLesson 3 →t tests, F tests and reading regression output

Read next

All resources →
EconometricsEconometrics 3: t tests, F tests and reading regression outputThe t test and interval for one coefficient, the overall F, F for restrictions from sums of squares and from R squared, F equals t squared for one restriction, and why two t tests are not one F, with an interactive regression table.Complete guide · 60 minEconometricsEconometrics 1: multiple regression, partial effects and holding other things constantThe multiple regression model, the normal equations, what a partial coefficient means and how partialling out shows it, the facts true of every OLS fit, reading a coefficient table, and standardised coefficients.Complete guide · 55 minStatisticsInference 4: estimators, bias, variance and consistencyWhat makes an estimator good: unbiasedness, efficiency, consistency and sufficiency, mean squared error, and the trade between bias and variance.Complete guide · 45 min