Econometrics · Lesson 1 of 18

Multiple regression: what holding other things constant means

Almost nothing in economics depends on one thing. Household spending depends on income, but also on how many people the household feeds and where it lives. A multiple regression puts all of them in at once and reports the effect of each with the others held fixed. This lesson says exactly what that phrase means, shows it on a picture you can change, and ends with the habit that matters most in an exam: reading a coefficient table and working out what happens when several variables change together.

← Econometrics moduleLesson 1 of 18 · about 55 minutes

A question to hold on to

Data from 107 home sales gave P̂ = 119.2 + 0.485 BD + 23.4 BA + 0.156 HS + 0.002 PS + 0.090 A − 35.6 PC, where P is price in ₹ thousand, BD is bedrooms, BA is baths, HS is house size in square feet, PS is plot size, A is age and PC is 1 if the house is in poor condition. If a homeowner converts a bedroom into a bathroom, what is the expected increase in the value of the house?

Delhi School 2019Solved at the end ↓

New here?What is the Ashanagar survey?The invented household survey behind this page's examples: 600 households, generated from a model whose true coefficients we publish.

The Ashanagar survey is invented. The statistics modules used one district of 60 households. Econometrics needs more households and more variables, so this is a new, larger survey of 600 households in the same imaginary district: income, spending, household size, whether the household is in the town or a village, the head’s schooling and earnings.

Because we wrote the data-generating process, we know the true answers, and every lab can show whether a regression recovers them. Household spending, in thousands of rupees a month, was generated as 1 + 0.45 × income + 0.9 × household size + 1.5 × town + an error with standard deviation 2.5. A real survey never tells you its true coefficients. This one does, so you can see how close an estimate comes.

It also holds a variable, the head’s ability, that a real survey could not observe. It is kept so that the lessons on omitted variables can show what happens when it is left out. Every figure here is illustrative, not official statistics.

Households: 600
Share in the town: 41%
Mean monthly income: ₹15.4 thousand
Mean monthly spending: ₹12.6 thousand
Mean schooling of the head: 8.2 years

How the module uses it →

The idea

From one regressor to several

In the regression primer one variable explained another, and the inference lesson asked how far that slope could be from the truth. Real outcomes have several causes, so the model grows a term for each of them:

The multiple regression model

y_{i} = β_{0} + β_{1} x_{i 1} + β_{2} x_{i 2} + \dots + β_{k} x_{ik} + u_{i}

k regressors and an intercept, so k + 1 coefficients. The error u collects everything the regressors leave out.

Each coefficient has a precise reading: β_j is the change in the expected value of y when x_j rises by one unit and every other regressor stays where it is. That last clause is the whole point, and it is called ceteris paribus, other things equal.

Take the Ashanagar survey. Regressing monthly household spending (₹ thousand) on income, household size and a town dummy gives

Estimated spending equation, 600 households

spend = 1.225 + 0.442 income + 0.904 size + 1.466 town

The true values we built into the data are 1, 0.45, 0.9 and 1.5. Each estimate is within about one standard error of its true value.

Income, 0.44. Two households of the same size, both in the town or both in villages, whose incomes differ by ₹1,000 a month differ in spending by about ₹442.
Household size, 0.9. Each extra person, at the same income and in the same place, adds about ₹904 a month.
Town, 1.47. A dummy variable. A town household spends about ₹1,466 more than a village household with the same income and size.
Intercept, 1.23. The predicted spending of a village household with no members and no income, which is outside the data and should not be read literally.

Watch out for

The mistakes examiners are fishing for

Reading a coefficient on its own. It is only meaningful given the list of the other variables in the regression. Change the list and the number changes, as the next chapter shows.
Reading the intercept as a prediction. It is the fitted value when every regressor is zero, which is often a household or a house that does not exist.

How it is computed

Choosing the coefficients

With one regressor, least squares chose the line with the smallest sum of squared vertical gaps. With several it does the same in more dimensions: it picks the k + 1 coefficients that make the sum of squared residuals Σe² as small as possible. You cannot draw the surface for three regressors, but nothing in the logic changes.

Setting the slope of Σe² to zero in every direction gives k + 1 equations, the normal equations. They say two things that every later result leans on: the residuals add up to zero, and they are uncorrelated with every regressor. The matrix form of the same equations is the subject of lesson 2.

Derive it

Least squares makes the residuals orthogonal to every regressor

We want to show

The coefficients that minimise the sum of squared residuals satisfy

X^{'} X b = X^{'} y

, which says the residuals sum to zero and are uncorrelated with every regressor.

Write the thing to be minimised.
With k coefficients b₀, b₁, …, the residual of household i is e_i = y_i − b₀ − b₁x_i1 − … and the criterion is the sum of their squares:
$S (b) = i \sum e_{i}^{2} = i \sum (y_{i} - b_{0} - b_{1} x_{i 1} - \dots - b_{k} x_{ik})^{2}$
Set each slope of S to zero.
The sum of squares has one minimum, where its slope in every direction is zero. Differentiating with respect to b_j and cancelling the −2:
$\frac{\partial S}{\partial b _{j}} = - 2 i \sum x_{ij} e_{i} = 0 for every j = 0, 1, \dots, k (x_{i 0} = 1)$
That is k + 1 equations, one per coefficient. In matrix form they say X′e = 0.
What those equations mean.
The j = 0 equation is Σe_i = 0: the residuals sum to zero, so their mean is zero. The others say Σx_ije_i = 0: the residuals are uncorrelated with each regressor. For the survey regression of spending on income, household size and town, the four sums are 0, 0, 0, 0, which are zero up to rounding.
The normal equations.
Put e = y − Xb into X′e = 0:
$X^{'} (y - X b) = 0 ⟹ X^{'} X b = X^{'} y ⟹ b = (X^{'} X)^{- 1} X^{'} y$
The last step needs X′X to have an inverse, which fails exactly when one column of X is an exact combination of the others.
A consequence worth knowing.
The fitted values are a combination of the columns of X, so they are orthogonal to the residuals too: Σŷe = 0. Then SST = Σ(y − ȳ)² splits cleanly into SSR + SSE with no cross term, which is why R² = SSR ÷ SST is a share between 0 and 1. For the survey regression, SST = 12,313.3, SSR = 8,395 and SSE = 3,918.2, and 8,395 + 3,918.2 = 12,313.3.

Step 1 of 5

Pause and predict

A student fits a regression that includes an intercept and finds that the residuals add up to 12.4. What is the likely explanation?

Interactive

What holding constant means: seeing it

In survey data nothing is literally held fixed: households differ in everything at once. The regression does something cleverer. To get the effect of income on spending with household size and town held constant, it first takes household size and town out of both variables, removing whatever they explain of spending and of income. Then it draws the ordinary line through what is left. That slope is the multiple-regression coefficient, exactly.

Try it. Pick a story, tick the controls, and compare the two pictures. The same households are in both, and colour shows town and village.

Hold constantTown or villageAbility (Invented: a real survey could not observe it.)

Horizontal: years of schooling of the head. Vertical: head's monthly earnings (₹ thousand).

VillageTownLine with no controls

Both axes after the ticked controls are removed from schooling and from earnings, each measured from its own average.

Same line, centred (no controls yet)

Slope with no controls1.807

Slope after the ticked controls1.807

Coefficient on x in the full regression1.807

True value (we wrote the model)1.3

What it shows

Nothing is held constant yet, so the right-hand picture is the left one with the centre moved to zero, and the slope is the raw 1.81. Tick a control to take its influence out of both variables.

In the spending story the raw slope of spending on income is 0.534, but with size and town held constant it falls to 0.442, close to the true 0.45. In the earnings story the slope of earnings on schooling shrinks as you tick the controls, and with both it lands on the truth. The coefficient you read from a regression depends on which controls are in it.

Pause and predict

Spending on household size alone has a slope of 1.16. With income and town in the regression the coefficient on size is 0.9. Why is the raw slope larger?

Derive it

A multiple-regression coefficient is a simple slope after the other variables are removed

We want to show

In the regression of y on x₁ and a set of other variables Z, the coefficient on x₁ equals the slope from regressing the part of y not explained by Z on the part of x₁ not explained by Z (Frisch-Waugh-Lovell).

Split the regression into the variable of interest and the rest.
$y = x_{1} b_{1} + Z c + e, X^{'} e = 0 so x_{1}^{'} e = 0 and Z^{'} e = 0$
Define the 'remove Z' operation.
Regress any variable on Z and keep the residual. Write M for that operation, so Mv is v with Z taken out. Two facts: MZ = 0, because Z is entirely explained by Z, and M e = e, because e is already orthogonal to Z.
Apply M to the whole equation.
$M y = (M x_{1}) b_{1} + (M Z) c + M e = (M x_{1}) b_{1} + 0 + e$
So the cleaned-up y equals b₁ times the cleaned-up x₁, plus the same residual e as before.
That is a simple regression.
The residual e is orthogonal to Mx₁, since Mx₁ is x₁ minus a combination of the columns of Z and e is orthogonal to all of those. A simple regression through the origin of My on Mx₁, whose error is orthogonal to the regressor, has slope
$b_{1} = \frac{( M x _{1} ) ^{'} ( M y )}{( M x _{1} ) ^{'} ( M x _{1} )}$
For the survey, take spending on income with household size and the town dummy removed from both. The slope of what is left is 0.442383. The coefficient on income in the regression with all three variables is 0.442383. They are the same number, as the lab on this page shows.

Step 1 of 4

Watch out for

The mistakes examiners are fishing for

Thinking the data were held fixed. Nothing was. Regression compares like with like statistically: it adjusts for the other variables’ measured influence, and cannot adjust for anything that is not in the regression.
Assuming extra controls always change the coefficient. A control changes it only if it is related to the variable of interest and to y. A control uncorrelated with x leaves the slope unchanged and only tightens the standard error.

Worked examples

Reading a coefficient table

Exams give you a fitted equation and ask what happens to the predicted value when something changes. The rule has one step: for each variable that changes, multiply its coefficient by the size of its change, then add. Variables that do not change contribute nothing. Here is the table from the opening question.

Variable	Meaning	Coefficient	Std. error
Intercept		119.2	23.9
BD	bedrooms	0.485	2.61
BA	bathrooms	23.4	10.76
HS	house size, sq ft	0.156	0.011
PS	plot size, sq ft	0.002	0.00048
A	age, years	0.090	0.311
PC	1 if poor condition	−35.6	10.5

Delhi School 2019, scenario 3. Dependent variable: price in ₹ thousand. n = 107, R² = 0.72, standard error of the regression 41.5.

Delhi School 2019 · easy

A homeowner adds a new bathroom, which increases the house size by 100 square feet. What is the expected increase in the value of the house?

List what changes.
Baths rise by 1 and house size by 100 square feet. Bedrooms, plot size, age and condition are unchanged.
Multiply each coefficient by its change and add.
23.4 × 1 + 0.156 × 100 = 23.4 + 15.6 = 39.0.
Mind the units.
Price is in ₹ thousand, so the increase is 39.0 thousand rupees.

Answer

₹39,000. Notice that the bathroom’s coefficient alone, 23.4, is not the answer: the question changes two variables, and both count.

Dummy variables work the same way. Moving a house from good to poor condition changes PC from 0 to 1 and nothing else, so the predicted price falls by exactly the dummy’s coefficient, ₹35,600. A dummy’s coefficient is a difference between two groups with all else equal.

Facts that hold whatever the data

What is true of every OLS fit

The normal equations force several features on any least-squares fit, and exams test them as statements to be called true or false. With an intercept in the model:

The residuals sum to zero, so the fitted values have the same mean as y.
The residuals are uncorrelated with every regressor, and with the fitted values.
The fitted surface passes through the point of means: ȳ = b₀ + b₁x̄₁ + … + b_kx̄_k.
Regressing y on the fitted values gives intercept 0 and slope 1, with the same R² as the original regression.
The residual degrees of freedom are n − k − 1: one is lost for each of the k slopes and one for the intercept.

Without an intercept, the first of these fails: the residuals need not sum to zero, and R² is computed differently (lesson 2 returns to this). Here is a five-statement drill.

True or false: facts about OLS · 1 of 5

In an OLS regression with an intercept, the residuals add up to zero.

Watch out for

The mistakes examiners are fishing for

Mixing up the regression of y on e with the regression of y on ŷ. Regressing on ŷ reproduces R². Regressing on e gives 1 − R² (the Delhi School asks exactly this).
Counting degrees of freedom as n − k. With k slopes plus an intercept it is n − k − 1. Quote the convention a question uses: some papers write k for all parameters, so n − k.

Comparing sizes

Which variable matters more?

Coefficients carry the units of their variables, so they cannot be compared: ₹0.44 thousand per ₹ thousand of income is not comparable with ₹0.9 thousand per person. A standardised coefficient removes the units. It says how many standard deviations y moves when x moves by one standard deviation:

Standardised coefficient

β_{j}^{*} = b_{j} \frac{s _{x_{j}}}{s _{y}}

The same number as the coefficient from regressing z-scores on z-scores.

Variable	Coefficient	Std. deviation of x	Standardised
Income (₹ thousand)	0.442	6.612	0.645
Household size (persons)	0.904	1.569	0.313
Town (1) or village (0)	1.466	0.492	0.159

Spending has a standard deviation of 4.53 (₹ thousand) in this survey.

Income dominates: a one-standard-deviation rise in income goes with a 0.65 standard-deviation rise in spending, against 0.31 for size and 0.16 for the town dummy. Size’s raw coefficient is about twice income’s, but household sizes vary little, so a standard-deviation change in size is a small step, and the town dummy, with the largest raw coefficient of all, only switches between two groups. In the Delhi School version of this question you are given b, s_x and s_y and asked for the product b × s_x ÷ s_y.

Not a measure of importance by itself

A large standardised coefficient says a variable moves y a lot per typical change in x. It does not say the variable can be changed, or that the relationship is causal. Use it to compare sizes, not to rank policies.

Solving the opening question

The opening question, solved

Delhi School 2019 · easy

If a homeowner converts a bedroom into a bathroom, what is the expected increase in the value of the house?

Two variables change at once.
Bedrooms fall by 1 and baths rise by 1. Nothing else in the equation changes: the house size is assumed the same, since a room was converted, not added.
Add each coefficient times its change.
0.485 × (−1) + 23.4 × (+1) = −0.485 + 23.4 = 22.915.
Convert the units.
The dependent variable is in ₹ thousand, so the change is 22.915 thousand rupees.

Answer

₹22,915. A careless reading takes only the bathroom’s coefficient, 23.4, and gets ₹23,400, or adds 0.485 instead of subtracting it. Always write the change in each variable, with its sign, before multiplying.

Revise

The lesson on one screen

Model: y = β₀ + β₁x₁ + … + β_k x_k + u. Each β_j is the effect of x_j with the other regressors held fixed.
Estimation: Least squares minimises Σe². The normal equations X′e = 0 say the residuals sum to zero and are uncorrelated with every regressor.
Partialling out: The coefficient on x₁ is the simple slope after the other regressors are removed from both y and x₁ (Frisch-Waugh-Lovell).
Changes: Predicted change = Σ coefficient × change in that variable. Several variables changing means several terms, with signs.
Always true (with an intercept): Σe = 0; e uncorrelated with each x and with ŷ; the line passes through the means; y on ŷ gives slope 1, intercept 0, same R²; y on e gives 1 − R².
Degrees of freedom: n − k − 1 with k slopes and an intercept.
Comparing sizes: Standardised coefficient = b × s_x ÷ s_y. Units drop out. Not a measure of causal importance.

Practice

Check yourself

0 of 13 correct so far

Compute

A prediction

The fitted equation is ŷ = 10 + 2x₁ + 3x₂. Find the predicted y when x₁ = 4 and x₂ = 5.

Two variables change

Using ŷ = 10 + 2x₁ + 3x₂, x₁ rises by 3 and x₂ falls by 2. Find the change in predicted y.

Change

A dummy coefficient

A wage regression gives ŵ = 8 + 1.5 × experience + 4D, where D = 1 for men and 0 for women. At the same experience, by how much is the predicted wage of a man above that of a woman?

Difference

The intercept from the means

A regression with an intercept has ȳ = 20, x̄₁ = 5, x̄₂ = 10, b₁ = 2 and b₂ = −1. Find the intercept.

b₀

Degrees of freedom

A regression of y on four regressors and an intercept uses 60 observations. How many residual degrees of freedom does it have?

A standardised coefficient

b = 2.3 for a regressor with standard deviation 5, and the dependent variable has standard deviation 500. A one-standard-deviation change in the regressor goes with how many standard deviations of change in y?

Standardised

A missing residual

A regression with an intercept on five observations has residuals 2, −1, 3 and −2 for the first four. Find the fifth.

Fifth residual

Partialling out

After removing x₂ from both y and x₁, the cleaned-up variables have Σx̃ỹ = 12 and Σx̃² = 24. Find the coefficient on x₁ in the regression of y on x₁ and x₂.

b₁

Quick questions

1In a multiple regression, the coefficient on x₁ measures the change in y per unit of x₁

2In an OLS regression that includes an intercept, which is always true?

3Adding a second regressor changes the estimated coefficient on the first if the second

4A regression has k slopes and an intercept, estimated on n observations. The residual degrees of freedom are

5The fitted OLS surface (with an intercept) always passes through

Real questions

Past papers, by level

Foundation

Definitions and one-step calculations: a t statistic, an R², which assumption is which. Most CUET and JNU questions sit here.

3 questionsPractise →

Delhi School level

This topic’s hardest questions, from the DSE entrance, are worked through in Lesson 18: The algebra of least squares.

4 questionsLesson 18 →

All 3 past-paper questions for this lesson, easiest first: practise the full set →

These are questions from the TestStop bank. A free account answers 10 a day, each with its worked solution; All-Access opens every set with no daily limit.

← The moduleOverview and diagnostic Lesson 2 →OLS in matrix form and the Gauss-Markov theorem

Multiple regression: what holding other things constant means

From one regressor to several

The mistakes examiners are fishing for

Choosing the coefficients

What holding constant means: seeing it

The mistakes examiners are fishing for

Reading a coefficient table

What is true of every OLS fit

The mistakes examiners are fishing for

Which variable matters more?

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Delhi School level

Read next

Multiple regression: what holding other things constant means

From one regressor to several

The mistakes examiners are fishing for

Choosing the coefficients

What holding constant means: seeing it

The mistakes examiners are fishing for

Reading a coefficient table

What is true of every OLS fit

The mistakes examiners are fishing for

Which variable matters more?

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Delhi School level

Read next