Statistical inference · Lesson 12 of 13

Inference in simple regression

A fitted line is an estimate, so it has a margin of error like any other. The regression primer drew the line and measured how much of the scatter it explains. This lesson asks how far the slope could be from the truth, whether it is distinguishable from zero, and the difference between an interval for the average household and an interval for one household you have not yet met. Everything rests on one formula, the standard error of the slope.

← Statistical inference moduleLesson 12 of 13 · about 55 minutes

A question to hold on to

A regression estimated on a large sample gives ŷ = 5.5 + 3.2x. The 95% confidence interval for the coefficient of x is [0.26, 6.14] and R² = 0.26. What is the standard error of the estimated coefficient?

IIT JAM 2023Solved at the end ↓

New here?What is Ashanagar?The invented district behind this page's examples: 60 households, with incomes you can inspect.

Ashanagar is an invented district. The statistics lessons use it for every worked example, so you meet the same households again as the ideas build: first as a list of numbers, then as a spread, then as a population to sample from.

It has 60 households, each with a monthly income in rupees. Unless a page says otherwise, those 60 are the whole population, and a “sample” means a few of them drawn at random. Where a page needs two variables, it uses a smaller set of 12 households with the head’s years of schooling beside income.

The figures are illustrative, made up once and fixed. They are not official statistics.

Households: 60
Mean income: ₹28,783 a month
Median income: ₹25,000 a month
Standard deviation: ₹13,366
Lowest and highest: ₹10,500 and ₹72,000

Meet the full data set →

The idea

The model, and what is random

The regression primer fitted a line to twelve households: years of schooling of the head, x, and monthly income, y. Its fitted line is ŷ = 2.47 + 1.97x, in thousands of rupees. To ask how reliable that is, we need a model of how the data came about. Suppose that in the whole district, income really is a straight-line function of schooling plus noise:

The simple regression model

y_{i} = α + β x_{i} + ε_{i}, E (ε_{i}) = 0, Var (ε_{i}) = σ^{2}, ε_{i} independent

α and β are fixed unknown numbers. The errors ε are what is random. The x values are treated as given.

A different sample of households would have different errors, so a different fitted line. The estimates a and b are random variables with sampling distributions, just like the sample mean in Lesson 2. Everything in this lesson is the answer to one question: what are the centre and the spread of the distribution of b?

Interactive

How precise is the slope?

The answer is short and worth memorising. The slope is unbiased, and its standard error is the typical noise divided by the root of how spread out the x values are:

Standard error of the slope

SE (b) = \frac{s}{S _{xx}}, s = \frac{\sum e _{i}^{2}}{n - 2}

s² = SSE ÷ (n − 2) estimates σ². Sxx = Σ(x − x̄)² measures how spread out the x values are.

Derive it

The slope is a weighted sum of the y's, so its variance is σ² ÷ Sxx

We want to show

For the model y = α + βx + ε with independent errors of variance σ², the least-squares slope b has

E (b) = β

and

Var (b) = \frac{σ ^{2}}{S _{xx}}

, where S_xx = Σ(x − x̄)².

Write the slope as a sum of weights times y.
From the regression primer, b = Σ(x − x̄)(y − ȳ) ÷ S_xx. Because Σ(x − x̄) = 0, the ȳ term drops out of the numerator, leaving
$b = i \sum w_{i} y_{i}, w_{i} = \frac{x _{i} - x ˉ}{S _{xx}}$
The weights depend only on the x values, which we treat as fixed. All the randomness is in the y’s.
Three facts about the weights.
Σw = Σ(x − x̄) ÷ S_xx = 0. Σwx = Σ(x − x̄)x ÷ S_xx = S_xx ÷ S_xx = 1. And Σw² = Σ(x − x̄)² ÷ S_xx² = 1 ÷ S_xx. For the twelve Ashanagar households S_xx = 132.92, and the weights sum to 0, with Σwx = 1 and Σw² = 0.00752 = 1 ÷ 132.92.
Unbiased.
Take the expectation of Σw y with E(y_i) = α + βx_i:
$E (b) = \sum w_{i} (α + β x_{i}) = α \sum w_{i} + β \sum w_{i} x_{i} = α \cdot 0 + β \cdot 1 = β$
Only the assumption that the errors average to zero and are unrelated to x was used. Nothing about their variance, which is why unbiasedness survives heteroskedasticity (DSE 2009).
Variance.
The y’s are independent, each with variance σ², so the variance of a weighted sum is the sum of weights squared times σ², the same rule as in the two-sample lesson:
$Var (b) = σ^{2} \sum w_{i}^{2} = \frac{σ ^{2}}{S _{xx}}, SE (b) = \frac{s}{S _{xx}}$
σ is unknown, so it is replaced by s, where s² = SSE ÷ (n − 2) (the divisor n − 2 because two parameters, α and β, were estimated; the lab shows s² averaging σ² while SSE ÷ n comes up short). Here s = 4.093 and SE(b) = 4.093 ÷ √132.92 = 0.3551.
What makes the slope precise.
Three levers, all visible in σ ÷ √S_xx: less noise (smaller σ), more spread in x (a larger S_xx), and more observations (S_xx grows with n). Squeezing the x values together, as in a survey of households that all have similar schooling, ruins the estimate of the slope however many there are. Move the three sliders and compare the fan of lines with the formula.

Step 1 of 5

Watch out for

The mistakes examiners are fishing for

Dividing SSE by n or n − 1. Two parameters were estimated, so the residuals have n − 2 degrees of freedom and s² divides by n − 2 (JNU 2021, CUET 2023).
Forgetting that x matters. The slope is hard to pin down if the x values are bunched. More observations at the same x help little.
Confusing s with SE(b). s is the spread of households around the line. SE(b) is the uncertainty in the slope, and it shrinks as n grows while s does not.

Testing

Testing the slope and bounding it

With normal errors, (b − β) ÷ SE(b) follows a t distribution with n − 2 degrees of freedom. Setting β = 0 gives the test of whether x has any effect, and rearranging gives an interval for β.

Test and interval for the slope

t = \frac{b}{SE ( b )} \sim t_{n - 2}, b \pm t_{α /2, n - 2} SE (b)

Reject H₀: β = 0 when |t| exceeds the critical value, equivalently when the interval for β excludes 0.

Ashanagar · twelve households

Households with 5, 6, 8, 8, 9, 10, 10, 12, 12, 14, 15 and 16 years of schooling have monthly incomes (₹ thousand) 15, 11, 20, 15, 21, 18, 27, 22, 32, 27, 37 and 31. Test whether schooling affects income, and give a 95% interval for the effect of one more year.

Fit the line.
x̄ = 10.42, ȳ = 23, S_xx = 132.92, S_xy = 262. The slope is b = 262 ÷ 132.92 = 1.9712, and the intercept a = 2.467.
Measure the noise.
SSE = 167.56 on n − 2 = 10 degrees of freedom, so s² = 16.756 and s = 4.093.
Standard error and t.
SE(b) = 4.093 ÷ √132.92 = 0.3551, so t = 1.971 ÷ 0.3551 = 5.55. The critical value for 10 degrees of freedom is 2.228.
Interval.
1.971 ± 2.228 × 0.3551, which is 1.18 to 2.76.

Answer

The slope is significant (t = 5.55, p = 0.0002). Each extra year of schooling goes with an extra ₹1.18 to ₹2.76 thousand of monthly income, with 95% confidence. With only twelve households the interval is wide: the plausible effect runs from 1.2 to 2.8.

Derive it

The t test on the slope, the t test on r, and why F = t²

We want to show

Under H₀: β = 0, t = b ÷ SE(b) follows a t distribution with n − 2 degrees of freedom. It equals

r n - 2 / 1 - r^{2}

, and F = SSR ÷ (SSE ÷ (n − 2)) = t².

Standardise the slope.
If the errors are normal, b is normal with mean β and variance σ² ÷ S_xx, so (b − β) ÷ (σ ÷ √S_xx) is standard normal. We do not know σ and use s instead. For normal errors, (n − 2)s² ÷ σ² follows a chi-square with n − 2 degrees of freedom and is independent of b. A standard normal divided by the square root of an independent chi-square over its degrees of freedom is a t, by definition (the t derivation):
$t = \frac{b - β}{s / S _{xx}} \sim t_{n - 2}$
Apply it to the households.
For H₀: β = 0, t = 1.971 ÷ 0.3551 = 5.55, on 10 degrees of freedom. The 5% two-sided critical value is 2.228, so the slope is significant (p = 0.00024). The 95% interval for β is b ± t × SE = 1.971 ± 2.228 × 0.3551, which is 1.18 to 2.76 thousand rupees per extra year of schooling, and it excludes zero, as it must when the test rejects.
Rewrite t in terms of r.
Let S_yy be the total sum of squares. Then b² S_xx = S_xy² ÷ S_xx = r² S_yy, since r² = S_xy² ÷ (S_xxS_yy). And SSE = (1 − r²) S_yy. So
$t^{2} = \frac{b ^{2} S _{xx}}{s ^{2}} = \frac{r ^{2} S _{y y}}{( 1 - r ^{2} ) S _{y y} / ( n - 2 )} = \frac{r ^{2} ( n - 2 )}{1 - r ^{2}}$
Take the root: t = r√(n − 2) ÷ √(1 − r²), the statistic used in the test for a correlation. With r = 0.869 and n = 12 it is 5.55, the same 5.55. Testing the slope and testing the correlation are one test.
And the F test is the square.
The regression splits the total variation into SSR (explained) and SSE (residual), exactly as in the ANOVA decomposition. With one slope, SSR has 1 degree of freedom and SSE has n − 2:
$F = \frac{S S R /1}{S S E / ( n - 2 )} = \frac{b ^{2} S _{xx}}{s ^{2}} = t^{2}$
Here SSR = 516.4, SSE = 167.6, so F = 516.4 ÷ (167.6 ÷ 10) = 30.82 = 5.55². For a single regressor the F test of the whole regression and the t test of the slope are the same test, and R² = SSR ÷ SST = 0.755.

Step 1 of 4

Decomposition

The ANOVA table, R² and F

Regression packages print the same test a second way. The total variation in income splits into the part the line explains, SSR, and the part it leaves, SSE, in exactly the proportions of the primer. Each is divided by its degrees of freedom, and their ratio is an F.

Source	Sum of squares	df	Mean square	F
Regression	516.4	1	516.4	30.82
Residual	167.6	10	16.76	30.82
Total	684	11

R² = SSR ÷ SST = 0.755: schooling accounts for about three-quarters of the variation in these twelve incomes. F = 30.82 on (1, 10) degrees of freedom is exactly t² = 5.55², and its p-value is the same 0.0002. For a single regressor, a significant slope and a significant F are the same event. With more regressors the F tests all slopes together (UGC NET 2023, UGC NET 2026) and the t tests one at a time, so they stop being the same thing.

Two facts about R² are asked often. If R² = 0, every slope estimate is zero, the fitted line is flat, and the variables explain none of the variation about the mean, but the intercept equals ȳ and is not zero (JNU 2020 and 2021). And a high R² does not make the slope significant, nor a significant slope a high R²: with a huge sample, a tiny R² can come with a very significant slope, the same lesson as Cramér’s V in Lesson 10.

Pause and predict

In a simple regression with n = 27 observations the slope's t statistic is 3.75. What is the F statistic for the whole regression?

Interactive

The line, and a new observation

Two different questions hide behind “predict the income at 12 years of schooling”. What is the average income of households with that schooling? Or what will this one new household earn? Both use ŷ₀ = 2.47 + 1.97 × 12 = 26.1, but their uncertainty differs.

Years of schooling, x₀12The sample's x values run from 5 to 16, with mean 10.4.

Fitted income at x₀ (₹ thousand)26.1

95% interval for the average23.2 to 29

95% interval for one household16.5 to 35.7

Standard error, average1.31

Standard error, one household4.3

Fitted line and 95% band for the average95% band for one new household

What it shows

At 12 years of schooling the line predicts ₹26.1 thousand. The average income of all households with that schooling is pinned down to 23.2 to 29, but a single new household can land anywhere from 16.5 to 35.7, because it carries its own noise on top of our uncertainty about the line. Slide away from the mean schooling of 10.4 years and both bands flare out, the narrow one fastest. Beyond the data, at 3 or 19 years, the line is an extrapolation and the bands say so.

Derive it

Why the interval for a new household is wider than the interval for the average

We want to show

At x₀, the variance of the fitted value is

σ^{2} (\frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}})

, and the variance of the error in predicting one new observation is that plus σ².

The fitted value at x₀.
The fitted line passes through (x̄, ȳ), so the fitted value at x₀ is ŷ₀ = ȳ + b(x₀ − x̄). It combines two estimates: the mean level ȳ and the slope b, each with its own error.
The two errors are uncorrelated.
Cov(ȳ, b) = Σ(1/n) w_i σ² = (σ² ÷ n) Σw_i = 0, because the weights sum to zero. So the variances simply add, with the slope’s error scaled by the distance (x₀ − x̄):
$Var (y^_{0}) = \frac{σ ^{2}}{n} + (x_{0} - x ˉ)^{2} \frac{σ ^{2}}{S _{xx}}$
It is smallest at x₀ = x̄ and grows with the square of the distance, which is why the band is narrowest in the middle of the data. At x₀ = 10.4 the standard error is 1.182; at x₀ = 12 it is 1.309.
A new observation brings its own error.
A household that was not in the sample has y₀ = α + βx₀ + ε₀, where ε₀ is a fresh draw, independent of the fit. The error in predicting it is y₀ − ŷ₀, and the two pieces are independent, so
$Var (y_{0} - y^_{0}) = σ^{2} + Var (y^_{0}) = σ^{2} (1 + \frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}})$
The extra 1 is the household’s own noise, and no amount of data removes it. At x₀ = 12 the standard error of prediction is 4.297 against 1.309 for the average, and the intervals are 23.2 to 29 and 16.5 to 35.7.
What it means for a forecast.
With a huge sample, 1/n and (x₀ − x̄)²/S_xx both vanish, the interval for the average collapses to a point, and the prediction interval stays at about ± t × s. Knowing the line perfectly still leaves each household free to differ from it by its own noise. That floor here is ±9.12 thousand rupees.

Step 1 of 4

Watch out for

The mistakes examiners are fishing for

Using the narrow interval for one household. The interval for the average understates the uncertainty about an individual by leaving out their own noise.
Extrapolating. Both bands widen away from x̄, and the straight-line model itself may fail out there, which no interval can show.

Assumptions

What each assumption buys

Different results need different assumptions, and exams like asking which. Take the four of DSE 2009: (1) the errors average zero, (2) they are uncorrelated with x, (3) they have constant variance, (4) they are uncorrelated with each other.

Unbiased estimates need only (1) and (2). The weights argument of the derivation used nothing else (DSE 2009 Q39).
The formula SE(b) = s ÷ √S_xx needs (3) and (4) as well. If the variance changes or errors are related, the slope is still unbiased but the standard error is wrong, and so are the t tests and intervals.
“Best” needs all four. Under (1) to (4), least squares has the smallest variance among all linear unbiased estimators: the Gauss-Markov theorem, or BLUE (DSE 2009 Q40).
Exact t and F distributions need normal errors. With non-normal errors and a large sample, the central limit theorem makes them approximately right.

Two features hold by construction, whatever the data: the residuals sum to zero (when there is an intercept) and are uncorrelated with x. Rescaling changes the slope in the obvious way: regressing 5Y on 10X multiplies the slope by 5 ÷ 10 = 0.5 (UGC NET 2023).

Where this goes next

The econometrics modules take each assumption in turn: what breaks when it fails, how to detect it, and how to repair it. Heteroskedasticity, autocorrelation, omitted variables and multicollinearity are all violations of (1) to (4) with their own tests.

Drill

Which standard error? A drill

Which standard error does each question need? · 1 of 4

A 95% interval for the average income of all households with 12 years of schooling.

Solving the opening question

The opening question, solved

IIT JAM 2023 · moderate

The 95% confidence interval for the coefficient of x is [0.26, 6.14], with estimate 3.2, in a large sample, and R² = 0.26. What is the standard error?

A large-sample interval is estimate ± 1.96 × SE.
The critical value is the normal 1.96, since the sample is large, and the interval is symmetric about the estimate. Check: (0.26 + 6.14) ÷ 2 = 3.2.
Half the width.
(6.14 − 0.26) ÷ 2 = 2.94 = 1.96 × SE.
Solve.
SE = 2.94 ÷ 1.96 = 1.5.

Answer

The standard error is 1.5. R² is a distraction: it plays no part in recovering the standard error from the interval. As a check, t = 3.2 ÷ 1.5 = 2.13, above 1.96, consistent with an interval that just excludes zero.

Revise

The lesson on one screen

Model: y = α + βx + ε, errors mean zero and variance σ². The estimates a, b are random.
Estimates: b = Sxy/Sxx = Σwy with Σw = 0, Σwx = 1. Unbiased. Var(b) = σ²/Sxx.
Noise: s² = SSE/(n − 2), because two parameters were estimated. SE(b) = s/√Sxx.
t test: t = b/SE(b) on n − 2 df. Interval b ± t·SE(b). Excludes 0 exactly when the test rejects.
Same test, three ways: t on the slope = t on r = r√(n − 2)/√(1 − r²). F = SSR/(SSE/(n − 2)) = t². R² = t²/(t² + n − 2).
Intervals at x₀: Average: ŷ₀ ± t·s·√(1/n + (x₀ − x̄)²/Sxx). One new case: add 1 inside the root. Narrowest at x̄.
Assumptions: Unbiased: zero mean and no correlation with x. Right SEs: also constant variance and no autocorrelation. Best (BLUE): all four. Exact t and F: normality.

Practice

Check yourself

0 of 14 correct so far

Compute

Back out the standard error

In a large sample the slope estimate is 16 and its 95% interval is 12.08 to 19.92. Find the standard error of the slope. (Use 1.96.)

A t statistic

A slope estimate is 0.8 with standard error 0.25. Find the t statistic for H₀: β = 0.

SE from the pieces

The residual standard deviation is s = 5 and the x values have S_xx = 100. Find the standard error of the slope.

Degrees of freedom

A simple regression has 27 observations. How many degrees of freedom does the t test on the slope have?

R² from t

In a simple regression with 11 observations the slope has t = 3. Find R².

R²

F from R²

A simple regression on 12 observations has R² = 0.6. Find the F statistic for the whole regression.

A prediction interval at the mean

n = 16, s = 4, and the t critical value for 14 degrees of freedom is 2.145. At x₀ = x̄, find the half-width of the 95% interval for one new observation.

Half-width

Standard error of a slope from the data

The x values are 2, 4, 6 and 8 and the residual standard deviation is s = 3. Find the standard error of the slope.

Quick questions

1A simple bivariate linear regression has n observations. The residual degrees of freedom are

2Which change makes the standard error of the slope smaller?

3In a simple regression the t statistic on the slope is 4. The F statistic for the regression is

4At the same x₀, the 95% interval for one new observation is, compared with the interval for the average response,

5Least-squares estimates are unbiased if the errors have zero mean and

6If the slope estimates are all zero in a regression with an intercept, then

Real questions

Past papers, by level

Foundation

Definitions and one-step calculations. Most CUET and Hyderabad questions sit here.

1 questionPractise →

Standard

Joint tables, standard errors and statement questions, up to JNU and IIT JAM level.

4 questionsPractise →

Delhi School level

This topic’s hardest questions, from the DSE entrance, are worked through in Lesson 13: Regression: what an assumption buys.

2 questionsLesson 13 →

All 5 past-paper questions for this lesson, easiest first: practise the full set →

These are questions from the TestStop bank. A free account answers 10 a day, each with its worked solution; All-Access opens every set with no daily limit.

← Lesson 11Analysis of variance and multiple comparisons Lesson 13 →The hard problems

Inference in simple regression

The model, and what is random

How precise is the slope?

The mistakes examiners are fishing for

Testing the slope and bounding it

The ANOVA table, R² and F

The line, and a new observation

The mistakes examiners are fishing for

What each assumption buys

Which standard error? A drill

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Standard

Delhi School level

Read next

Inference in simple regression

The model, and what is random

How precise is the slope?

The mistakes examiners are fishing for

Testing the slope and bounding it

The ANOVA table, R² and F

The line, and a new observation

The mistakes examiners are fishing for

What each assumption buys

Which standard error? A drill

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Standard

Delhi School level

Read next