Statistical inference · Lesson 11 of 13

Analysis of variance and multiple comparisons

Two means are compared with a t test. Three or more need something else, and the reason is subtler than it looks. This lesson explains why running all the pairwise tests fails, how analysis of variance replaces them with one test that asks a different question, why that test is an F, and what to do once it comes out significant. The ANOVA table of Lesson 8 is derived here from the ground up.

← Statistical inference moduleLesson 11 of 13 · about 50 minutes

A question to hold on to

A researcher compares the mean wages of five groups. In truth all five have exactly the same mean. She runs all ten pairwise t tests at the 5% level. Roughly how often will she find at least one “significant difference”, and does it matter that the groups are truly identical?

Reasoning, inventedSolved at the end ↓

New here?What is Ashanagar?The invented district behind this page's examples: 60 households, with incomes you can inspect.

Ashanagar is an invented district. The statistics lessons use it for every worked example, so you meet the same households again as the ideas build: first as a list of numbers, then as a spread, then as a population to sample from.

It has 60 households, each with a monthly income in rupees. Unless a page says otherwise, those 60 are the whole population, and a “sample” means a few of them drawn at random. Where a page needs two variables, it uses a smaller set of 12 households with the head’s years of schooling beside income.

The figures are illustrative, made up once and fixed. They are not official statistics.

Households: 60
Mean income: ₹28,783 a month
Median income: ₹25,000 a month
Standard deviation: ₹13,366
Lowest and highest: ₹10,500 and ₹72,000

Meet the full data set →

Interactive

Why not just run every t test?

A t test at the 5% level raises a false alarm one time in twenty when there is nothing to find. Run two such tests on a pair of independent comparisons and the chance of at least one false alarm is 1 − 0.95² = 0.0975 (DSE 2015 asks exactly this). With k groups there are k(k − 1) ÷ 2 pairs, and the chances of at least one false alarm climb quickly: three pairs, ten pairs, forty-five. The more comparisons you run, the more likely something looks significant by luck.

Number of groups, k5Observations per group, n8

Pairs to compare, k(k − 1) ÷ 210

Formula 1 − 0.95^pairs40.1%

Simulated, uncorrected28.6%

What it shows

Every group in every experiment has the same true mean, so any declared difference is a false alarm. The single F test keeps its promise, at about 5% whatever k is. Running every pairwise t test at 5% does not: with 5 groups there are 10 pairs, and 29% of experiments find a “difference”. The textbook formula 1 − 0.95¹⁰ = 40% is only an upper guide, because the pairs share groups and are not independent. Dividing the level by the number of pairs brings the false-alarm rate back under 5%.

Note two things in the lab. First, the plain formula 1 − 0.95^m is not the real rate: pairs that share a group are positively linked, so for five groups the true rate is about 28% and not 40%. It is a guide, and for independent comparisons it is exact. Second, there are two cures. A single test of all the means at once, the F test of ANOVA, holds its error at 5% by construction. Or keep the pairwise tests and tighten each one to 5% ÷ m, the Bonferroni correction. This lesson develops both.

The table

Splitting the variation: the ANOVA table

One-way ANOVA asks one question: are all k population means equal? Its idea is to split the total variation in the data, SST, into the part explained by which group an observation belongs to, SSB, and what is left, SSW. That the pieces add up exactly, SST = SSB + SSW, is proved in the ANOVA derivation of Lesson 8. Dividing each by its degrees of freedom gives mean squares, and the ratio is the test statistic.

One-way ANOVA

F = \frac{MSB}{MSW} = \frac{SSB / ( k - 1 )}{SSW / ( N - k )}, η^{2} = \frac{SSB}{SST}

k groups, N observations in all. Reject H₀: all means equal when F exceeds the upper 5% point of F(k − 1, N − k).

Move the Factory group's wages+₹0Spread of wages inside each group× 1Scales every worker's distance from their own group's mean.

grand mean ₹461group means

Source	Sum of squares	df	Mean square	F	p-value
Between groups	51,458	2	25,729	25.17	< 0.0001
Within groups	21,466	21	1,022	25.17	< 0.0001
Total	72,924	23

The group means differ by more than the scatter inside the groups would lead us to expect: F = 25.2 on (2, 21) degrees of freedom, so we reject the hypothesis that all three mean wages are equal. ANOVA does not say which group differs. It says at least one does. Widen the spread inside the groups to watch F fall without moving a single group mean.

Source	Sum of squares	df	Mean square	F
Between sectors	51,458	2	25,729	25.17
Within sectors	21,466	21	1,022	25.17
Total	72,924	23

The wages of three sectors, eight workers each, give F = 25.17 on (2, 21) degrees of freedom. The 5% cut-off is 3.47, and p = 0.000003, so the hypothesis of equal mean wages is rejected. The share of the variation tied to sector is η² = SSB ÷ SST = 0.71, the same idea as R² in a regression: ANOVA is a regression of wages on sector dummies, and its F is the F of that regression.

Why it works

Why the ratio is an F

Why divide the between variation by the within variation? Because the two mean squares are two different estimates of one number, σ², that agree when the means are equal and disagree when they are not. The within estimate is always right. The between estimate is right only when there is nothing to find.

Derive it

What the two mean squares estimate, and why their ratio is an F

We want to show

For k groups of n from normal populations with a common variance σ², E(MSW) = σ² always, and

E (MSB) = σ^{2} + \frac{n \sum _{g} ( μ _{g} - μ ˉ ) ^{2}}{k - 1}

. When all the means are equal, MSB ÷ MSW follows an F distribution with (k − 1, N − k) degrees of freedom.

The within mean square is the pooled variance.
MSW = SSW ÷ (N − k) adds up each group’s sum of squared deviations from its own mean and divides by the total degrees of freedom. It is the pooled variance of the two-sample lesson extended to k groups. Each group’s own variance s_g² estimates σ² whether or not the group means differ, so MSW is an unbiased estimate of σ² in every case.
When the means are equal, the between mean square estimates σ² too.
If every group has the same mean, the k sample means are k draws from the same distribution, each with variance σ² ÷ n. The sample variance of the k means therefore estimates σ² ÷ n, and n times it estimates σ². That is exactly MSB = n Σ(ȳ_g − ȳ)² ÷ (k − 1).
So when H₀ is true both mean squares estimate the same σ², and their ratio hovers near 1.
When the means differ, only the between mean square grows.
If the true means are μ_g, the sample means vary by sampling noise and by the real gaps, and the second source adds to MSB:
$E (MSB) = σ^{2} + \frac{n \sum _{g} ( μ _{g} - μ ˉ ) ^{2}}{k - 1}$
MSW is untouched. With three groups of 8, true means −10, 0 and 10 and σ = 10, E(MSB) = 100 + 8 × 200 ÷ 2 = 900, while E(MSW) is still 100. The ratio is about 9, far above 1. This is why a large F points to real differences between groups.
The ratio is an F.
Under H₀, SSB ÷ σ² is a chi-square with k − 1 degrees of freedom and SSW ÷ σ² is an independent chi-square with N − k. A ratio of two independent chi-squares, each over its degrees of freedom, is an F (the F derivation), so
$F = \frac{MSB}{MSW} = \frac{SSB / ( k - 1 )}{SSW / ( N - k )} \sim F_{k - 1, N - k}$
Drag the true means apart in the lab and watch MSB leave 100 behind while MSW stays put.

Step 1 of 4

Derive it

With two groups, ANOVA is the pooled t test: F = t²

We want to show

For k = 2,

F = \frac{( y ˉ _{1} - y ˉ _{2} ) ^{2}}{s _{p}^{2} ( 1/ n _{1} + 1/ n _{2} )} = t^{2}

, with F on (1, n₁ + n₂ − 2) degrees of freedom.

Find the between sum of squares for two groups.
Let d = ȳ₁ − ȳ₂ and N = n₁ + n₂. The grand mean is (n₁ȳ₁ + n₂ȳ₂) ÷ N, so ȳ₁ − ȳ = n₂d ÷ N and ȳ₂ − ȳ = −n₁d ÷ N. Then
$SSB = n_{1} \frac{n _{2}^{2} d ^{2}}{N ^{2}} + n_{2} \frac{n _{1}^{2} d ^{2}}{N ^{2}} = \frac{n _{1} n _{2} ( n _{1} + n _{2} )}{N ^{2}} d^{2} = \frac{n _{1} n _{2}}{N} d^{2}$
Divide.
With k = 2 the between degrees of freedom are 1, so MSB = SSB. MSW is the pooled variance s_p² on N − 2 degrees of freedom. And n₁n₂ ÷ N = 1 ÷ (1/n₁ + 1/n₂), so
$F = \frac{n _{1} n _{2} d ^{2} / N}{s _{p}^{2}} = \frac{d ^{2}}{s _{p}^{2} ( 1/ n _{1} + 1/ n _{2} )} = t^{2}$
Check it on two sectors.
Farm and Construction workers, eight in each. The pooled t statistic is −3.668, so t² = 13.451, and the ANOVA on the same sixteen wages gives F = 13.451 on (1, 14) degrees of freedom. The F table with 1 numerator degree of freedom is the t table squared, which is why a two-sided t test and the F test always agree.

Step 1 of 3

Pause and predict

Three groups of equal size have the same spread inside each group. The true group means are pulled further apart, but the spread inside the groups does not change. The F ratio's numerator MSB and denominator MSW

Post-hoc

After a significant F

A significant F says that at least one mean differs. It does not say which. To find out, compare the groups in pairs, but now with a correction, because you are back in the multiple-comparisons problem of Chapter 1. Use the within mean square from the ANOVA, not each pair’s own, since MSW pools all the information about σ². The standard error of the gap between groups g and h is √(MSW × (1/n_g + 1/n_h)), with N − k degrees of freedom.

Bonferroni. Test each of the m pairs at level α ÷ m. Simple, safe, slightly conservative.
Tukey’s honestly significant difference. Uses the distribution of the largest gap among k means, the studentised range, and is the standard choice when all pairs are compared. It is a little more powerful than Bonferroni.
Scheffé. Protects every possible comparison, including combinations like “farm against the average of the other two”, at the cost of power.

Ashanagar · three sectors

After the significant F, which sectors differ in mean daily wage? Means: Farm ₹409.4, Construction ₹453.1, Factory ₹521.9, eight workers each, MSW = 1,022.2 on 21 degrees of freedom.

Standard error of any gap.
√(1,022.2 × (1/8 + 1/8)) = 15.99.
Three pairs, so Bonferroni uses 5% ÷ 3.
The two-sided critical value is the t point with 0.05 ÷ 6 in each tail on 21 degrees of freedom: 2.601, instead of the uncorrected 2.08.
The three comparisons.
Farm and Construction: 43.75 ÷ 15.99 = 2.74. Construction and Factory: 68.75 ÷ 15.99 = 4.3. Farm and Factory: 112.5 ÷ 15.99 = 7.04.

Answer

All three exceed 2.6, so every pair of sectors differs, even after the correction. The closest call is Farm against Construction: its uncorrected p is 0.012 and its Bonferroni-adjusted p is 0.037, still below 0.05. Factory pays the most: about ₹113 a day more than Farm.

Watch out for

The mistakes examiners are fishing for

Reading a significant F as “all means differ”. One mean may differ from the other four, which agree.
Skipping the correction in the follow-up. Uncorrected pairwise tests bring the false alarms of Chapter 1 straight back.
Running ANOVA only if the pairwise tests look promising. Choose the test before looking at the data. Looking first and testing the biggest gap is the same problem in disguise.

Conditions

Assumptions, and what to do when they fail

The exact F distribution needs three things: the observations are independent, each group is roughly normal, and the groups share one variance. Of these, independence matters most and cannot be repaired afterwards. Moderate departures from normality hurt little, especially with equal group sizes. Unequal variances matter more when the group sizes differ.

Unequal variances: use Welch’s ANOVA, which does not pool the variances.
Clearly non-normal data, small groups: use the Kruskal-Wallis test, the rank-based replacement for one-way ANOVA (Lesson 8). It compares the groups through their ranks.
Two factors, such as sector and gender together: two-way ANOVA splits SST into two main effects and an interaction. The idea of this lesson carries over unchanged.

ANOVA is regression

Write the group means as coefficients on 0/1 group dummies and ANOVA’s F is the regression F of Lesson 12, with η² as R². With two groups the F is the square of the t statistic. These are one idea at three levels of generality.

Drill

Which method? A drill

Which method fits each situation? · 1 of 5

Mean wages of three sectors are to be compared in one test.

Solving the opening question

The opening question, solved

Reasoning · the puzzle

Five groups with identical true means, all ten pairwise t tests at 5%. How often is at least one found significant?

If the ten tests were independent.
The chance that none rejects is 0.95¹⁰ = 0.599, so the chance of at least one false alarm is 0.401, about 40%.
They are not independent.
Two tests that share a group share part of the noise, so they tend to agree. The simulation, 3,000 experiments of five groups of eight, finds at least one false alarm in 28% of them.
Either way it is far above 5%.
The F test on the same data raises a false alarm 4.5% of the time, and Bonferroni at 0.5% per test, 4%.

Answer

About 40% by the independence formula, about 28% in reality. It matters a great deal that the groups are truly identical: every such finding is false, and a researcher who reports the one “significant” pair has found noise. One F test, or corrected pairwise tests, would have protected her.

Revise

The lesson on one screen

Why not pairwise t tests: k groups have k(k − 1)/2 pairs, and each at 5% adds false alarms. For independent tests the chance of at least one is 1 − 0.95^m. For shared groups it is lower but still far above 5%.
ANOVA table: SST = SSB + SSW. MSB = SSB/(k − 1), MSW = SSW/(N − k), F = MSB/MSW on (k − 1, N − k) df. η² = SSB/SST.
Why an F: E(MSW) = σ² always. E(MSB) = σ² + nΣ(μ − μ̄)²/(k − 1). Equal means: the ratio is near 1, and an F.
Two groups: F = t² for the pooled t test, on (1, N − 2) df.
After a significant F: Pairwise gaps with SE = √(MSW(1/n_g + 1/n_h)), N − k df, level corrected: Bonferroni α/m, Tukey, Scheffé. A significant F does not say which mean differs.
Conditions: Independent observations, roughly normal groups, equal variances. Fix with Welch's ANOVA or Kruskal-Wallis.

Practice

Check yourself

0 of 14 correct so far

Compute

An F ratio

Three groups of 5 observations have SSB = 60 and SSW = 96. Find F. The 5% critical value of F(2, 12) is 3.885.

Degrees of freedom

Four groups of 7 observations each. How many degrees of freedom does the denominator of the F ratio have?

Eta squared

SSB = 60 and SSW = 96. What share of the total variation is explained by group membership?

η²

How many pairs

Six groups are to be compared in pairs. How many pairwise comparisons are there?

Pairs

Bonferroni

Ten pairwise comparisons are made and the family-wise level is to be 5%. At what level is each individual test run? Give your answer as a decimal.

Level

The chance of a false alarm

Three independent tests, each at the 5% level, are run on true nulls. What is the chance that at least one rejects?

Probability

Two groups

A pooled two-sample t test on two groups gives t = 3. What F statistic does one-way ANOVA on the same data give?

A standard error after ANOVA

An ANOVA gives MSW = 40, and two of the groups have 10 observations each. Find the standard error of the difference of their means.

Quick questions

1One-way ANOVA tests the null hypothesis that

2With k groups and N observations in all, the F statistic has degrees of freedom

3Many pairwise t tests at 5% are run in place of ANOVA. Compared with one F test, the chance of at least one false alarm is

4In ANOVA, the within mean square MSW is an unbiased estimate of σ²

5For two groups, one-way ANOVA gives an F statistic equal to

6The rank-based test that replaces one-way ANOVA for clearly non-normal data is

This lesson has no past-paper set because none of the questions in our bank asks about ANOVA directly. The tests-in-practice lesson has its non-parametric alternative, and the Delhi School lesson has the false-alarm arithmetic for two tests run together. The exercise answers below were checked against the model file.

For the F table: the 5% point of F(2, 12) is 3.885.

← Lesson 10Chi-square tests: goodness of fit and independence Lesson 12 →Inference in simple regression

Analysis of variance and multiple comparisons

Why not just run every t test?

Splitting the variation: the ANOVA table

Why the ratio is an F

What the two mean squares estimate, and why their ratio is an F

With two groups, ANOVA is the pooled t test: F = t²

After a significant F

The mistakes examiners are fishing for

Assumptions, and what to do when they fail

Which method? A drill

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Read next

Analysis of variance and multiple comparisons

Why not just run every t test?

Splitting the variation: the ANOVA table

Why the ratio is an F

What the two mean squares estimate, and why their ratio is an F

With two groups, ANOVA is the pooled t test: F = t²

After a significant F

The mistakes examiners are fishing for

Assumptions, and what to do when they fail

Which method? A drill

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Read next