Statistics · Complete guide
Hypothesis testing, all in one place
A hypothesis test is a trial. The claim being tested is presumed innocent until the data make that very hard to believe. Every term in the topic (Type I and Type II errors, significance level, power, p-value) is a way of describing the ways that trial can go right or wrong. Start with the 2×2, watch the trade-off on a live picture, then learn the procedure and which test to pick.
Start here
The idea: a trial for a claim
A company claims its bulbs last 1,000 hours on average. You suspect they last less. You can't test every bulb, so you test a sample and ask: if the claim were true, how surprising would this sample be? Very surprising, and you reject the claim. Not very, and you let it stand.
Null hypothesis, H₀
The claim on trial, presumed true
Always contains the equality: . It is the “nothing unusual is going on” position: no effect, no difference, no relationship.
Alternative hypothesis, H₁
What you conclude if H₀ falls
(one-tailed) or (two-tailed). Its direction decides which tail(s) of the distribution count as evidence.
| Term | Meaning |
|---|---|
| Test statistic | One number summarising how far the sample is from H₀, in standard-error units, e.g. . |
| Critical (rejection) region | The values of the test statistic that lead you to reject H₀. The rest is the acceptance (non-rejection) region. |
| Critical value | The boundary between the two regions, e.g. 1.96 for a two-tailed z test at 5%. |
| Significance level, α | The risk of rejecting H₀ when it is true that you are prepared to accept, chosen before looking at the data. Usually 5% or 1%. |
Key idea
A test never proves H₀. “Not guilty” is not “innocent”: failing to reject H₀ only means the evidence against it was not strong enough. That is why careful writers say “do not reject H₀” rather than “accept H₀”, though many exam papers use “accept” to mean the same thing.
The core
The 2×2: four possible outcomes
The truth is either H₀ or not; your decision is either to reject or not. That gives exactly four outcomes: two right, two wrong. Click each cell, and switch the analogy to see the same structure in three settings.
Decision ↓Truth →
H₀ is trueThe accused is innocent
H₀ is falseThe accused is guilty
Reject H₀Convict
Do not reject H₀Acquit
Type I error · probability α
An innocent person is convicted. The legal system is built to make this rare: “beyond reasonable doubt” is a small α.
False positive. Its probability α is the size or level of significance of the test.
The three probabilities to know by heart
Size and level: α is the maximum Type I error probability the test allows; the size is the probability it actually delivers. For the usual tests they are equal.
Remember it with the boy who cried wolf. The first time, the villagers believed an alarm when there was no wolf: a Type I error, a false positive. The second time, they ignored him when the wolf really came: a Type II error, a false negative. First the false alarm, then the miss.
Interactive
The error lab: α, β and power on one picture
A quality inspector tests using the mean of n items, with σ = 15 known. The black curve is where the sample mean lands if H₀ is true; the blue curve is where it lands given the true mean. Everything is an area under one of these two curves. Try the trade-off buttons in order.
H₀: μ = 100 vs H₁: μ > 100
Power 63.9%
α: reject a true H₀β: keep a false H₀Power: reject a false H₀
What you are looking at
The black curve is where x̄ would land if H₀ were true; the blue one is where it lands given the true mean, 106. You reject H₀ when x̄ falls above 104.93. With these settings you would wrongly reject a true H₀ 5.0% of the time and miss this particular false H₀ 36.1% of the time.
The starting point. Note how much of the blue curve still falls in the acceptance region: that is β.
α and β pull against each other
For a fixed sample size, moving the critical value to cut α makes the acceptance region wider, so β rises. You cannot shrink both by moving the line.
Only more data cuts both
A larger n shrinks the standard error σ/√n. The curves narrow and separate, so β falls while α stays where you set it.
β has no single value
β depends on the true value of μ. Close to μ₀, a false H₀ is hard to detect and β is near 1 − α; far away, it is near 0.
Procedure
The five steps, and one- vs two-tailed tests
- State the hypotheses. H₀ with the equality; H₁ as the question asks (<, > or ≠).
- Choose the significance level α, before seeing the data.
- Pick the test and compute the statistic from the sample (chapter 6 says which).
- Find the critical region from α and the tail(s), or compute the p-value.
- Decide and conclude in words. Reject H₀ if the statistic is in the critical region (equivalently, if p ≤ α), and say what that means for the original question.
One-tailed
or . All of α sits in one tail, so the critical value is closer to the centre and the test is more powerful in that direction, but blind to the other.
Two-tailed
. α is split, α/2 in each tail. Use it whenever the question doesn't tell you the direction in advance.
| α | One-tailed z | Two-tailed z |
|---|---|---|
| 10% | 1.282 | ±1.645 |
| 5% | 1.645 | ±1.960 |
| 1% | 2.326 | ±2.576 |
Worth memorising
The two-tailed 10% value equals the one-tailed 5% value (1.645), because both put 5% in a single tail. Likewise two-tailed 5% (1.96) matches one-tailed 2.5%.
Evidence
p-values
The p-value is the probability, assuming H₀ is true, of getting a test statistic at least as extreme as the one you observed. Small p means the data would be surprising under H₀. It is also the smallest α at which you would reject H₀, which is why it is sometimes called the observed or exact significance level.
The decision rule
p = 0.0359
p ≤ α: reject H₀ at the 5% level
If H₀ were true, a test statistic at least as extreme as 1.80 would turn up 3.59% of the time. That is rare enough, by the standard you set, to reject H₀. The shaded area always crosses the dashed critical line at exactly the moment p crosses α: the p-value rule and the critical-value rule are the same test.
Watch out for
What a p-value is not
- Not the probability that H₀ is true. It is computed assuming H₀ is true.
- Not the probability you made a mistake. That is α (if H₀ is true) or β (if it isn't).
- Not a measure of how big or important an effect is. With a huge sample a trivial difference can give a tiny p.
Choosing
Which test? z, t, χ² and F
| Test | Use it for | Statistic | Distribution under H₀ |
|---|---|---|---|
| z | A mean when σ is known, or any large sample (n ≥ 30); a proportion | , | N(0, 1) |
| t | A mean when σ is unknown and the sample is small (normal population); a regression coefficient | , | t with n − 1 df (n − k in regression) |
| χ² | A single variance; goodness of fit; independence in a contingency table | , | χ² with n − 1 df; k − 1 (fit); (r − 1)(c − 1) (independence) |
| F | Equality of two variances; ANOVA; joint significance of several regression coefficients | (larger variance on top) | F with (n₁ − 1, n₂ − 1) df |
The t-distribution
Like the normal, but with fatter tails to allow for estimating σ by s. Its only parameter is the degrees of freedom; as df grows it approaches N(0, 1), so for large samples z and t agree.
Non-parametric tests
Tests that don't assume a distribution for the population: the sign test, Wilcoxon, Mann–Whitney U, Kruskal–Wallis, the runs test and Spearman's rank correlation. Useful for ranks, medians and small non-normal samples.
Examples
Worked questions
Worked question 1 · two-tailed z test
A machine is set to fill packets with a mean of 50 g; σ = 10 g is known. A sample of 25 packets has mean 54 g. Test at 5% whether the machine is off target.
- Hypotheses.
- Statistic.
- Compare.|z| = 2 > 1.96. Equivalently, p = 2(1 − Φ(2)) = 0.0455 < 0.05.
Answer
Reject H₀ at 5%: the machine is off target. (At 1%, where the critical value is 2.576, you would not.)
Worked question 2 · one-tailed t test
A tutor claims her students average more than 50. A sample of 16 has mean 52 and standard deviation s = 8. Test at 5%.
- Hypotheses and test.H₀: μ = 50, H₁: μ > 50. σ unknown and n small: a t test with 15 df.
- Statistic.
- Compare.The one-tailed 5% critical value with 15 df is 1.753. Since 1 < 1.753, the statistic is not in the critical region.
Answer
Do not reject H₀: the sample doesn't give enough evidence that the average exceeds 50.
Worked question 3 · finding α, β and power from a rule
X ~ N(μ, 1). To test H₀: μ = 0 against H₁: μ = 1 with a sample of 4, you reject H₀ if . Find α, β and the power.
- Standard error.
- α: reject although μ = 0.
- β: keep H₀ although μ = 1.
Answer
α ≈ 0.055, β ≈ 0.345, power = 1 − β ≈ 0.655.
Worked question 4 · a discrete test
A coin is tossed 3 times to test H₀: p = ½ against H₁: p = ¾, where p is the probability of heads. H₀ is rejected only if all three tosses are heads. Find the size and the power of the test.
- Size (α).
- Power.
- β.
Answer
Size 1/8; power 27/64. A tiny sample gives a test that misses the biased coin more often than it catches it.
Practice
Exercises, with answers
A. Compute
Give probabilities as decimals (0.0668, not 6.68%). Answers within 0.002 are accepted.
Problem 1 · a z statistic and its p-value
σ = 12 is known. A sample of 36 has mean 53. For against , find z and the p-value.
Problem 2 · α, β and power
X ~ N(μ, 16). With a sample of 16 you test H₀: μ = 10 against H₁: μ = 12, rejecting H₀ if . Find α, β and the power.
Problem 3 · a critical value
σ = 10, n = 25. For against at 5%, above what value of do you reject H₀?
Problem 4 · a discrete test
A coin is tossed 4 times to test H₀: p = ½ against H₁: p = ⅔. H₀ is rejected only if all four tosses are heads. Find the size and the power.
B. Concept checks
One attempt each. Read the explanation even when you get it right.
1A Type I error occurs when you:
2The power of a test is:
3With the sample size fixed, lowering α from 5% to 1% will:
4The p-value is:
5A test gives p = 0.03. Then H₀ is:
6To test H₀: μ = μ₀ with a small sample from a normal population, σ unknown, you use:
7Equality of two population variances is tested with:
8Keeping α fixed, increasing the sample size:
Revise
One-page revision sheet
The 2×2
Reject a true H₀: Type I (α). Keep a false H₀: Type II (β). Reject a false H₀: power (1 − β). Keep a true H₀: 1 − α.
Trade-offs
Fixed n: lower α ⇒ higher β. Larger n ⇒ lower β at the same α. Bigger true effect ⇒ higher power.
Five steps
Hypotheses → α → test statistic → critical region or p-value → decision in words.
Critical z
One-tailed: 1.282 (10%), 1.645 (5%), 2.326 (1%). Two-tailed: 1.645, 1.96, 2.576.
p-value
P(data at least this extreme | H₀). Reject when p ≤ α. The smallest α at which H₀ would be rejected.
Which test
σ known or large n: z. σ unknown, small n: t (n − 1 df). Variance, fit, independence: χ². Two variances, ANOVA: F.
Watch out for
The mistakes examiners are fishing for
- Swapping Type I and Type II. Type I is the false alarm (rejecting a true H₀); Type II is the miss.
- Saying α = P(H₀ is true | rejected). α is conditional the other way round: P(reject | H₀ true).
- Thinking power = 1 − α. 1 − α is the chance of keeping a true H₀. Power is 1 − β.
- Using 1.96 for a one-tailed 5% test. One tail at 5% is 1.645.
- Putting the inequality in H₀. H₀ holds the equality; the direction goes in H₁.
- Using z with a small sample and unknown σ. That calls for t.
Try it on real questions
Fourteen past-paper questions on this
From UGC NET, IIT JAM, JNU, CUET-PG and the University of Hyderabad: Type I and II errors, power, significance levels, p-values, the order of the steps, choosing the statistic, and three numeric tests.
These are questions from the TestStop bank. A free account answers 10 a day, each with its worked solution; All-Access opens every set with no daily limit.
Read next
All resources →StatisticsInference 7: hypothesis testing, errors, power and p-valuesNull and alternative hypotheses, test statistics and rejection regions, type I and type II errors, size and power, one and two tails, and what a p-value does not mean.StatisticsInference 8: the tests you will be askedz and t tests for means and proportions, two-sample and paired tests, the F test for two variances, chi-square tests for fit and independence, and a one-way ANOVA.StatisticsBasic statistics for economics students, from zero to exam levelA module in ten lessons: types of data, the three averages, the geometric and harmonic means, variance and standard deviation, then shape, correlation, regression and index numbers, with a short add-on on growth rates, on one running example, with graded past papers and the Delhi School’s hard problems.