Statistical inference · Lesson 7 of 13

Hypothesis testing: errors, power and p-values

An interval says what values are plausible. A test answers a yes-or-no question: do the data contradict a claim? The logic is that of a trial, with the same two ways to go wrong, and the same temptation to read the verdict as more than it is. This lesson builds the machinery, the two errors, size and power, critical values and p-values, with a picture for each, and ends with a test built from scratch on a single observation.

← Statistical inference moduleLesson 7 of 13 · about 50 minutes

A question to hold on to

You have one observation x from one of two densities on 0 ≤ x ≤ 1: H₀ has f(x) = 2x and H₁ has f(x) = 2 − 2x. Take a test of the form “reject H₀ if x < k” with Type I error α = 0.09. Find k, and find β, the probability of a Type II error.

DSE 2012Solved at the end ↓

The idea

A trial, not a measurement

A hypothesis test starts with a claim to be challenged, the null hypothesis H₀: the mean income is ₹30,000, the scheme had no effect, the coin is fair. The alternative H₁ is what you would believe if the null failed. Like a court that presumes innocence, the test keeps the null unless the evidence against it is strong. Five steps, in order (UGC NET 2026):

Set up the null and the alternative hypothesis.
Select the level of significance, α.
Select the test statistic.
Compute it from the sample and compare it with the critical value, or compute its p-value.
Draw the conclusion.

A hypothesis is simple if it fixes every parameter (μ = μ₀ and σ² = σ₀²), and composite if it leaves some free (μ = μ₀ with σ² unknown), a distinction a matching question tests (UGC NET 2025). The choice between null and alternative is not symmetric. The null is the claim that is kept unless the data say otherwise, and the burden of proof sits on the alternative.

Interactive

Two ways to be wrong

A test ends in reject or do not reject, and the truth is either H₀ or H₁. That gives four cases and two of them are mistakes.

	H₀ is true	H₀ is false
Reject H₀	Type I error, a false alarm. Probability α, the size of the test.	Correct. Probability 1 − β, the power.
Do not reject H₀	Correct. Probability 1 − α.	Type II error, a miss. Probability β.

Type I error is rejecting a true null (Hyderabad 2011, 2018). Type II error is accepting, or more carefully failing to reject, a false one (Hyderabad 2017, JNU 2021). The level of significance is the maximum probability of Type I error you are willing to risk (CUET 2018, Hyderabad 2020), and it is the number the significance level of an estimation question refers to (Hyderabad 2019). Power is the probability of rejecting H₀ when H₁ is true, which is one minus the probability of a Type II error (JNU 2016, JAM 2022, UGC NET 2023, JAM 2025). The area of non-rejection is one minus the probability of rejecting a true null (JAM 2025).

The two errors pull against each other. Move the critical value outwards and false alarms fall, but real effects are missed more often. The lab draws the two distributions of the test statistic, one when H₀ is true and one when it is not, and lets you move everything that matters: the size of the true effect, the sample size and α.

True effect, d (in standard deviations)0.5Sample size, n25

size α: reject a true H₀β: fail to reject a false H₀

Size α (Type I)0.05

β (Type II)0.196

Power, 1 − β0.804

n for 80% power at this d25

With a shift of 2.5 standard errors the test catches the effect 80% of the time: it clears the usual 80% power benchmark.

Four things to try, each of which is an exam fact:

Lower α from 5% to 1% and watch the gold β area grow. You buy fewer false alarms with more misses.
Raise n. The blue curve moves away from the grey one without changing either shape, so both errors can fall at once. A bigger sample is the only free lunch.
Shrink the true effect. When the effect is tiny, the two curves overlap almost completely and power sits near α. The test cannot see what is not there to be seen.
Switch to two-sided. The same α is split between two tails, so each is smaller and the one-sided test is more powerful when the effect is in the direction you guessed.

Pause and predict

A doctor testing a diagnostic tool for a rare disease wants to minimise the chance of telling a sick patient she is healthy. With the null being that the patient is healthy, which error must she keep small?

The shape of the rejection region

One tail or two, and the critical value

The alternative decides where the rejection region sits. H₀: π = 0.16 against H₁: π > 0.16 is a one-tailed test with the rejection area in the right tail (JNU 2012). H₀: β₂ = 0 against H₁: β₂ ≠ 0 is a two-tailed test (JNU 2021): large values of either sign count against the null, so α is split between two tails. The critical value is the cut-off beyond which the statistic is too extreme for H₀.

Size α	One-sided critical z	Two-sided critical z
10%	1.282	1.645
5%	1.645	1.960
1%	2.326	2.576

A statistic between 1.96 and 2.58 in a two-sided test is significant at 5% but not at 1%, which is how a question that asks the decision about H₀ is settled: reject it at the 5% level (JNU 2021). A common slip is to read a rejection at 5% as a rejection at 1%. The stricter the level, the harder it is to reject.

Interactive

The p-value

Instead of fixing α in advance and asking whether the statistic crosses a line, report how surprising the data are. The p-value is the probability, if H₀ were true, of a test statistic at least as extreme as the one observed. A small p-value means the data would be rare under the null. The decision rule is the same either way: reject when the p-value is no larger than α (Hyderabad 2021), which is exactly when the statistic is beyond the critical value. The p-value depends on the data (DSE 2021): a different sample would give a different one.

p-value of a z statistic

upper-tailed: p = 1 - Φ (z_{c}), two-sided: p = 2 (1 - Φ (∣ z_{c} ∣))

Φ is the standard normal distribution function. For a lower-tail alternative use Φ(z).

For H₁: B > B₀ with the variance given, the p-value is 1 − Φ(z_c), the area to the right of the computed statistic (UGC NET 2023). Slide z in the lab and read the area. Then switch to two-sided.

Observed z statistic2The coin that shows 60 heads in 100 tosses has z = 2.

p-value: area at least as extreme as z (under H₀)critical value of a size α test

p-value0.0228

Critical value1.645

Size α0.05

DecisionReject H₀

The p-value, 0.0228, is below α = 0.05, and z has crossed the critical value: the same verdict, said two ways. A p-value is the chance of data this extreme if H₀ were true, not the chance that H₀ is true.

GATE 2025 · moderate

A coin is tossed 100 times and shows heads 60 times. Test H₀: μ = 0.5 against H₁: μ > 0.5, using the central limit theorem. Find the p-value. (Given P(Z ≤ 2) = 0.977.)

Standard error under H₀.
√(0.5 × 0.5 ÷ 100) = 0.05.
z statistic.
(0.60 − 0.50) ÷ 0.05 = 2.
Upper tail.
p = P(Z > 2) = 1 − 0.977 = 0.023 (the exact value is 0.0228).

Answer

p ≈ 0.023. Below 0.05, so the test rejects fairness at the 5% level, and above 0.01, so not at 1%. It is the 2 in the lab’s slider.

What a p-value is not

It is not the probability that H₀ is true. It is a probability about the data, computed assuming H₀. Whether H₀ is true is not a probability in this framework.
It is not the size of the effect. A huge sample gives a tiny p-value for a trivial difference. The p-value and significance alone are not enough to assess the relation between two variables (JNU 2021).
Failing to reject is not proof that H₀ is true. The data may be too thin to detect a real effect, which is low power.
It is not the Type I error rate of the test. That is α, chosen before the data are seen. The p-value is the smallest α at which this sample would have led to rejection.

Intervals and tests, again

Reject H₀: μ = μ₀ at the 5% level (two-sided) exactly when μ₀ lies outside the 95% confidence interval. The test gives a yes or no for one value, and the interval gives the answer for every value at once.

Planning

Power, and how many you need

Power is not something you compute after the event to excuse a null result. It is a planning tool: choose the effect size worth detecting, set α and the power you want, usually 80%, and solve for n. For a z test with standardised effect d = (μ₁ − μ₀) ÷ σ:

Sample size for a given power

n = (\frac{z _{1 - α / tails} + z _{power}}{d})^{2}

z at 1 − α/2 for a two-sided test, 1 − α for a one-sided one. z at 0.80 power is 0.8416; at 0.90, 1.2816.

To detect half a standard deviation (d = 0.5) with a two-sided 5% test and 80% power: (1.96 + 0.8416)² ÷ 0.25 = 31.4, so 32 observations. Halve the effect to d = 0.25 and the requirement quadruples to 126, which is the same square law as the standard error and the same reason small studies find only big effects. Compare with the lab’s last readout, “n for 80% power”.

Derive it

Deriving the power of a z test, and the sample size it needs

We want to show

For H₀: μ = μ₀ against H₁: μ = μ₁ > μ₀ with known σ, a size-α test has

power = Φ (δ - z_{α})

with

δ = \frac{( μ _{1} - μ _{0} ) n}{σ}

, and needs

n = (\frac{( z _{α} + z _{β} ) σ}{μ _{1} - μ _{0}})^{2}

to reach power 1 − β.

Place the cut-off using the null.
The test rejects H₀ when the sample mean exceeds a critical value c. We choose c so that, if H₀ is true, this happens with probability α. Under H₀ the sample mean is normal with mean μ₀ and standard error σ ÷ √n, so
$P (X ˉ > c ∣ μ_{0}) = α ⟹ c = μ_{0} + z_{α} \frac{σ}{n}$
With μ₀ = 50, σ = 10, n = 25 and α = 5%: the standard error is 2 and z_α = 1.645, so c = 50 + 1.645 × 2 = 53.29.
Measure the cut-off from the truth instead.
Power is the probability of rejecting when the true mean is μ₁. Now the sample mean is normal around μ₁, with the same standard error. Standardise using μ₁:
$power = P (X ˉ > c ∣ μ_{1}) = P (Z > \frac{c - μ _{1}}{σ / n})$
Substitute c from step 1 and the awkward-looking fraction simplifies:
$\frac{c - μ _{1}}{σ / n} = \frac{μ _{0} + z _{α} σ / n - μ _{1}}{σ / n} = z_{α} - δ, δ = \frac{( μ _{1} - μ _{0} ) n}{σ}$
So power = P(Z > z_α − δ) = Φ(δ − z_α), using the symmetry of the normal curve. Here δ = 4 ÷ 2 = 2, so power = Φ(2 − 1.645) = Φ(0.355) = 0.639.
Read the formula.
Power depends on the data only through δ, the true shift measured in standard errors, and on α only through z_α. A bigger effect μ₁ − μ₀, a bigger n (through √n) and a smaller σ all raise δ and so raise power. A smaller α raises z_α and lowers power: the cost of being more careful about false alarms is more misses. Move the sliders in the picture and watch the blue area to the right of c.
Turn it round to choose the sample size.
Ask for power 1 − β. Power = Φ(δ − z_α) = 1 − β means δ − z_α = z_β, where z_β is the point with area β beyond it. So δ = z_α + z_β, and since δ = (μ₁ − μ₀)√n ÷ σ,
$n = (\frac{( z _{α} + z _{β} ) σ}{μ _{1} - μ _{0}})^{2}$
For 80% power at α = 5%, z_α + z_β = 2.486 (1.645 and 0.842 before rounding the last digit), so n = (2.486 × 10 ÷ 4)² = 38.6, which rounds up to 39 households. Always round up: 38 would fall just short.

Step 1 of 4

Building a test

A test from first principles

Where does a rejection region come from? Not always from a formula in a table. With one observation and two fully specified densities you can build the best test directly, and examiners like to see it done. The Neyman-Pearson lemma says the most powerful test of a simple null against a simple alternative rejects where the likelihood ratio f₁(x) ÷ f₀(x) is large. In the opening question that ratio is (2 − 2x) ÷ 2x, which falls as x rises, so evidence for H₁ is a small x, and the rule “reject H₀ if x < k” is the right form.

A cruder example of a size calculation: X is uniform on [0, b], and we test H₀: b = 2 against b ≠ 2 with the critical region x ≤ 0.1 or x ≥ 1.9 from a single draw. Under H₀ the density is 1/2, so the probability of the region is 0.1 × ½ + 0.1 × ½ = 0.1, the Type I error (IIT JAM 2021). Size is just a probability computed under H₀.

Many tests at once

Each test has its own false-alarm rate, and they add up. If two independent t tests are each run at 5%, and the joint null is rejected when either one rejects, the Type I error is 1 − 0.95² = 0.0975, more than 5% (DSE 2015). With twenty tests, the chance of at least one false rejection is 64%. This is why a significant result found after trying many comparisons deserves suspicion.

Watch out for

The mistakes examiners are fishing for

Type I and Type II are easy to swap. Type I rejects a true null. Type II keeps a false one. Power is 1 − β, not 1 − α.
p-value is not P(H₀ is true). And a large p-value does not prove H₀.
Cutting α alone lowers power. Fewer false alarms means more misses, unless n grows.
Rejecting at 5% does not mean rejecting at 1%. The reverse is true: a 1% rejection implies a 5% one.
Size is computed under H₀, β and power under H₁. Use the right density for each.

Solving the opening question

The opening question, solved

DSE 2012 · hard

H₀: f(x) = 2x and H₁: f(x) = 2 − 2x on [0, 1]. Test: reject H₀ if x < k, with α = 0.09. Find k and β.

Size is computed under H₀.
α = P(x < k | H₀) = ∫₀^k 2x dx = k². Setting k² = 0.09 gives k = 0.3.
β is computed under H₁.
β = P(do not reject | H₁) = P(x ≥ 0.3 | H₁) = ∫ from 0.3 to 1 of (2 − 2x) dx = [2x − x²] from 0.3 to 1 = 1 − (0.6 − 0.09) = 0.49.
Check the other options.
k = 0.5 would give α = 0.25, not 0.09, so options A and B fail on the first line. Option C has k = 0.3 but β = 0.16, which matches neither area under H₁: 0.49 lies above 0.3 and 0.51 below it.

Answer

k = 0.3 and β = 0.49: option D. Power is 1 − 0.49 = 0.51. The test is weak because one observation carries little information: with α = 0.09, it misses H₁ nearly half the time.

Next, Lesson 8 puts these ideas to work on the tests you will actually be asked to run.

Revise

The lesson on one screen

Steps: Set H₀ and H₁, choose α, choose the statistic, compute it, conclude. Simple: all parameters fixed. Composite: some free.
Errors: Type I: reject a true H₀, probability α (the size). Type II: keep a false H₀, probability β. Power = 1 − β = P(reject H₀ | H₁ true).
Trade-off: Lowering α raises β unless n rises. Larger n, larger effect, or a one-sided test each raise power.
Tails: H₁ with > or < gives one tail. H₁ with ≠ gives two, with α split. Critical z: 1.645, 1.96, 2.576 two-sided at 10%, 5%, 1%.
p-value: P(data at least this extreme | H₀). Reject when p ≤ α. Upper-tailed p = 1 − Φ(z). Not P(H₀ true), not the effect size.
Sample size: n = ((z(α) + z(power)) ÷ d)². Halve d, quadruple n.
Building a test: Neyman-Pearson: reject where f₁/f₀ is large. Compute α under H₀, β under H₁. Many tests inflate the false-alarm rate: 1 − 0.95² = 0.0975.

Practice

Check yourself

0 of 13 correct so far

Compute

A z statistic

H₀: μ = 50, σ = 10 is known, and a sample of 25 has mean 52. Find the z statistic.

An upper-tailed p-value

A z statistic of 2.00 for H₁: μ > μ₀. Find the p-value.

A two-sided p-value

A z statistic of 2.10 for H₁: μ ≠ μ₀. Find the p-value.

The size of a rule

A test rejects H₀ whenever |z| > 1.645. What is its size (Type I error probability)?

Power of a one-sided test

H₀: μ = 0 against H₁: μ > 0, σ = 10, n = 25, α = 0.05. The true mean is 3. Find the power. (Reject if x̄ > 1.645 × 2 = 3.29.)

Power

Sample size for power

How many observations give 80% power for a two-sided 5% test against an effect of d = 0.5 standard deviations? Use z = 1.96 and 0.8416.

A one-observation test

Use the densities of the opening question (H₀: 2x, H₁: 2 − 2x on [0, 1]) and the rule “reject H₀ if x < k” with α = 0.16. Find β.

Two tests at once

Three independent tests are each run at the 5% level, and H₀ is rejected if any of them rejects. If all three nulls are true, what is the probability of at least one false rejection?

Quick questions

1A p-value of 0.03 means that

2Other things equal, lowering α from 5% to 1% will

3A test fails to reject H₀. This means

4A Type I error is

5H₀: μ = 100 against H₁: μ > 100 is tested at the 5% level with a z statistic. The rejection region is

Real questions

Past papers, by level

Foundation

Definitions and one-step calculations. Most CUET and Hyderabad questions sit here.

13 questionsPractise →

Standard

Joint tables, standard errors and statement questions, up to JNU and IIT JAM level.

9 questionsPractise →

Delhi School level

This topic’s hardest questions, from the DSE entrance, are worked through in Lesson 13: Errors, power and the test of a single observation.

3 questionsLesson 13 →

All 22 past-paper questions for this lesson, easiest first: practise the full set →

These are questions from the TestStop bank. A free account answers 10 a day, each with its worked solution; All-Access opens every set with no daily limit.

← Lesson 6Confidence intervals Lesson 8 →The tests you will be asked

Hypothesis testing: errors, power and p-values

A trial, not a measurement

Two ways to be wrong

One tail or two, and the critical value

The p-value

What a p-value is not

Power, and how many you need

A test from first principles

Many tests at once

The mistakes examiners are fishing for

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Standard

Delhi School level

Read next

Hypothesis testing: errors, power and p-values

A trial, not a measurement

Two ways to be wrong

One tail or two, and the critical value

The p-value

What a p-value is not

Power, and how many you need

A test from first principles

Many tests at once

The mistakes examiners are fishing for

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Standard

Delhi School level

Read next