Statistical inference · Lesson 4 of 13

Estimators: bias, variance, consistency, sufficiency

We already use the sample mean to estimate a population mean without asking whether it is any good. This lesson asks. Several different rules can be applied to the same sample, and they can differ in whether they are right on average, how much they scatter, whether they settle down as the sample grows, and whether they waste information. Those four questions give the vocabulary of every estimation question in the exams.

← Statistical inference moduleLesson 4 of 13 · about 45 minutes

A question to hold on to

S² = Σ(xᵢ − x̄)² ÷ (n − 1) is an unbiased and consistent estimator of the population variance. What can you conclude about S = √S² as an estimator of the population standard deviation? Is it unbiased and consistent, biased and consistent, unbiased and inconsistent, or biased and inconsistent?

GATE 2021Solved at the end ↓

New here?What is Ashanagar?The invented district behind this page's examples: 60 households, with incomes you can inspect.

Ashanagar is an invented district. The statistics lessons use it for every worked example, so you meet the same households again as the ideas build: first as a list of numbers, then as a spread, then as a population to sample from.

It has 60 households, each with a monthly income in rupees. Unless a page says otherwise, those 60 are the whole population, and a “sample” means a few of them drawn at random. Where a page needs two variables, it uses a smaller set of 12 households with the head’s years of schooling beside income.

The figures are illustrative, made up once and fixed. They are not official statistics.

Households: 60
Mean income: ₹28,783 a month
Median income: ₹25,000 a month
Standard deviation: ₹13,366
Lowest and highest: ₹10,500 and ₹72,000

Meet the full data set →

The idea

A rule, and the number it gives

An estimator is a rule for turning a sample into a guess at a parameter: a sample statistic used to estimate a population parameter (Hyderabad 2015). The sample mean is an estimator of μ. The sample median is another. An estimate is the number the rule gives for one particular sample. The distinction is the one from Lesson 2 in new clothes: the estimator is a random variable with a sampling distribution, and the estimate is one draw from it.

Since several rules are available for the same parameter, we need ways to compare them. Each question below is about the sampling distribution of the estimator: where it is centred, how wide it is, and what happens as n grows. The lab at the heart of this lesson has five rival estimators of Ashanagar’s mean income. Keep it in mind as you read.

Centre

Unbiased: right on average

The bias of an estimator is the distance between the centre of its sampling distribution and the true value. An estimator is unbiased if that distance is zero: its expected value is the parameter.

Bias and unbiasedness

Bias (θ^) = E (θ^) - θ, θ^unbiased ⟺ E (θ^) = θ

Unbiasedness is about the average over all samples, not about any one sample.

Lesson 2 showed that the sample mean is unbiased: E(X̄) = μ whatever the population. So is the single first observation, E(X₁) = μ, which is a warning that unbiasedness is a weak property. A rule that ignores nine-tenths of the data is unbiased and useless, as the lab will show.

The sample variance, and its square root

Dividing the sum of squared deviations by n gives a variance estimator that is too small on average: its expected value is (n − 1)/n times σ². Dividing by n − 1 removes the bias, which is why S² uses n − 1 (Module 1, Lesson 4). Take n = 10 and σ² = 50: the ÷ n version averages 45, a bias of −5, and the ÷ (n − 1) version averages 50.

Now the opening question. S² is unbiased for σ². Is S = √S² unbiased for σ? No. The square root is a curved function, and for a curved function the average of the root is not the root of the average. By Jensen’s inequality E(S) is strictly less than √E(S²) = σ, so S is biased downwards. The size can be large in small samples: for n = 2 from a normal population, S is |X₁ − X₂| ÷ √2, and E(S) = √(2/π) σ = 0.80σ, a bias of 20%. The bias fades as n grows. Unbiasedness does not survive a non-linear transformation.

Derive it

Why the sample mean and the n − 1 variance are unbiased

We want to show

E(X̄) = μ, and for S² = Σ(Xᵢ − X̄)² ÷ (n − 1), E(S²) = σ². Dividing by n gives a lower average, (n − 1)σ² ÷ n.

The sample mean is unbiased.
$E (X ˉ) = \frac{1}{n} \sum E (X_{i}) = \frac{1}{n} (n μ) = μ$
Expectation of a sum is the sum of expectations, whether or not the draws are independent. Nothing else is needed.
Split the sum of squares about X̄.
Insert μ in each deviation and use the identity from the spread lesson, where the cross term vanishes because deviations from X̄ add to zero:
$\sum (X_{i} - X ˉ)^{2} = \sum (X_{i} - μ)^{2} - n (X ˉ - μ)^{2}$
Take the expected value of each piece.
Each (X_i − μ)² averages σ², so the first piece averages nσ². The second is n times the variance of X̄, which the standard error derivation gave as σ² ÷ n, so it averages n × σ² ÷ n = σ².
$E [\sum (X_{i} - X ˉ)^{2}] = n σ^{2} - σ^{2} = (n - 1) σ^{2}$
Divide by the right number.
$E (S^{2}) = \frac{( n - 1 ) σ ^{2}}{n - 1} = σ^{2}, E [\frac{1}{n} \sum (X_{i} - X ˉ)^{2}] = \frac{n - 1}{n} σ^{2}$
Dividing by n leaves a bias of −σ² ÷ n: for n = 5 the average estimate is 80% of σ², and 4 ÷ 5 = 0.8 is the factor the simulation in the spread lesson showed. The bias is real but fades as n grows, so the version with n is biased and still consistent.
Is unbiased the same as best?
No. For a normal population Σ(X − X̄)² ÷ σ² follows a chi-square distribution with n − 1 degrees of freedom, whose variance is 2(n − 1). So the estimate Σ(X − X̄)² ÷ c has variance 2(n − 1)σ⁴ ÷ c² and bias ((n − 1) ÷ c − 1)σ². Larger c shrinks the variance faster than it grows the bias, up to c = n + 1.

Step 1 of 5

Interactive

Variance, and the price of bias

Two unbiased estimators can still differ hugely in how far they stray in any one sample. Spread is the second property, measured by the variance of the estimator, whose square root is its standard error. Combining bias and spread gives a single score, the mean squared error: the average squared distance from the truth.

Mean squared error

MSE (θ^) = E [(θ^- θ)^{2}] = Var (θ^) + Bias (θ^)^{2}

The same split as the variance of a sum: a systematic part and a random part.

An estimator with a small bias and a small variance can beat an unbiased one with a large variance. Bias and variance trade against each other, and a good estimator is one with small total error, not necessarily zero bias. Use the lab. Each estimator is tried on 2,000 samples of Ashanagar households, and the readouts split its error into the bias squared and the variance. Try each estimator at n = 5, then at n = 160, and read the table underneath.

Sample size, n202,000 samples of 20 households, drawn with replacement

true mean 28.78average of the estimates 28.77

Bias (average − truth)−0.01

Spread (SD of estimates)2.99

Bias² + variance0 + 8.9

Mean squared error8.9

Sample mean: as n grows	n = 5	n = 20	n = 80
Bias	+0.27	−0.01	−0
Spread	5.92	2.99	1.47
Mean squared error	35.1	8.9	2.2

Right on average and its spread falls as 1/√n. Unbiased and consistent: the estimator the whole module leans on.

Pause and predict

In the lab, the sample median is a steady, low-variance estimator. Is it a good estimator of mean income?

Derive it

Mean squared error is variance plus bias squared

We want to show

E [(θ^- θ)^{2}] = Var (θ^) + (E θ^- θ)^{2}

Add and subtract the estimator's own average.
The miss of the estimator θ̂ from the truth θ can be routed through its own mean E(θ̂), which is a fixed number:
$θ^- θ = (θ^- E θ^) + (E θ^- θ)$
The first bracket is the wobble from sample to sample. The second is the bias, the same in every sample.
Square it.
$(θ^- θ)^{2} = (θ^- E θ^)^{2} + 2 (θ^- E θ^) (E θ^- θ) + (E θ^- θ)^{2}$
Take the expected value and watch the middle term go.
The first term averages to Var(θ̂) by definition. The last is a constant, so it stays: bias squared. In the middle, (E θ̂ − θ) is a constant and comes outside, leaving 2(Eθ̂ − θ) × E(θ̂ − Eθ̂), and the average of the wobble around its own mean is zero. So the cross term is exactly zero, and
$MSE (θ^) = Var (θ^) + Bias (θ^)^{2}$
Watch the two bars agree.
The picture computes the error two ways for 2,000 samples: directly, as the average of (estimate − truth)², and as variance plus bias squared. The two bars are the same length for every estimator and every sample size. Switch to the sample median and the bias bar dominates, and it stays put as n grows, which is why that estimator is inconsistent for the mean.
A worked trade.
Take the Ashanagar mean, μ = 28.78 and σ² = 178.6, with n = 5. The sample mean has no bias and variance σ² ÷ 5 = 35.7, so its MSE is 35.7. The estimator Σ X ÷ (n + 1) has bias −μ ÷ 6 = −4.80, so bias squared is 23.0, and variance nσ² ÷ (n + 1)² = 24.8, so its MSE is 47.8. Shrinking cut the variance by 10.9 and added 23.0 of squared bias, a poor bargain here. Where the variance is large compared with the squared bias, the bargain can go the other way, as it did for the variance estimators in the previous derivation.

Step 1 of 5

The limit

Consistent: right in the limit

Bias and variance describe an estimator at a fixed n. Consistency describes what happens as the sample grows: the estimates converge on the true value. Formally, for any small distance ε, the probability that the estimate is within ε of the parameter tends to 1.

Consistency

θ^_{n} consistent for θ ⟺ n \to \infty lim P (∣ θ^_{n} - θ ∣ < ε) = 1 for every ε > 0

The definition behind JAM 2025: for a consistent estimator the probability is at least 1 − δ in the limit, for any small δ.

In words, a consistent estimator converges upon the true value as the sample size increases (JNU 2021). That is a different claim from being right on average: a consistent estimator need not be unbiased at any finite n, and an unbiased estimator need not be consistent. The lab’s five estimators fill in the grid:

Estimator of the mean	Unbiased?	Consistent?	Why
Sample mean	Yes	Yes	Bias zero, variance σ²/n goes to zero.
First observation	Yes	No	Variance stays at σ² however large n is.
Sum ÷ (n + 1)	No	Yes	Bias is −μ/(n + 1), which vanishes, and so does the variance.
Sample median	No	No, for the mean	Converges to the population median, a different number in a skewed population.
Midrange	No	No	The extremes keep moving outwards, so the bias grows with n.

A practical shortcut covers most cases: if an estimator’s bias and its variance both go to zero as n grows, its mean squared error goes to zero and it is consistent. This is a sufficient condition, not a definition, and it is the reason the sample mean, the sample proportion and S² are all consistent. The law of large numbers of Lesson 3 is the statement that the sample mean is consistent. Consistency also survives continuous transformations, which is why S is consistent for σ although it is biased (GATE 2021): the opening question’s answer is biased and consistent.

Spread, compared

Efficient, and the best way to combine

Among unbiased estimators of the same parameter, the one with the smallest variance is the most precise, and is called efficient (Hyderabad 2019: if Var(Tₙ) is minimum among the given class of estimators, Tₙ is efficient). For normal data the sample mean beats the sample median: the median’s variance is about π/2, roughly 1.57 times larger, so it needs about 57% more data for the same precision.

Efficiency also answers a practical question. Two independent unbiased estimators of the same parameter, with different precision, are available. How should they be combined? Any weighted average with weights adding to 1 is unbiased, so the choice is made on variance alone, and the answer is to weight each estimator by the inverse of its variance.

IIT JAM 2021 · moderate

α̂₁ and α̂₂ are independent unbiased estimators of α with standard errors σ₁ and σ₂. Which linear combination is unbiased for α and has the smallest variance?

Unbiased means the weights add to 1.
wα̂₁ + (1 − w)α̂₂ has expectation wα + (1 − w)α = α for every w.
Its variance.
The two are independent, so V(w) = w²σ₁² + (1 − w)²σ₂².
Minimise.
V′(w) = 2wσ₁² − 2(1 − w)σ₂² = 0 gives w = σ₂² ÷ (σ₁² + σ₂²). It is the minimum-variance portfolio weight of Lesson 1 with zero covariance.

Answer

The weight on α̂₁ is σ₂² ÷ (σ₁² + σ₂²) and the weight on α̂₂ is σ₁² ÷ (σ₁² + σ₂²): option D. The estimator with the larger variance gets the smaller weight. With σ₁ = 2 and σ₂ = 4 the weight on α̂₁ is 16 ÷ 20 = 0.8, and the combination has variance 3.2, below both 4 and 16.

Information

Sufficient: nothing thrown away

The last property is about information. A statistic is sufficient if it contains all the information the sample holds about the parameter (Hyderabad 2011, Hyderabad 2020): once you know it, the rest of the sample adds nothing. In a sample of coin tosses, the number of heads is sufficient for the probability of heads: the order of the heads and tails tells you nothing more about the coin. The sample mean is sufficient for the mean of a normal population, and the sample maximum is sufficient for the upper limit of a uniform one. It does not contain all information needed to solve the research problem, nor all information about the population, only everything in the sample about the parameter.

Sufficiency is a separate property from the others. A sufficient statistic may be biased and need not be consistent, so sufficiency does not guarantee either. Whether a given statistic is sufficient is settled by the factorisation theorem of the next lesson.

Watch out for

The mistakes examiners are fishing for

Unbiased is not the same as good. The first observation alone is unbiased. Judge an estimator by its mean squared error.
Consistent is not the same as unbiased. Sum ÷ (n + 1) is biased and consistent. The first observation is unbiased and inconsistent.
Unbiasedness does not survive a square root. S² is unbiased for σ², and S is biased for σ.
Efficiency compares unbiased estimators. It means smallest variance in a class.
Weights go with the inverse of the variance, not the variance: the precise estimator gets the bigger weight.

Solving the opening question

The opening question, solved

GATE 2021 · hard

S² is unbiased and consistent for σ². Is S = √S² unbiased and consistent for σ?

Bias.
The square root is strictly concave. By Jensen’s inequality E(√S²) < √E(S²) = σ, unless S² is constant. So S is biased, always on the low side.
Consistency.
S² converges to σ² in probability as n grows, and the square root is continuous, so S converges to σ.

Answer

Biased and consistent: option B. For n = 2 the bias is already known: E(S) = 0.80σ for normal data.

Revise

The lesson on one screen

Estimator and estimate: A rule applied to a sample is an estimator (a random variable). The number for one sample is the estimate.
Unbiased: E(θ̂) = θ. The sample mean and S² are unbiased. S is biased for σ (Jensen). Unbiasedness alone is weak: the first observation is unbiased.
MSE: Variance + bias². A little bias can be worth it for a lot less variance.
Consistent: P(|θ̂ − θ| < ε) → 1. Bias and variance both going to zero is enough. Consistent does not imply unbiased, nor the reverse.
Efficient: Smallest variance among unbiased estimators. Combine independent unbiased estimators with weights proportional to 1 ÷ variance.
Sufficient: Contains all the information in the sample about the parameter. Found by Neyman's factorisation theorem.

Practice

Check yourself

0 of 13 correct so far

Compute

Bias of the divide-by-n variance

A sample of n = 10 is drawn from a population with σ² = 50. The estimator (1/n)Σ(xᵢ − x̄)² has expected value (n − 1)/n × σ². Find its bias.

Bias

Mean squared error

An estimator has bias 2 and variance 5. Find its mean squared error.

MSE

Which is better?

T₁ is unbiased with variance 9. T₂ has bias 1 and variance 4. Find the mean squared error of T₂. (It is smaller than T₁’s, so T₂ is the better estimator by this score.)

MSE of T₂

The best weight

Independent unbiased estimators α̂₁ and α̂₂ have standard errors 2 and 4. Find the weight on α̂₁ in the minimum-variance unbiased combination.

The best variance

For the same two estimators, find the variance of the best combination.

Variance

An unequal weighted mean

X₁, X₂, X₃ are independent draws with variance σ². T = (X₁ + 2X₂ + 3X₃) ÷ 6 is an unbiased estimator of the mean. Find Var(T) ÷ σ².

Var(T) ÷ σ²

The bias of S

A sample of n = 2 is drawn from a normal population with σ = 10. E(S) = √(2/π) σ. Find E(S).

E(S)

The unbiased variance estimate

A sample of n = 20 has a sum of squared deviations from its mean of 190. Find the unbiased estimate of the population variance.

s²

Quick questions

1An estimator T is unbiased for θ if

2An estimator is unbiased, but its variance does not shrink as n grows. It is

3The mean squared error of an estimator equals

4A statistic that contains all the information about a parameter available in the sample is

5Which statement about a consistent estimator is possible?

Real questions

Past papers, by level

Foundation

Definitions and one-step calculations. Most CUET and Hyderabad questions sit here.

6 questionsPractise →

Standard

Joint tables, standard errors and statement questions, up to JNU and IIT JAM level.

4 questionsPractise →

Delhi School level

This topic’s hardest questions, from the DSE entrance, are worked through in Lesson 13: Estimation and confidence intervals.

6 questionsLesson 13 →

All 10 past-paper questions for this lesson, easiest first: practise the full set →

These are questions from the TestStop bank. A free account answers 10 a day, each with its worked solution; All-Access opens every set with no daily limit.

← Lesson 3Standard error and the central limit theorem Lesson 5 →Maximum likelihood and the method of moments

Estimators: bias, variance, consistency, sufficiency

A rule, and the number it gives

Unbiased: right on average

The sample variance, and its square root

Variance, and the price of bias

Consistent: right in the limit

Efficient, and the best way to combine

Sufficient: nothing thrown away

The mistakes examiners are fishing for

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Standard

Delhi School level

Read next

Estimators: bias, variance, consistency, sufficiency

A rule, and the number it gives

Unbiased: right on average

The sample variance, and its square root

Variance, and the price of bias

Consistent: right in the limit

Efficient, and the best way to combine

Sufficient: nothing thrown away

The mistakes examiners are fishing for

The opening question, solved

The lesson on one screen

Check yourself

Compute

Quick questions

Past papers, by level

Foundation

Standard

Delhi School level

Read next