Maximum likelihood and the method of moments

You toss a coin ten times and get seven heads. Is the coin fair? Suppose the probability of heads were p = 0.5. The chance of exactly this sequence of seven heads and three tails would be 0.5¹⁰ = 0.00098. Suppose instead p = 0.7. The chance of the same sequence is 0.7⁷ × 0.3³ = 0.0022, about 2.3 times larger. The data are more probable under 0.7 than under 0.5, so 0.7 is the better explanation. Run through every value of p between 0 and 1 and one value wins. That value is the maximum likelihood estimate.

The function that gives the probability of the observed data for each candidate value of the parameter is the likelihood. It is the same expression as the probability of the data, read the other way round: the data are fixed, and the parameter is the variable.

Note what the likelihood is not. It is not the probability that the parameter equals θ, and it does not add to 1 as θ varies. It only ranks the candidate values against each other.

The same four steps find the estimator in almost every question.

1. Write the likelihood as a product of the density or probability of each observation.
2. Take logs. A log turns the product into a sum, which is easy to differentiate, and it leaves the maximum where it was, since the log is increasing.
3. Differentiate with respect to the parameter and set it to zero. Solve for the parameter.
4. Check it is a maximum (the second derivative is negative), or that it is the best of the boundary values, as in the uniform case below.

For the coin: L(p) = p^k (1 − p)^(n − k), so the log-likelihood is k ln p + (n − k) ln(1 − p). Setting the derivative k/p − (n − k)/(1 − p) to zero gives p̂ = k/n, the sample proportion. The MLE of a probability is the share of successes.

The lab shows the likelihood and the log-likelihood side by side for four models. Slide the parameter along its range. The left panel scales the likelihood to 1 at its peak, so you read “this value makes the data 40% as likely as the best one”. The right panel is the log, which has the same peak and is what you would differentiate.

The Bernoulli, and a single draw

The estimator is p̂ = k/n, and the lab’s sliders let you test the corner cases. After a single draw, n = 1, the likelihood is p if the draw was a success and 1 − p if it was a failure. The first rises all the way to p = 1 and the second falls from p = 0, so the maximum likelihood estimate is 1 or 0, the end point itself (DSE 2019). It is either 0 or 1, never strictly between: one observation is too little to say anything more moderate.

The exponential

With density λe^(−λx), the likelihood is λⁿ e^(−λΣx), the log is n ln λ − λΣx, and the derivative n/λ − Σx is zero at λ̂ = n/Σx = 1/x̄. The mean of an exponential variable is 1/λ, so the estimator says: set the rate to the reciprocal of the sample mean. For the five observations in the opening question the sum is 12.5, so λ̂ = 5/12.5 = 0.4.

The normal variance

For a normal sample with known mean μ the log-likelihood in σ² is −(n/2) ln(2πσ²) − Σ(xᵢ − μ)²/(2σ²), and the maximum is at σ̂² = Σ(xᵢ − μ)²/n: the average squared deviation from the known mean. For the sample 4.8, 4.5, 5.1, 5.2, 5.3, 5.5 with mean 5 the deviations square to 0.04, 0.25, 0.01, 0.04, 0.09 and 0.25, which sum to 0.68, so σ̂² = 0.68/6 = 0.11(IIT JAM 2024).

When the mean is unknown too, the maximum likelihood estimate of μ is x̄, and that of the variance is σ̂² = Σ(xᵢ − x̄)²/n, dividing by n and not by n − 1. It is therefore biased but consistent (GATE 2024): its expected value is (n − 1)/n times σ², the bias that Lesson 4 met. Maximum likelihood does not promise unbiasedness. Variants with n − 1 in the denominator, or sums that stop at n − 1, are not the MLE.

The uniform, where calculus fails

For a uniform variable on (0, θ), each density is 1/θ if the observation is below θ and 0 otherwise. The likelihood is therefore 0 if θ is smaller than the largest observation, and θ⁻ⁿ above it. That function is decreasing for every θ above the largest observation, so the maximum sits at the left edge: θ̂ = the sample maximum. There is no point where the derivative is zero, so solving the first-order condition would find nothing. For a uniform variable on [a, b] the same logic gives â = the smallest observation and b̂ = the largest. For the sample 1.3, 0.8, 9.5, 20.2, 8.2 that is 0.8 and 20.2 (IIT JAM 2021).

The maximum likelihood estimator is not always unbiased in small samples. It is popular because of what it does as n grows, under mild conditions:

• Consistent. It converges to the true parameter.
• Asymptotically normal and efficient. For large n its sampling distribution is close to normal and no other well-behaved consistent estimator has a smaller variance, so it reaches the lowest possible error in the limit.
• Invariant. If θ̂ is the MLE of θ, then g(θ̂) is the MLE of g(θ), for any function g. The MLE of σ is √σ̂², and the MLE of the mean of an exponential variable, 1/λ, is x̄. Invariance is why we can take the square root of the MLE of variance without redoing the calculation. It is also why the MLE of σ is biased: invariance holds for the MLE, and unbiasedness does not.

The method of moments is simpler and older. Write the population mean (and, if needed, higher moments) as a function of the parameter, then set it equal to the sample mean and solve. For the exponential the mean is 1/λ, so x̄ = 1/λ̃ and λ̃ = 1/x̄, the same as the MLE. For a uniform variable on (0, θ) the mean is θ/2, so θ̃ = 2x̄. That is different from the MLE, and it can be impossible: the sample 1, 1, 1, 9 has mean 3, so θ̃ = 6, yet an observation of 9 has just occurred. The method of moments needs no density and is easy to compute, but it can give nonsensical values and it is generally less efficient than maximum likelihood.

Lesson 4 defined a sufficient statistic as one that carries all the sample’s information about the parameter. The factorisation theorem of Neyman is the test for it (GATE 2026). If the likelihood splits into a part that depends on the data only through a statistic T and the parameter, and a part that does not involve the parameter at all, then T is sufficient.

Reading the likelihoods above: the Bernoulli likelihood p^(Σx)(1 − p)^(n − Σx) depends on the data only through Σx, so the number of successes is sufficient. The exponential λⁿe^(−λΣx) depends only on Σx. The uniform θ⁻ⁿ with the condition max ≤ θ depends only on the maximum. In every case the sufficient statistic is exactly what the MLE uses, which is no accident: the MLE is always a function of a sufficient statistic. The factorisation theorem identifies sufficient statistics, and not consistent, efficient or unbiased ones.