Moment generating functions, derived

The moment generating function of X is M(t) = E[e^tX]. For each fixed t, e^tX is just another random variable, so its average is a number, and as t varies that number traces a function. For a discrete X it is Σ e^txP(x), and for a continuous X it is ∫ e^txf(x) dx. Why e^tX? The next step shows it carries every power of X at once.

The series e^u = 1 + u + u²/2! + u³/3! + … with u = tX gives e^tX = 1 + tX + t²X²/2! + t³X³/3! + …. Take the average of each term, remembering that t is a constant:

The moments E[X], E[X²], E[X³] and so on sit in a row as the coefficients, each attached to its own power of t. That is the sense in which M “generates” them.

Differentiate once: the constant 1 disappears, the term in t becomes E[X], and every other term still has a factor of t. Set t = 0 and only E[X] survives. Differentiate twice: the t² ÷ 2! term becomes E[X²], and setting t = 0 leaves just that. In general

Geometrically, M passes through (0, 1), its slope there is the mean and its curvature gives the second moment. The lab draws the curve for each family with its tangent.

A Bernoulli trial is 1 with chance p and 0 with chance q, so M(t) = q · e⁰ + p · e^t = q + pe^t. Then M′(t) = pe^t and M″(t) = pe^t, so M′(0) = p and M″(0) = p, and the variance is p − p² = pq, as found earlier by hand.

The middle sum is the series for e^u with u = λe^t. Differentiate: M′(t) = λe^t M(t), so M′(0) = λ. And M″(t) = λe^t M(t) + (λe^t)² M(t), so M″(0) = λ + λ² and the variance is λ + λ² − λ² = λ.

The restriction t < λ is needed for the integral to converge, and it is all that is needed, since M only has to exist around t = 0. Then M′(t) = λ ÷ (λ − t)², so M′(0) = 1 ÷ λ, and M″(0) = 2 ÷ λ², giving the variance 2 ÷ λ² − 1 ÷ λ² = 1 ÷ λ².

For Z standard normal, M_Z(t) = ∫ e^tz e^−z²/2 ÷ √(2π) dz. Combine the exponents and complete the square: tz − z²/2 = −(z − t)²/2 + t²/2. So

The remaining integral is a normal density centred at t, so its area is 1. For a general normal X = μ + σZ, use e^t(μ+σZ) = e^μt e^(σt)Z, so M_X(t) = e^μt M_Z(σt) = e^{μt + σ²t²/2}. Its slope at 0 is μ and M″(0) = σ² + μ², so the variance is σ².

A gamma with shape α and rate λ is the sum of α independent exponentials, so its function is the exponential one to the power α: (λ ÷ (λ − t))^α, which the sums derivation below confirms. With α = k ÷ 2 and λ = ½ this is the chi-square: (1 − 2t)^−k/2, for t < ½.

M(t) must be finite for t in some interval around 0. For the heavy-tailed Cauchy distribution, and for the lognormal, e^tX has infinite average for every t ≠ 0, so no moment generating function exists, although other tools (the characteristic function, which uses an imaginary exponent) always do.

e^t(X+Y) = e^tX · e^tY. When X and Y are independent, the average of a product of functions of them is the product of the averages, the same factorisation that made independent variables uncorrelated. So

Finding the distribution of a sum directly means adding up every pair of values with the same total, a convolution. The moment generating function turns that into one multiplication.

A Bernoulli has M(t) = q + pe^t, and a binomial is the sum of n independent copies, so M(t) = (q + pe^t)ⁿ. Its slope at 0 is n(q + p)ⁿ⁻¹p = np, and the variance comes out as npq. For n = 5 and p = 0.4, M(1) = 13.677.

The product has the form of a Poisson function with mean λ₁ + λ₂. The lab adds up pairs the long way and compares: the bars agree exactly. For example, with means 1.5 and 2.5 the chance that the total is 3 is the Poisson(4) value 0.1954.

(λ ÷ (λ − t))^α₁ · (λ ÷ (λ − t))^α₂ = (λ ÷ (λ − t))^α₁+α₂: shapes add when the rate is the same. This is the step that was assumed when a gamma was built from exponential waits, and when a chi-square was built by adding squared normals. Likewise two independent normals give e^{(μ₁+μ₂)t + (σ₁²+σ₂²)t²/2}, which is normal with the means and the variances added.

A theorem, quoted without proof here, says a moment generating function that is finite around 0 belongs to exactly one distribution. So recognising the shape is enough to identify a variable. (¼ + ¾e^t)⁶ is a binomial with n = 6 and p = ¾. e^2(et−1) is a Poisson with mean 2. And 1 ÷ (1 − 2t) = ½ ÷ (½ − t) is an exponential with rate ½, that is, mean 2.

A binomial with p = λ ÷ n has M(t) = (1 + λ(e^t − 1) ÷ n)ⁿ. Since (1 + x ÷ n)ⁿ tends to e^x, this tends to e^λ(et−1), the Poisson function. By uniqueness the binomial distribution tends to the Poisson, which is the result derived by hand on the binomial and Poisson page. The same method, applied to a standardised sum, shows a function tending to e^t²/2, which is the central limit theorem.