Covariance: what it is, where the formula comes from, and what it means

Take 12 Ashanagar households and, for each, the head’s years of schooling (x) and the monthly income in thousands of rupees (y). We want one number that is positive when households with more schooling than average tend to have more income than average, negative when they tend to have less, and near zero when there is no such tendency. It should also get bigger as the tendency gets stronger.

The averages are x̄ = 10.42 years and ȳ = ₹23 thousand. Subtract them to get each household’s deviations, how far it sits from the average on each variable:

Household 1 has 5 years of schooling and an income of ₹15 thousand, so d_x = 5 − 10.42 = −5.42 and d_y = 15 − 23 = −8. It is below average on both.

Add all the d_x and you always get zero, because Σ(x_i − x̄) = Σx_i − n x̄ = 0. The deviations cancel by construction, so a sum cannot measure anything. What matters is whether the two deviations of the same household point the same way. Multiply them:

The sign of the product is exactly the “move together or apart” test, and a bigger deviation on either side makes the product bigger.

Mark the point of means on the scatter. For each household draw the rectangle from that point to the household: its width is d_x, its height d_y, and its area is the product. Blue rectangles lie in the two quadrants where both deviations share a sign, red ones in the other two. Click through the households, then try the other two scatters.

The total signed area is the sum of all the products, and dividing by n turns a total into an average, so the number does not grow just because there are more households. That gives the formula. For the 12 households the products add to 262, and

The units are years of schooling times thousands of rupees, so the size of the number depends on the units chosen. That is the reason the covariance is usually divided by both standard deviations to give the correlation.

• A variable with itself. Put y = x. Each rectangle becomes a square of area (x_i − x̄)², and the average of those is the variance. So Cov(X, X) = Var(X): the variance is the covariance of a variable with itself.
• Population or sample. Dividing by n gives the population form, 21.83 here. Dividing by n − 1 gives the sample form, 262 ÷ 11 = 23.82, which is the one to use when the 12 households are a sample from a bigger population. The correlation is unchanged, since the same divisor appears in the top and in both standard deviations.

Write μ_X = E[X] and μ_Y = E[Y]. The definition is Cov = E[(X − μ_X)(Y − μ_Y)]. Multiply out the bracket, as in (a − b)(c − d) = ac − ad − bc + bd:

The average of a sum is the sum of the averages, and a constant comes out in front of an average. Here μ_X and μ_Y are constants, since they are fixed numbers. So

Two of the three last terms are the same product with a minus sign, and the third is the same product with a plus. They leave one copy with a minus:

JNU 2021 gives a joint table: X takes 1 or 3, Y takes −3, 2 or 4. The row totals give E[X] = 0.5 × 1 + 0.5 × 3 = 2, the column totals give E[Y] = (−3)(0.4) + (2)(0.3) + (4)(0.3) = 0.6, and multiplying each cell by its x and y gives E[XY] = 0. Then

That is the key, option D. The shortcut needs only one pass for E[XY] and the two means, instead of computing a deviation for every cell.

Adding a constant b to every x moves the average by b too, so x_i + b − (x̄ + b) = x_i − x̄. The deviations, and so every rectangle, are unchanged. Measuring income in rupees from zero or from some other base makes no difference.

Multiplying every x by a multiplies every deviation by a, and so every rectangle’s width, so every area, by a. The same holds for c on the y side, which is why Cov(aX, cY) = ac Cov(X, Y). Change income from rupees to thousands of rupees, a factor of 1000, and the covariance changes by 1000 too.

The deviation of a sum is the sum of the deviations: (x + z) − (x̄ + z̄) = (x − x̄) + (z − z̄). Multiply by (y − ȳ) and average, and the covariance splits in two:

The same holds in the second argument. A covariance of a variable with a constant is zero, since a constant has no deviations.

JNU 2018: y = 2w + 4u, var(w) = 5 and cov(w, u) = 1. What is cov(w, y)? Split it, then pull out the constants, then use Cov(w, w) = Var(w):

The same splitting, applied to Cov(X + Y, X + Y), is how the variance of a sum, Var(X) + Var(Y) + 2 Cov(X, Y), is derived in the correlation lesson.

Write U = X − μ_X and V = Y − μ_Y for the deviations from the means. The mean of aX + bY is aμ_X + bμ_Y, so the deviation of the combination from its own mean is aU + bV.

This is the ordinary expansion of (p + q)². Nothing about statistics has happened yet.

The average of U² is σ_X² and of V² is σ_Y², by the definition of variance, and the average of UV is Cov(X, Y), by the definition of covariance. That gives the formula. The cross term is there because the squared bracket has a middle term, and it carries a factor of 2 for the same reason.

Draw a square of side aσ_X and another of side bσ_Y, and put the two rectangles between them. Their areas are a²σ_X², b²σ_Y² and two copies of abσ_Xσ_Y. The covariance is ρσ_Xσ_Y, so each rectangle is filled in only to the fraction ρ. Move ρ to 1 and the pieces make one big square, (aσ_X + bσ_Y)². Move it to 0 and the rectangles vanish.

• Independent variables have Cov = 0, so the variances just add: Var(X + Y) = σ_X² + σ_Y².
• A difference has b = −1, so the cross term changes sign but the variances still add: Var(X − Y) = σ_X² + σ_Y² − 2 Cov. Never subtract the variances themselves.
• Example (JNU 2013). With Var X = 2, Var Y = 4 and Cov = 2, Var(2X + 3Y) = 4 × 2 + 9 × 4 + 2 × 2 × 3 × 2 = 8 + 36 + 24 = 68.

For independent discrete variables, P(X = x, Y = y) = P(X = x) P(Y = y) for every pair. The average of the product is the double sum over all pairs:

The double sum splits into a product because each x multiplies the whole sum over y. By the shortcut, Cov = E[XY] − E[X]E[Y] = 0.

Let X be −1, 0 or 1, each with probability 1/3, and let Y = X². Y is completely decided by X, so they are as dependent as two variables can be. But E[X] = 0, E[XY] = E[X³] = (−1 + 0 + 1) ÷ 3 = 0, and so Cov = 0 − 0 × E[Y] = 0.

The rectangles explain it: the points lie on a U shape, and for every point on the right arm there is a mirror point on the left arm with the opposite sign of d_x but the same d_y, so the areas cancel. Covariance sees only the straight-line part of a relationship.

DSE 2012: X is uniform on [−1, 1]. Find Cov(X, X²). The same argument gives E[X] = 0 and E[X³] = 0, since X³ is an odd function and the interval is symmetric, so the covariance is 0 even though X² is a function of X.

Standardise each variable: $z_x=\frac{x-\bar{x}}{{\sigma }_X}$ and $z_y=\frac{y-\bar{y}}{{\sigma }_Y}$. Two facts follow at once. By the definition of the standard deviation the average of z_x² is 1, and so is the average of z_y². And the correlation is the average product of the two z scores:

For each household the gap z_x − z_y squared is at least 0, so its average is too. Expand it:

So 2 − 2r ≥ 0, which says r ≤ 1. For the twelve households r = 0.869 and the average squared gap is 2 − 2 × 0.869 = 0.262.

So 2 + 2r ≥ 0, which says r ≥ −1. Both bounds come from nothing more than the fact that a square cannot be negative. The picture shows what the two averages measure: how far the standardised points stray from the line y = x and from the line y = −x.

An average of squares is 0 only if every square is 0. So r = 1 means z_x = z_y for every household, which is the straight line y = ȳ + (σ_Y ÷ σ_X)(x − x̄) with a positive slope. In the same way r = −1 forces z_x = −z_y, a straight line sloping down. Any scatter that is not exactly a straight line leaves some gap, and then r is strictly inside the interval.