Expectation as a balance point, variance as spread
Put each value of a random variable on a number line and give it its probability as a weight. Where does the line balance, and how do you measure how spread out the weight is?
The balance point summarizes where the distribution is located. The variance summarizes how spread out it is around that point. You will find the first by eye, then see how the second responds when the weights move.
Before you start · 3 quick checks on reading a PMF, weighted averages and balancing a seesaw, about 3 min
A random variable and its PMF. A discrete random variable \(X\) takes values \(x_1, x_2, \dots\) with probabilities \(p(x) = P(X = x)\), each at least 0 and together adding to 1. The lesson on PMF, PDF and CDF drew these as bars; the lesson on conditional probability and the Monty Hall problem worked with the events behind them. Unless stated otherwise, the random variables in the activities on this page take only finitely many values.
Probabilities as weights. Picture a rod with no weight of its own lying along the number line. At each value \(x\) of \(X\) stands a mass equal to \(p(x)\), so the masses add up to 1. A fulcrum, a support under the rod, sits at a point \(c\). A mass \(p(x)\) at signed distance \(x - c\) from the fulcrum turns the rod with moment \(p(x)(x - c)\): positive to the right of \(c\), negative to the left, and larger the farther out the mass is. The rod balances where the moments of all the masses cancel.
Fine print: what the picture leaves out
A real rod tipped off balance keeps turning until something stops it. The figures on this page draw the tilt in proportion to the total moment instead, so that a small miss shows as a small tilt. The strength of gravity and the physical size of the weights play no part: only probabilities and positions matter.
Find the balance point
Choose a layout, move the fulcrum to where you think the rod balances, then check your guess.
How to move the fulcrum, and what happens after a check
Drag the triangle under the rod, or focus it and use the arrow keys (steps of 0.05), Page Up and Page Down (steps of 1), Home (0) and End (10). Check my balance point then shows the balance point, tilts the rod by the amount you were off, and says what your choice coincided with.
After a check you can keep moving the fulcrum: the verdict stays with your checked guess, and the line under it follows the current fulcrum. Press Guess again or choose another layout to start a new guess.
Expectation. The expectation, or mean, of a discrete \(X\) is the probability-weighted average of its values,
\[E[X] = \sum x\,p(x).\]Balance. About a fulcrum at \(c\) the total moment is
\[\begin{aligned} \sum p(x)(x - c) &= \sum x\,p(x) - c \sum p(x) \\ &= E[X] - c, \end{aligned}\]because the probabilities add to 1. The total is 0 for exactly one fulcrum position, \(c = E[X]\): the rod balances at the expectation and nowhere else. To the left of \(E[X]\) the total moment is positive and the right end goes down; to the right of it the total is negative.
Spreading and shifting the masses
Expectation. For a discrete \(X\), \(E[X] = \sum x\,p(x)\), the balance point from stage 1. For a function \(g\), the random variable \(g(X)\) has expectation
\[E[g(X)] = \sum g(x)\,p(x).\]Variance. Write \(\mu = E[X]\). The variance is the probability-weighted average of the squared distances from \(\mu\),
\[\begin{aligned} \operatorname{Var}(X) &= E[(X - \mu)^2] \\ &= \sum (x - \mu)^2 p(x). \end{aligned}\]Standard deviation. \(\operatorname{SD}(X) = \sqrt{\operatorname{Var}(X)}\). If \(X\) is measured in dollars, \(E[X]\) and \(\operatorname{SD}(X)\) are in dollars and \(\operatorname{Var}(X)\) is in squared dollars.
Fine print: densities, and expectations that do not exist
For a continuous \(X\) with density \(f\), the sums become integrals: \(E[X] = \int x f(x)\,dx\) and \(\operatorname{Var}(X) = \int (x - \mu)^2 f(x)\,dx\). For the uniform density \(\tfrac{1}{2}\) on \([1, 3]\) from the previous lesson, \(E[X] = \int_1^3 \tfrac{x}{2}\,dx = 2\), the middle of the interval.
The integral gives a finite expectation only when \(\int \lvert x \rvert f(x)\,dx\) is finite. That can fail when the density has a heavy tail. For \(f(x) = 1/x^2\) on \(x \ge 1\) (and 0 below 1), the integral of \(x f(x) = 1/x\) from 1 to \(t\) is \(\ln t\), which grows without bound, so this \(X\) has no finite expectation, and no variance either. Sums over infinitely many values can fail in the same way.
Spread and shift explorer
The masses start at \(\tfrac{1}{4}\) at 3, \(\tfrac{1}{4}\) at 5 and \(\tfrac{1}{2}\) at 6, with \(E[X] = 5\) and \(\operatorname{Var}(X) = \tfrac{3}{2}\). The spread factor \(k\) moves each mass to \(5 + k(x - 5)\), multiplying its distance from 5 by \(k\); the shift \(s\) then moves every mass by \(s\).
Goals
Linear changes. For numbers \(a\) and \(b\), the random variable \(aX + b\) has
\[\begin{aligned} E[aX + b] &= aE[X] + b, \\ \operatorname{Var}(aX + b) &= a^2 \operatorname{Var}(X), \\ \operatorname{SD}(aX + b) &= \lvert a \rvert \operatorname{SD}(X). \end{aligned}\]The added \(b\) moves every value and the mean alike, so it leaves every distance from the mean unchanged. The factor \(a\) multiplies every distance from the mean by \(\lvert a \rvert\) and every squared distance by \(a^2\). The explorer's spread is the case \(a = k\), \(b = 5 - 5k + s\).
Sums. For two random variables \(X\) and \(Y\) on the same experiment, each with an expectation, \(E[X + Y] = E[X] + E[Y]\), whether or not they are independent. Roll one fair die, let \(X\) be the number shown and \(Y = 7 - X\): \(Y\) is determined by \(X\), and still \(E[X + Y] = E[7] = 7 = \tfrac{7}{2} + \tfrac{7}{2}\). Variances do not add in this way: here \(X + Y = 7\) has variance 0, while \(\operatorname{Var}(X) = \operatorname{Var}(Y) = \tfrac{35}{12}\).
The shortcut formula for the variance
Assumptions. \(X\) is a random variable with finitely many values, each with positive probability, and \(\mu = E[X]\). The proof uses three facts about expectations of such variables:
- for a function \(g\), \(E[g(X)] = \sum g(x)\,p(x)\);
- linearity: for numbers \(a\), \(b\), \(d\) and random variables \(Y\), \(Z\) on the same experiment, \(E[aY + bZ + d] = aE[Y] + bE[Z] + d\), with no independence needed;
- a probability-weighted average of numbers that are all at least 0 is at least 0.
Conclusion.
\[\operatorname{Var}(X) = E[X^2] - (E[X])^2.\]In particular \(E[X^2] \ge (E[X])^2\), with equality exactly when \(X\) is a constant.
- The same algebra proves the rule for linear changes from stage 2. The mean of \(aX + b\) is \(a\mu + b\), so its deviation is \(aX + b - (a\mu + b) = a(X - \mu)\). Squaring gives \(a^2 (X - \mu)^2\), and linearity gives \(\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)\).
- The theorem holds for a continuous \(X\) whenever \(E[X^2]\) is finite, with the same proof and integrals in place of sums.
- \(E[g(X)]\) and \(g(E[X])\) can differ. For \(g(x) = x^2\) the theorem says how far apart they are: \(E[X^2] - (E[X])^2 = \operatorname{Var}(X)\). For a linear \(g(x) = ax + b\) they are equal, by linearity.
Problems
Each problem has hints and a full solution.
More problems
Next in Probability: From binomial to Poisson: the law of rare events (in preparation).
Further reading
- J. K. Blitzstein and J. Hwang, Introduction to Probability, 2nd ed. (CRC Press, 2019), ch. 4, "Expectation."
- S. Ross, A First Course in Probability, 10th ed. (Pearson, 2019), ch. 4, "Random Variables," sections 4.3 to 4.5 on expected value and variance.
- D. P. Bertsekas and J. N. Tsitsiklis, Introduction to Probability, 2nd ed. (Athena Scientific, 2008), ch. 2, "Discrete Random Variables," section 2.4, "Expectation, Mean, and Variance."
- Z. Bodie, A. Kane and A. J. Marcus, Investments, 12th ed. (McGraw-Hill, 2021), ch. 5, "Risk, Return, and the Historical Record," on expected return and standard deviation in scenario models.