[Beginner] — how spread out the data is, and why the variance formula divides by rather than .

Before you start

What you will be able to do after this lesson:

  1. Compute range, quartiles, interquartile range, variance and standard deviation, and say what each is resistant to.
  2. Explain why the sample variance divides by , and demonstrate the bias that appears if you use .
  3. Build and read a cumulative frequency curve, and get the median and quartiles off it.
  4. Apply the rule for outliers, and say what it does and does not claim.

Study route: read 1–5, attempt the prediction in section 3, then run the lab. Block 3 settles the question with numbers rather than assertion.


1. Why this exists

Two production servers both average ms response time. On the first, almost every request takes between and ms. On the second, most take ms and a few take seconds.

Same centre. Completely different service. The difference is dispersion, and it is often the thing that actually matters — an average that is right on average and wrong in both directions is not a useful promise to make to a user.

Centre tells you where the data sits. Spread tells you how much you should trust the centre.

2. Terminology

TermPlain-English definitionNote
RangeLargest minus smallestUses only two values; destroyed by one outlier
QuartileThe values cutting the sorted data into four, (the median),
IQR — the spread of the middle halfResistant to outliers
PercentileThe value below which a given percentage fallsThe 90th percentile is the of latency dashboards
Deviation, how far one value is from the meanThese always sum to exactly zero
VarianceThe mean of the squared deviationsIn squared units
Standard deviationThe square root of the varianceBack in the original units
Cumulative frequencyRunning total of frequenciesPlotted as an ogive
OgiveThe cumulative frequency curvePlotted at class upper bounds

3. Variance, and the

Deviations from the mean sum to zero by construction:

So you cannot average them directly. Squaring them fixes the sign problem, and gives the variance:

Standard deviation is , which puts the answer back in the original units — metres rather than square metres.

Why and not

The intuition first, then the demonstration.

When you compute the variance of a sample, you do not know the population mean . You use the sample mean instead — and is, by construction, the number that makes as small as it can possibly be. It is fitted to this sample.

So the sum of squared deviations about is systematically smaller than it would have been about the true . Dividing by inherits that shortfall and underestimates the population variance, every time. Not sometimes — on average, always.

The correction is to divide by : you spent one degree of freedom estimating the mean, so only of the deviations are free to vary. Once you know of them, the last is determined, because they must sum to zero.

The factor works out exactly: dividing by gives, on average, of the true variance. For that is — a underestimate, which is not a rounding detail.

TIP

Predict before running the lab. A population has variance exactly . You take thousands of samples of size , computing the variance of each with and with , then average each over all the samples. What two numbers do you expect? Commit before opening the answers.

When is right. If you have the entire population, there is no estimation and no lost degree of freedom: divide by . The distinction is not about sample size, it is about whether the mean you subtracted was estimated from the same data.

4. Quartiles, ogives and outliers

Quartiles cut the sorted data into four equal parts. Conventions for the exact positions differ between textbooks and software — the lab states which one it uses, and that is the honest way to handle it.

Cumulative frequency curves. Plot the running total against the upper bound of each class, and join the points. Read off:

  • the median at up the vertical axis,
  • at , at ,
  • any percentile at the corresponding height.

Plotting at class midpoints instead of upper bounds is the standard error: cumulative frequency counts everything up to and including the class, so the point belongs at its top.

The rule. A value is flagged as an outlier if it lies below or above .

This is a convention, not a test. It flags roughly of data drawn from a perfectly normal distribution — so on a large clean dataset it will flag points that are not errors at all. It is a prompt to look, not a verdict.

5. Which measure to use

MeasureResistant to outliers?Uses all data?Use when
RangeNo — worst possibleNoQuick sanity check only
IQRYesNoSkewed data, or outliers present
Standard deviationNoYesRoughly symmetric data, and further mathematics

The pairing matters: report median with IQR, or mean with standard deviation. Mixing them — a mean with an IQR — signals that the choice was not deliberate.

Worked example — runnable

Runnable example: save as dispersion.py in any empty directory and run python3 dispersion.py. Standard library only; writes no files.

"""Spread, quartiles, ogives, and a demonstration of why variance uses n-1."""
import math
import random
 
 
def mean(xs):
    return sum(xs) / len(xs)
 
 
def median(xs):
    s = sorted(xs); n = len(s); m = n // 2
    return s[m] if n % 2 else (s[m - 1] + s[m]) / 2
 
 
def quartiles(xs):
    """Tukey's hinges: split at the median, EXCLUDING it when n is odd."""
    s = sorted(xs); n = len(s); m = n // 2
    lower, upper = s[:m], s[m + 1:] if n % 2 else s[m:]
    return median(lower), median(s), median(upper)
 
 
def variance(xs, ddof=1):
    """ddof=1 divides by n-1 (a sample); ddof=0 divides by n (a population)."""
    m = mean(xs)
    return sum((x - m) ** 2 for x in xs) / (len(xs) - ddof)
 
 
def stdev(xs, ddof=1):
    return math.sqrt(variance(xs, ddof))
 
 
def outliers(xs):
    q1, _, q3 = quartiles(xs)
    iqr = q3 - q1
    lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
    return [x for x in xs if x < lo or x > hi], (lo, hi)
 
 
def ogive(classes):
    """classes: (low, high, frequency). Returns cumulative points at UPPER bounds."""
    total = 0
    pts = []
    for lo, hi, f in classes:
        total += f
        pts.append((hi, total))
    return pts
 
 
def read_from_ogive(points, target):
    """Linear interpolation between plotted points - what you do by eye on paper."""
    prev_x, prev_y = points[0][0], 0
    for x, y in points:
        if y >= target:
            if y == prev_y:
                return x
            return prev_x + (target - prev_y) * (x - prev_x) / (y - prev_y)
        prev_x, prev_y = x, y
    return points[-1][0]
 
 
if __name__ == "__main__":
    print("Block 1 - two servers with the same mean")
    steady = [95, 98, 99, 100, 100, 101, 102, 103, 104, 98]
    spiky = [20, 21, 22, 20, 23, 21, 19, 22, 80, 752]   # same total, so the same mean
    for name, data in [("steady", steady), ("spiky ", spiky)]:
        q1, q2, q3 = quartiles(data)
        print(f"  {name}: mean {mean(data):7.2f}  median {q2:6.1f}  sd {stdev(data):8.2f}"
              f"  IQR {q3 - q1:6.1f}  range {max(data) - min(data):6}")
    assert abs(mean(steady) - mean(spiky)) < 1.0, "the two means are close"
    assert stdev(spiky) > 10 * stdev(steady)
    print("  near-identical means; the standard deviations differ by more than 10x")
 
    print()
    print("Block 2 - deviations from the mean always sum to zero")
    for data in (steady, spiky, [1, 2, 3]):
        total = sum(x - mean(data) for x in data)
        print(f"  n={len(data):2}: sum of deviations = {total:.12f}")
        assert abs(total) < 1e-9
    print("  which is why they must be squared before averaging")
 
    print()
    print("Block 3 - why n-1: the bias, measured")
    # A population we know everything about.
    population = [2, 4, 4, 4, 5, 5, 7, 9]
    mu = mean(population)
    true_var = variance(population, ddof=0)          # a real population: divide by n
    print(f"  population {population}")
    print(f"  true mean {mu}, true variance {true_var}")
 
    random.seed(20260910)
    TRIALS, N = 200000, 5
    tot_n = tot_n1 = 0.0
    for _ in range(TRIALS):
        sample = [random.choice(population) for _ in range(N)]
        tot_n += variance(sample, ddof=0)
        tot_n1 += variance(sample, ddof=1)
    avg_n, avg_n1 = tot_n / TRIALS, tot_n1 / TRIALS
    print(f"  {TRIALS:,} samples of size {N}:")
    print(f"    average of (divide by n)   = {avg_n:.4f}   vs true {true_var:.4f}"
          f"   -> {100 * avg_n / true_var:.1f}% of the truth")
    print(f"    average of (divide by n-1) = {avg_n1:.4f}   vs true {true_var:.4f}"
          f"   -> {100 * avg_n1 / true_var:.1f}% of the truth")
    print(f"  theory says dividing by n gives (n-1)/n = {(N-1)/N:.3f} of the variance")
    assert abs(avg_n / true_var - (N - 1) / N) < 0.01, "the n version is biased low"
    assert abs(avg_n1 / true_var - 1.0) < 0.01, "the n-1 version is unbiased"
    print("  the n-1 version lands on the truth; the n version is short by exactly 1/n")
 
    print()
    print("Block 4 - quartiles, the IQR rule, and what it flags")
    marks = [12, 15, 18, 22, 25, 27, 28, 30, 31, 33, 35, 38, 41, 44, 91]
    q1, q2, q3 = quartiles(marks)
    found, (lo, hi) = outliers(marks)
    print(f"  data: {marks}")
    print(f"  Q1={q1}  median={q2}  Q3={q3}  IQR={q3-q1}")
    print(f"  fences: below {lo} or above {hi}  ->  flagged: {found}")
    assert found == [91]
    print("  91 is flagged. That is a prompt to investigate, not a verdict:")
    print("  the rule flags about 0.7% of perfectly normal data as well")
 
    print()
    print("Block 5 - a cumulative frequency curve, read by interpolation")
    classes = [(0, 10, 4), (10, 20, 11), (20, 30, 19), (30, 40, 12), (40, 50, 4)]
    pts = ogive(classes)
    n = pts[-1][1]
    print("   upper bound   cumulative")
    for x, y in pts:
        bar = "#" * int(40 * y / n)
        print(f"   {x:11}   {y:10}  {bar}")
    for label, frac in [("Q1", 0.25), ("median", 0.50), ("Q3", 0.75), ("90th pct", 0.90)]:
        print(f"   {label:9} at height {frac * n:5.1f}  ->  {read_from_ogive(pts, frac * n):.3f}")
    med = read_from_ogive(pts, n / 2)
    assert 20 < med < 30, "the median class is 20-30"
    q1o, q3o = read_from_ogive(pts, n / 4), read_from_ogive(pts, 3 * n / 4)
    print(f"   IQR from the curve: {q3o - q1o:.3f}")
    assert q3o > med > q1o
 
    print()
    print("dispersion: passed")

Expected output:

Block 1 - two servers with the same mean
  steady: mean  100.00  median  100.0  sd     2.67  IQR    4.0  range      9
  spiky : mean  100.00  median   21.5  sd   229.84  IQR    3.0  range    733
  near-identical means; the standard deviations differ by more than 10x

Block 2 - deviations from the mean always sum to zero
  n=10: sum of deviations = 0.000000000000
  n=10: sum of deviations = 0.000000000000
  n= 3: sum of deviations = 0.000000000000
  which is why they must be squared before averaging

Block 3 - why n-1: the bias, measured
  population [2, 4, 4, 4, 5, 5, 7, 9]
  true mean 5.0, true variance 4.0
  200,000 samples of size 5:
    average of (divide by n)   = 3.2012   vs true 4.0000   -> 80.0% of the truth
    average of (divide by n-1) = 4.0015   vs true 4.0000   -> 100.0% of the truth
  theory says dividing by n gives (n-1)/n = 0.800 of the variance
  the n-1 version lands on the truth; the n version is short by exactly 1/n

Block 4 - quartiles, the IQR rule, and what it flags
  data: [12, 15, 18, 22, 25, 27, 28, 30, 31, 33, 35, 38, 41, 44, 91]
  Q1=22  median=30  Q3=38  IQR=16
  fences: below -2.0 or above 62.0  ->  flagged: [91]
  91 is flagged. That is a prompt to investigate, not a verdict:
  the rule flags about 0.7% of perfectly normal data as well

Block 5 - a cumulative frequency curve, read by interpolation
   upper bound   cumulative
            10            4  ###
            20           15  ############
            30           34  ###########################
            40           46  ####################################
            50           50  ########################################
   Q1        at height  12.5  ->  17.727
   median    at height  25.0  ->  25.263
   Q3        at height  37.5  ->  32.917
   90th pct  at height  45.0  ->  39.167
   IQR from the curve: 15.189

dispersion: passed

Block 3 is the argument. Two hundred thousand samples, and dividing by lands at exactly of the true variance — which is , as predicted. The correction is not a convention; it is the size of a measurable bias.

Common pitfalls and traps

  • Using for a sample. It underestimates, systematically, by a factor of . The smaller the sample, the worse — and small samples are exactly when you are most likely to be doing this by hand.
  • Using for a full population. There is no estimation, so there is no correction. Dividing by then overestimates.
  • Reporting variance in the write-up. It is in squared units — “the variance of the delay was seconds squared” is not a sentence anyone can act on. Report the standard deviation.
  • Plotting an ogive at class midpoints. Cumulative frequency counts everything up to the top of the class, so the point goes at the upper bound.
  • Treating the rule as a test for errors. It is a screening convention. On a large clean dataset it will flag genuine data.
  • Mixing centre and spread measures. Median with IQR, or mean with standard deviation. A mean reported with an IQR usually means neither was chosen on purpose.
  • Assuming quartile values are universal. Textbooks and libraries use different interpolation conventions and give slightly different answers on small datasets. Say which you used.

Check your understanding

  1. Find the range, , median, and IQR of .
  2. A dataset has variance cm². What is its standard deviation, and in what units?
  3. Why can you not simply average the deviations from the mean?
  4. When is dividing by the correct choice?
  5. A cumulative frequency curve for students reaches at mark and at mark . Estimate the interquartile range.
Answers — open only after an attempt
  1. Range . With eight values the median is . Lower half gives ; upper half gives . IQR .
  2. cm — the square root returns it to the original units, which is the entire reason standard deviation is reported instead of variance.
  3. Because they always sum to exactly zero: by construction. Their average is therefore for every dataset, carrying no information. Squaring removes the cancellation.
  4. When you have the whole population, so the mean you subtracted is the true mean rather than one estimated from the same data. No degree of freedom was spent, so no correction is needed.
  5. is at height , which the curve reaches at . is at height , reached at . So IQR .

And the prediction from section 3: the average lands on — it is unbiased. The average lands on — short by exactly . The lab confirms both to within over 200,000 samples.

Practice — independent task

Implement box_plot_summary(data) returning the five-number summary and a text box plot, then investigate what the rule actually flags.

  1. Return minimum, , median, , maximum, the IQR, the fences, and the flagged outliers. State which quartile convention you used and why.
  2. Draw the box plot as text: whiskers to the most extreme values inside the fences, outliers marked individually beyond them.
  3. Measure the false-positive rate. Generate many samples of size from a normal distribution (random.gauss) with a fixed seed, and count what fraction of points get flagged. Compare against the theoretical .
  4. Then vary from to and report the fraction of samples containing at least one flagged point. Explain the trend — it is not the same question as step 3, and the answer surprises people.
  5. Finally, run the same experiment on data from a genuinely skewed distribution (an exponential, say random.expovariate(1)), and report how the flagging rate changes. Say what that means for using the rule on latency data.

Edge cases: fewer than four data points; all values identical (IQR of zero — what do the fences become, and what does the rule then flag?); exactly one outlier at the fence boundary.

Done when: your step-3 rate is close to , your step-4 answer explains the trend rather than just reporting it, and you can say in one sentence why applying this rule to response-time data flags things that are not errors.

Tradeoffs, limits and extensions

Standard deviation assumes symmetry to be interpretable. “Mean one standard deviation” suggests a symmetric band. On skewed data that band can extend below zero for a quantity that cannot be negative — a sign the summary does not fit the shape. Percentiles do not have this problem, which is why latency is reported as rather than mean sd.

Squaring is a choice with consequences. Squaring makes the algebra work — it gives the mean its special status, and makes variances of independent quantities add. It also means one point twice as far from the centre contributes four times as much, so the standard deviation is strongly influenced by extremes. The mean absolute deviation avoids that and is barely used, because it has no comparable algebra.

Where this goes next. Variance is what makes the normal distribution’s parameters meaningful, and the fact that variances add for independent quantities is the engine of the Central Limit Theorem — the reason the standard error of a mean is , which is the whole basis of inference.

Before moving on

You are done with this lesson when you can:

  • Compute all five of range, IQR, variance, standard deviation and any percentile.
  • Explain the correction from degrees of freedom, and state the size of the bias without it.
  • Build an ogive with the points in the right places and read quartiles off it.
  • Apply the IQR outlier rule and describe its limits.

Recap for later lookup: range is worst-case fragile; IQR is resistant; for a sample, for a population; dividing by on a sample underestimates by ; standard deviation restores the original units; ogives are plotted at class upper bounds; outlier fences at and .

Next: Probability — from describing data you have to reasoning about data you have not seen yet.

↑ Back to Probability and Statistics