[Intermediate] — Machine epsilon, cancellation, and how to analyse the error in a computation rather than hoping.

The format itself is in Data Representation — sign/exponent/mantissa, NaN, denormals. This note is what it means for numerical work: how error propagates, and how to write algorithms that don’t amplify it.

Machine epsilon

The gap between 1.0 and the next representable number.


Every stored value has relative error at most :

And every arithmetic operation is correctly rounded (IEEE 754 guarantees this):

The mental model that matters: floating point numbers are relatively spaced, not absolutely. Near 1.0 the gap is ; near it’s ; near it’s .

So “how many decimal places” is the wrong question. A double has ~16 significant digits, wherever it sits on the number line. Absolute tolerances are almost always a bug — use relative ones, or a hybrid.

The tolerance you actually want:

def close(a, b, rtol=1e-9, atol=1e-12):
    return abs(a - b) <= max(rtol * max(abs(a), abs(b)), atol)

Relative for normal magnitudes, absolute as a floor near zero — because relative comparison is meaningless when the true answer is 0.

Catastrophic cancellation

The single biggest source of accuracy loss, and it’s worth being able to spot on sight.

Subtracting two nearly-equal numbers destroys significant digits:

  1.2345678901234567
- 1.2345678901234000
  ─────────────────────
  0.0000000000000567     ← 3 significant digits left, from 17

The inputs were accurate to 16 digits. The result is accurate to 3. No arithmetic error occurred — the information simply wasn’t there.

The canonical example — the quadratic formula:

When , . So one of the two roots involves subtracting nearly-equal quantities and loses precision catastrophically.

The fix — compute the well-conditioned root, then use the product of roots:

q = -0.5 * (b + copysign(sqrt(b*b - 4*a*c), b))
x1 = q / a
x2 = c / q          # from x1·x2 = c/a — no subtraction

Same mathematics, completely different accuracy. This is the classic demonstration that algebraically equivalent expressions are not numerically equivalent.

Other cancellations and their fixes:

NaiveBetter
(small )
(small )expm1(x)
(small )log1p(x)
variance via Welford’s online algorithm

expm1 and log1p exist in every standard library for exactly this reason. If you’re computing for small by hand, you’re throwing away most of your precision, and the library function is there to stop you.

The variance one is worth naming because it’s so common: the textbook formula subtracts two large nearly-equal numbers and can return a negative variance. Welford’s algorithm updates incrementally and is stable. → Expectation and Variance

Summation

Adding many numbers loses accuracy, because each addition rounds and the errors accumulate.

Naive summation of terms has error growing as — and worse if magnitudes vary, because small terms get swallowed:

sum([1e16, 1.0, -1e16])   # → 0.0 in naive order
                          # → 1.0 if you add the small ones first

Floating-point addition is not associative, so the order changes the answer. → Non-associativity

Kahan summation carries a running compensation term:

total = 0.0; c = 0.0
for x in values:
    y = x - c
    t = total + y
    c = (t - total) - y      # recovers the lost low-order bits
    total = t

Error becomes — independent of . Roughly 4× the arithmetic, and worth it for long sums.

Pairwise summation — recursively split and add halves — gives for essentially free. NumPy’s sum does this, which is why it’s more accurate than a Python loop.

A practical consequence for ML: parallel reductions sum in a non-deterministic order, so the same training run gives slightly different results. That’s not a bug and it can’t be fixed without giving up the parallelism — it’s why bit-exact reproducibility across GPU counts is generally not offered. → GPU and Parallel Computing

Error propagation

How error travels through a computation.

Through a function, by Taylor:

That coefficient is the condition number of at , and it tells you the amplification factor.

Examples worth knowing:

OperationConditionNote
Multiplication, divisionalways well-conditioned
Addition of same-signfine
Subtraction of near-equalscancellation
error-reducing
$x
$1/\log x

Multiplication and division never lose relative precision. Subtraction can lose all of it. That single asymmetry explains most of the reformulations in this note — the goal is almost always to rewrite a subtraction away.

Forward and backward error

The framing that makes numerical analysis tractable.

Forward error — how far is my answer from the true answer? What you want to know.

Backward error — for what perturbed input would my answer be exactly right? What you can actually compute.

An algorithm is backward stable if it gives the exact answer to a nearby problem. That’s the realistic goal — and it’s what LAPACK routines guarantee.

The insight this buys you: if your input data is measured to 3 digits anyway, an algorithm with backward error at machine precision is already better than your data deserves. Chasing more accuracy is wasted effort — the uncertainty is in the measurement, not the arithmetic.

Gaussian elimination with partial pivoting is backward stable in practice; without pivoting it isn’t. That’s the entire justification for pivoting. → Linear Systems

Special values in practice

NaN propagates, which is a feature: one NaN anywhere poisons everything downstream, so you find out.

NaN != NaN — the standard test:

if x != x:  # x is NaN

Where NaN comes from: , , , , .

Debugging NaN: enable floating-point exceptions (feenableexcept in C, np.seterr(all='raise') in NumPy) so you trap at the point of creation rather than discovering it a thousand operations later. This is the single most useful NaN-debugging technique and almost nobody uses it.

Infinity is often more useful than an error — it propagates sensibly, and does the right thing.

Denormals near zero can be 10–100× slower on some hardware. Audio and signal-processing code flushes them to zero for this reason.

Practical notes

Use double by default. float has ~7 digits, which sounds fine and isn’t once errors accumulate. Use float deliberately — for memory bandwidth, or for ML where the noise floor is high anyway.

Never test floats for equality. Use a hybrid tolerance.

Never accumulate in the input’s precision. Sum float data into a double accumulator — free, and it removes most of the error.

Prefer expm1, log1p, hypot, fma. They exist because the naive versions lose precision.

Scale your problem. A system mixing and is badly conditioned by construction. Non-dimensionalise — the same advice as state-space scaling.

Test with -ffast-math off. It permits reassociation and assumes no NaN/Inf, so it can silently break carefully-written numerical code. If your results change when you enable it, one of the two versions is wrong.

Check convergence empirically. Halve the step, confirm the error falls as the theory says. It’s the cheapest correctness test available in this whole domain.


↑ Back to Numerical Methods