Computing answers to problems that have no closed form — and knowing how wrong the answer is. The gap three domains in this vault explicitly said they needed.
~16,000 words across 12 notes (including practice + solutions). Built August 2026. [reference].
The premise: most equations describing the world can’t be solved exactly. No formula for a general quintic, no antiderivative for , no closed form for Navier–Stokes. So you approximate — and the discipline is producing an answer close enough, fast enough, with an error you can bound.
That last clause is the subject. Anyone can produce a number; the work is knowing how wrong it is.
Why this exists
The vault asked for it by name. engineering/index said “numerical methods — not yet written; FEM and CFD both need it.” Continuum mechanics listed “quadrature, linear solvers, stability analysis” in its gaps. Control theory needs discretisation. ML optimisation runs on conditioning. The original CS-curriculum audit flagged it as “thin”.
| Already written | Was leaning on this |
|---|---|
| FEM | quadrature, sparse solvers, stability |
| digital control | discretisation, ODE integration |
| inverse kinematics | Newton’s method, damped least squares |
| Kalman filters | conditioning, numerical stability |
| ML optimisation | conditioning, line search, convergence |
Reading order
02 is the foundation — everything assumes it. 03–05 are the core, and 04 is the most-used note here because almost everything reduces to a linear solve. 06–07 are approximation. 08–09 are differential equations — what the engineering tracks need. 10 is where ML meets this domain.
- Why Numerical Methods — [Intermediate] — truncation vs rounding error, conditioning vs stability, orders of convergence, and why smaller steps eventually make things worse
- Floating Point and Error — [Intermediate] — machine epsilon, catastrophic cancellation, Kahan summation, forward vs backward error
- Root Finding — [Intermediate] — bisection, Newton, secant, and why Brent’s method is what you should actually call
- Linear Systems — [Intermediate → Advanced] — never invert the matrix. LU, Cholesky, QR, SVD, conditioning, sparse direct vs iterative, preconditioning
- Eigenvalues — [Advanced] — why not the characteristic polynomial, the power method, the QR algorithm, and why
eighbeatseigfor free - Interpolation and Approximation — [Intermediate] — the Runge phenomenon, splines, least squares done properly
- Numerical Integration — [Intermediate] — Newton–Cotes, Gaussian quadrature’s factor-of-two, adaptive rules, Monte Carlo and when dimension forces it
- Ordinary Differential Equations — [Intermediate → Advanced] — RK4, adaptive stepping, stiffness, and why orbital mechanics uses Verlet instead
- Partial Differential Equations — [Advanced] — elliptic/parabolic/hyperbolic, the CFL condition, finite difference vs volume vs element, and the method of manufactured solutions
- Numerical Optimisation — [Intermediate → Advanced] — gradient descent’s conditioning dependence, line search, quasi-Newton, SGD and Adam, convexity
The things worth carrying
- Conditioning is a property of the problem; stability is a property of the algorithm. If your answer is bad, that distinction tells you whether to fix the code or reformulate → 01
- Smaller step sizes eventually make things worse. Truncation error falls, rounding error rises, and the optimum is around → 01
- Multiplication and division never lose relative precision; subtraction of near-equals can lose all of it. Most reformulations in this track are rewriting a subtraction away → 02
- Never invert a matrix. Solve. 3× cheaper, more accurate, and it preserves sparsity → 04
- Never form the normal equations for least squares — it squares the condition number. Use QR or SVD. The most common numerical mistake in ML and statistics code → 04
- means you lose about digits, and a small residual doesn’t imply a small error → 04
- Never fit a high-degree polynomial to equally-spaced data. Runge — more points makes it worse → 06
- An -point Gaussian rule is exact to degree — twice what equal spacing buys. It’s why FEM uses 2×2 Gauss points → 07
- Stiffness means the fastest mode dictates your step even after it’s died out. Switch to an implicit method before optimising anything else → 08
- CFL: information must not travel more than one cell per time step — and the heat equation’s makes explicit refinement brutally expensive → 09
- Slow gradient descent is usually a conditioning problem, not a learning-rate problem. Which is why feature scaling matters → 10
- Method of manufactured solutions — pick the answer, derive the source term, test the whole code against it. The most rigorous verification available → 09
Where this connects
| control theory · continuum mechanics | The domain that asked. FEM, CFD, control discretisation |
| ML optimisation | Note 10 is the numerical view of the same subject |
| data representation | IEEE 754, which note 02 builds on |
| robotics/ | IK, Kalman filtering, dynamics integration |
| GPU and parallel | Where the big linear algebra actually runs |
The honest note
[reference], and this domain has a specific failure mode: it is very easy to produce a plausible number that is wrong.
Unlike a compiler error or a failing test, a bad numerical result looks exactly like a good one. The whole discipline is about not being fooled by that, and reading about it doesn’t build the instinct.
The good news is that the reps here are cheap — every one of these is a short script:
- Plot the finite-difference error U-curve. Compute at down to and plot the error. The V shape is the whole of note 01 in one picture, and it takes five minutes
- Break the quadratic formula. Solve naively and with the stable form. Compare against the true roots
- Watch conditioning bite. Build a Hilbert matrix ( at ), solve, and measure how many digits survive
- Reproduce Runge’s phenomenon. Interpolate at 5, 10, 20 equally-spaced points and plot. Then do it at Chebyshev nodes and watch it converge
- Hit a stiffness wall. Integrate a stiff system with
RK45and withBDF. Compare step counts — it’s usually orders of magnitude - Do a convergence study. Halve the step on anything here and check the error falls at the promised rate. This is the single most useful habit in the domain
- The books: Trefethen & Bau’s Numerical Linear Algebra (outstanding, and readable); Press et al.’s Numerical Recipes (dated on some algorithms, excellent on judgement); LeVeque for finite differences and PDEs; Nocedal & Wright for optimisation
What’s missing here: exercises — closed by notes 11–12 (Aug 2026); complex analysis and contour methods, FFT and spectral methods in depth (only mentioned), interval arithmetic, arbitrary-precision arithmetic, numerical linear algebra at scale (communication-avoiding algorithms), and uncertainty quantification.
Practice
- Practice Exercises — twelve short scripts — the finite-difference U-curve, catastrophic cancellation, Hilbert conditioning, Runge’s phenomenon, convergence studies
- Solutions — worked answers, after you’ve tried
Related
- control theory · continuum mechanics — the domain that needed this
- Data Representation — the floating point underneath
- The maths notes — linear algebra and calculus at the level ML needs
- Build Plan
Software engineering
Infrastructure & security
Computer science
Hardware & engineering