0 of 14 done
Start hereWhy Numerical Methods

Computing answers to problems that have no closed form — and knowing how wrong the answer is. The gap three domains in this vault explicitly said they needed.

~16,000 words across 12 notes (including practice + solutions). Built August 2026. [reference].

The premise: most equations describing the world can’t be solved exactly. No formula for a general quintic, no antiderivative for , no closed form for Navier–Stokes. So you approximate — and the discipline is producing an answer close enough, fast enough, with an error you can bound.

That last clause is the subject. Anyone can produce a number; the work is knowing how wrong it is.

Why this exists

The vault asked for it by name. engineering/index said “numerical methods — not yet written; FEM and CFD both need it.” Continuum mechanics listed “quadrature, linear solvers, stability analysis” in its gaps. Control theory needs discretisation. ML optimisation runs on conditioning. The original CS-curriculum audit flagged it as “thin”.

Already writtenWas leaning on this
FEMquadrature, sparse solvers, stability
digital controldiscretisation, ODE integration
inverse kinematicsNewton’s method, damped least squares
Kalman filtersconditioning, numerical stability
ML optimisationconditioning, line search, convergence

Reading order

02 is the foundation — everything assumes it. 03–05 are the core, and 04 is the most-used note here because almost everything reduces to a linear solve. 06–07 are approximation. 08–09 are differential equations — what the engineering tracks need. 10 is where ML meets this domain.

  1. Why Numerical Methods — [Intermediate] — truncation vs rounding error, conditioning vs stability, orders of convergence, and why smaller steps eventually make things worse
  2. Floating Point and Error — [Intermediate] — machine epsilon, catastrophic cancellation, Kahan summation, forward vs backward error
  3. Root Finding — [Intermediate] — bisection, Newton, secant, and why Brent’s method is what you should actually call
  4. Linear Systems — [Intermediate → Advanced] — never invert the matrix. LU, Cholesky, QR, SVD, conditioning, sparse direct vs iterative, preconditioning
  5. Eigenvalues — [Advanced] — why not the characteristic polynomial, the power method, the QR algorithm, and why eigh beats eig for free
  6. Interpolation and Approximation — [Intermediate] — the Runge phenomenon, splines, least squares done properly
  7. Numerical Integration — [Intermediate] — Newton–Cotes, Gaussian quadrature’s factor-of-two, adaptive rules, Monte Carlo and when dimension forces it
  8. Ordinary Differential Equations — [Intermediate → Advanced] — RK4, adaptive stepping, stiffness, and why orbital mechanics uses Verlet instead
  9. Partial Differential Equations — [Advanced] — elliptic/parabolic/hyperbolic, the CFL condition, finite difference vs volume vs element, and the method of manufactured solutions
  10. Numerical Optimisation — [Intermediate → Advanced] — gradient descent’s conditioning dependence, line search, quasi-Newton, SGD and Adam, convexity

The things worth carrying

  1. Conditioning is a property of the problem; stability is a property of the algorithm. If your answer is bad, that distinction tells you whether to fix the code or reformulate → 01
  2. Smaller step sizes eventually make things worse. Truncation error falls, rounding error rises, and the optimum is around → 01
  3. Multiplication and division never lose relative precision; subtraction of near-equals can lose all of it. Most reformulations in this track are rewriting a subtraction away → 02
  4. Never invert a matrix. Solve. 3× cheaper, more accurate, and it preserves sparsity → 04
  5. Never form the normal equations for least squares — it squares the condition number. Use QR or SVD. The most common numerical mistake in ML and statistics code → 04
  6. means you lose about digits, and a small residual doesn’t imply a small error → 04
  7. Never fit a high-degree polynomial to equally-spaced data. Runge — more points makes it worse → 06
  8. An -point Gaussian rule is exact to degree — twice what equal spacing buys. It’s why FEM uses 2×2 Gauss points → 07
  9. Stiffness means the fastest mode dictates your step even after it’s died out. Switch to an implicit method before optimising anything else → 08
  10. CFL: information must not travel more than one cell per time step — and the heat equation’s makes explicit refinement brutally expensive → 09
  11. Slow gradient descent is usually a conditioning problem, not a learning-rate problem. Which is why feature scaling matters → 10
  12. Method of manufactured solutions — pick the answer, derive the source term, test the whole code against it. The most rigorous verification available → 09

Where this connects

control theory · continuum mechanicsThe domain that asked. FEM, CFD, control discretisation
ML optimisationNote 10 is the numerical view of the same subject
data representationIEEE 754, which note 02 builds on
robotics/IK, Kalman filtering, dynamics integration
GPU and parallelWhere the big linear algebra actually runs

The honest note

[reference], and this domain has a specific failure mode: it is very easy to produce a plausible number that is wrong.

Unlike a compiler error or a failing test, a bad numerical result looks exactly like a good one. The whole discipline is about not being fooled by that, and reading about it doesn’t build the instinct.

The good news is that the reps here are cheap — every one of these is a short script:

  1. Plot the finite-difference error U-curve. Compute at down to and plot the error. The V shape is the whole of note 01 in one picture, and it takes five minutes
  2. Break the quadratic formula. Solve naively and with the stable form. Compare against the true roots
  3. Watch conditioning bite. Build a Hilbert matrix ( at ), solve, and measure how many digits survive
  4. Reproduce Runge’s phenomenon. Interpolate at 5, 10, 20 equally-spaced points and plot. Then do it at Chebyshev nodes and watch it converge
  5. Hit a stiffness wall. Integrate a stiff system with RK45 and with BDF. Compare step counts — it’s usually orders of magnitude
  6. Do a convergence study. Halve the step on anything here and check the error falls at the promised rate. This is the single most useful habit in the domain
  7. The books: Trefethen & Bau’s Numerical Linear Algebra (outstanding, and readable); Press et al.’s Numerical Recipes (dated on some algorithms, excellent on judgement); LeVeque for finite differences and PDEs; Nocedal & Wright for optimisation

What’s missing here: exercises — closed by notes 11–12 (Aug 2026); complex analysis and contour methods, FFT and spectral methods in depth (only mentioned), interval arithmetic, arbitrary-precision arithmetic, numerical linear algebra at scale (communication-avoiding algorithms), and uncertainty quantification.

→ Reading is not a rank.

Practice

  • Practice Exercises — twelve short scripts — the finite-difference U-curve, catastrophic cancellation, Hilbert conditioning, Runge’s phenomenon, convergence studies
  • Solutions — worked answers, after you’ve tried
↑ Back to Mathematics