[Intermediate] — The two paradigms, why one won for real-time and the other for film, and what they’re both approximating.

Source: [reference] — see the domain note.

The problem

Given a description of a 3D scene, produce a 2D image.

The scene: geometry (triangles, usually), materials, lights, and a camera.

The image: a grid of pixels, each with a colour.

Two fundamentally different ways to bridge that gap, and the difference is which direction you iterate.

Rasterisation — loop over geometry

for each triangle:
    project to screen space
    for each pixel it covers:
        compute colour, keep if nearest

Object-order. You ask “where does this triangle land?”

The killer property: it’s embarrassingly parallel over triangles and pixels, and it never needs the whole scene at once — stream triangles through, and each is independent.

Which is why it’s what GPUs were built for. The entire fixed-function graphics pipeline is a rasteriser in silicon. → Rasterisation

What it’s bad at: anything requiring a triangle to know about other triangles. Shadows, reflections, refraction and indirect light are all global effects, and rasterisation is fundamentally local. Every real-time technique for those is an approximation bolted on — shadow maps, environment maps, screen-space reflections.

Ray tracing — loop over pixels

for each pixel:
    cast a ray from the camera
    find the nearest intersection
    compute colour (possibly casting more rays)

Image-order. You ask “what does this pixel see?”

The killer property: global effects are natural. A reflection is just another ray. A shadow is a ray toward the light. Refraction, depth of field and motion blur all fall out of casting the right rays.

What it’s bad at: you need the whole scene queryable at once, and naive intersection is per ray. Acceleration structures (BVH) fix the complexity; the memory requirement is inherent. → Ray Tracing

The honest summary: rasterisation is fast and local; ray tracing is slow and global. Real-time rendering spent thirty years finding clever approximations of global effects within a local framework — and hardware ray tracing (2018 onwards) is the beginning of that pressure easing.

Modern engines are hybrid: rasterise primary visibility (fast, and every pixel needs it), ray-trace the effects that need global information.

The rendering equation

What both approaches are approximating. Kajiya, 1986:

In words: light leaving a point in a direction = light it emits + all incoming light, weighted by how the surface reflects it and by the angle.

TermMeaning
outgoing radiance — what you want
emitted (only for light sources)
BRDF — how the material reflects → Shading
incoming radiance from everywhere
Lambert’s cosine — glancing light contributes less

The difficulty is that depends on at every other surface. It’s a recursive integral equation over an infinite-dimensional space — light bounces forever.

Every rendering algorithm ever written is a way of approximating this integral. Rasterised direct lighting truncates it at one bounce with no integral at all. Path tracing estimates it by Monte Carlo. Knowing the equation tells you exactly what each technique is throwing away.

And note the integral is high-dimensional, which is why Monte Carlo is the method of choice — quadrature dies above ~4 dimensions and Monte Carlo’s error is dimension-independent. → Monte Carlo

The real-time pipeline

What happens for each frame at 60 Hz:

 scene data
     │
     ▼  VERTEX PROCESSING   transform to clip space
     ▼  CLIPPING & CULLING  discard what's invisible
     ▼  RASTERISATION       triangles → fragments
     ▼  FRAGMENT SHADING    compute colour per fragment
     ▼  OUTPUT MERGING      depth test, blending
  framebuffer

Covered in detail across notes 09–13. The structure to notice: it’s a stream, with each stage massively parallel and no stage needing global scene information.

Rendering as sampling

A framing that unifies a surprising amount, and connects to information theory.

A pixel is not a point — it’s an area. Its correct colour is the average of the image over that area:

Rendering is estimating that integral by sampling. And sampling theory then explains most visual artefacts:

Aliasing — sampling below the Nyquist rate. Jagged edges, shimmering textures, moiré patterns. Fixed by supersampling (more samples), mipmapping (pre-filtering), or reconstruction filters. → Textures and Sampling

Noise — too few Monte Carlo samples. Path tracing’s grainy look, and it decreases as , which is why film renders take hours.

Banding — quantisation error from too few bits per channel. Dithering trades it for noise, which the eye tolerates better.

Noise and aliasing are the same problem sampled differently. Regular sampling produces structured error (aliasing — visually objectionable because the eye finds patterns). Random sampling produces unstructured error (noise — less objectionable). Stratified and blue-noise sampling deliberately sit between them, which is why blue-noise dithering looks so much better than white noise at the same error level.

Where graphics connects

The reason this domain earns its place here, given it’s the furthest from what you build:

GPU programming. Graphics is why GPUs exist, and the compute model is the graphics model generalised. Understanding the pipeline explains why GPUs are shaped the way they are.

Linear algebra and transforms. The same homogeneous transforms as robotics — same matrices, different application. Learning one gives you the other.

Numerical methods. Monte Carlo integration, interpolation, splines and ODE integration for physics. → Numerical Methods

Signal processing. Sampling, filtering, aliasing, reconstruction.

Simulation and robotics. Physics engines, collision detection, and rendering for robot simulators — MuJoCo, Isaac Sim and Gazebo all need this.

Machine learning. Differentiable rendering, NeRFs and Gaussian splatting are graphics and ML fused, and the rendering equation is what they’re learning to invert.

Reading order

This note opens Part 2. It assumes Part 1 (notes 01–07): how a screen shows pixels, drawing lines and shapes, and moving shapes with matrices. If those are new, start at note 01.

09–13 are the steps a graphics card goes through and build in order: placing objects, filling in triangles, lighting, textures, then how the card is programmed.

14 is ray tracing and can be read on its own.

15–16 are shapes and movement.

Prerequisites: linear algebra — vectors, matrices, dot and cross products — is essential. Trigonometry. Rigid body transforms covers most of note 09 already if you’ve read it.


↑ Back to Computer Graphics