[Intermediate → Advanced] — Differentiation when a function has several inputs. The idea behind every gradient in machine learning.

Before you start

  • You can differentiate fluently, especially the chain rule — derivative rules.
  • You know what a function of one variable is — variables.

What you will be able to do after this lesson:

  1. Compute a partial derivative and say precisely what it measures.
  2. Explain what the gradient vector is and why it points uphill.
  3. Apply the multivariable chain rule.
  4. Explain why gradient descent works, in terms of the gradient’s direction.

1. Several inputs, one output

This describes a surface, not a curve — a height above every point of the -plane.

“How steep is it?” is now ambiguous. Steep in which direction? Walking east may be uphill while walking north is downhill.

So the question must be made specific. A partial derivative answers it for one axis at a time.

2. The definition

Note that does not change. The partial derivative with respect to measures the rate of change as varies and everything else is held fixed.

Which makes it trivially easy to compute: treat every other variable as a constant and differentiate normally.

For :

In the first, differentiates to because is a constant multiplier, and differentiates to zero because it contains no at all.

The symbol (“del”) rather than signals that other variables exist and are being held still.

   the surface z = f(x,y)

   ∂f/∂x  = the slope of the slice taken parallel to the x-axis
   ∂f/∂y  = the slope of the slice taken parallel to the y-axis

   each partial derivative is an ORDINARY derivative
   of a one-variable slice through the surface

3. The gradient

Collect the partials into a vector:

Two facts make this the central object of the subject:

  1. points in the direction of steepest increase.
  2. Its magnitude is how steep that steepest direction is.

Why direction 1 is true, informally: moving a small step changes by approximately — a dot product of the gradient with your step. A dot product is largest when the two vectors point the same way, so the step that increases most is the one aligned with .

Example: for at :

Steepest ascent from that point is in direction , and the steepest slope is .

4. Why this is the foundation of machine learning

Training a model means minimising a loss function with respect to millions of parameters.

The gradient points uphill, so points downhill — the direction that reduces the loss fastest. That gives the entire algorithm:

Take a small step against the gradient, recompute, repeat. That is gradient descent, and is the learning rate — how far to step.

NOTE

Two things follow immediately from this being a local rule.

It can get stuck. The gradient only knows the slope where you are standing. In a local minimum every direction is uphill, so the rule stops — even though a deeper valley may exist elsewhere.

The step size matters enormously. Too small and training crawls; too large and you overshoot the valley and may diverge. Everything about learning-rate schedules, momentum and adaptive optimisers exists to manage that one tension.

See ai-ml, and build your own neural network where you compute these gradients by hand.

5. Higher-order and mixed partials

Differentiate twice:

Clairaut’s theorem: for a well-behaved function, the mixed partials are equal:

The order of differentiation does not matter. That is a genuinely surprising result — there is no obvious reason changing then should match changing then — and it fails for pathological functions, which is what “well-behaved” is doing in the statement.

6. The multivariable chain rule

If and both and depend on :

Read it as: total change is the sum of the contributions through each path. influences via and also via , so both routes contribute and they add.

This is exactly backpropagation. In a network, a weight influences the loss through every downstream path, and the total gradient is the sum over all of them. The single-variable chain rule multiplies along a path; the multivariable one adds across paths — and a neural network needs both.

7. Practice — problems

  1. Find both partials: (a) (b) (c) .
  2. Find for at . What is the steepest slope there, and in which direction?
  3. Explain why at a maximum, a minimum and a saddle point — and what extra information distinguishes them.
  4. Verify Clairaut’s theorem for by computing both mixed partials.
  5. For with , , find using the chain rule. Then explain the answer geometrically.
  6. Gradient descent by hand. Minimise starting at with . Compute three steps. What happens if instead?
Answers — open only after an attempt
  1. (a) , . (b) , . (c) , .
  2. . Steepest slope , in the direction — which points directly away from the origin, as expected for a bowl.
  3. All three are stationary points: the surface is locally flat, so every partial is zero. The gradient alone cannot tell them apart. The second derivatives distinguish them — a saddle curves up in one direction and down in another, which the Hessian matrix captures.
  4. . . Equal ✓
  5. . Geometrically: on the unit circle, so is constant — you are walking along a contour line, and moving along a contour changes nothing.
  6. . Steps: ; ; — converging towards 0. With : , then — oscillating and growing. The steps overshoot the minimum by more than they started from, so it diverges.

Before moving on

  • Compute partial derivatives and say what is held constant.
  • Explain what the gradient is and why it points uphill.
  • Apply the multivariable chain rule and explain the summation across paths.
  • Explain gradient descent, and both ways it can fail.

Recap: A partial derivative measures the rate of change along one axis with all other variables held fixed, so it is computed by treating them as constants. Collecting the partials gives the gradient, which points in the direction of steepest increase with magnitude equal to that steepness — which is why stepping against it minimises a function, and why gradient descent is the algorithm behind model training. The multivariable chain rule adds contributions across paths, and applying it through a deep composition is backpropagation.

Next: Multiple Integrals — accumulation over a region rather than an interval.

↑ Back to Mathematics