[Advanced] — Finding maxima and minima of a surface, distinguishing them from saddle points, and optimising subject to a constraint.

Before you start

What you will be able to do after this lesson:

  1. Find critical points of a function of two variables.
  2. Classify them using the second derivative test, including saddle points.
  3. Use Lagrange multipliers for constrained optimisation.
  4. Explain why saddle points matter more than local minima in high dimensions.

1. Critical points

At an interior extremum the surface is level in every direction, so every partial derivative vanishes:

Example: .

One critical point, at .

But as in one variable, a critical point need not be an extremum — and in two variables there is a new possibility that has no one-dimensional analogue.

2. The saddle point

At the origin both partials vanish. But along the -axis the surface curves up, and along the -axis it curves down.

   a saddle: minimum in one direction,
             maximum in another

        \         /
         \.     ./
          \`._.'/          along x: a valley
           `._.'           along y: a ridge
          ./   \.
         /       \

It is a minimum and a maximum at the same time, in different directions — hence the name, from the shape of a horse’s saddle. There is nothing like this in one variable, where a critical point is a max, a min, or a flat inflection.

3. The second derivative test

Compute the discriminant:

ConditionConclusion
and local minimum
and local maximum
saddle point
inconclusive — investigate directly

What is measuring: it is the determinant of the Hessian, the matrix of second partials

A negative determinant means the curvatures have opposite signs — up one way, down the other — which is exactly a saddle. That is why the test works, and why it generalises to dimensions via the eigenvalues of .

Applied to : , , , so — a saddle ✓

NOTE

In high dimensions, saddle points dominate.

For a critical point of a function of variables to be a true minimum, the surface must curve upward in all directions at once. If curvature directions were roughly independent, that is like requiring coin flips to all come up heads — vanishingly unlikely for large .

So in a neural network with millions of parameters, almost every critical point is a saddle, not a local minimum. This reversed a long-standing belief: training was assumed to get trapped in bad local minima, and the actual difficulty is escaping the flat regions around saddles — which is why momentum and stochastic noise help so much. See ai-ml.

4. Constrained optimisation and Lagrange multipliers

Often you must optimise subject to a restriction. Maximise area given a fixed perimeter; minimise cost given a required output.

The insight: at the constrained optimum, the contour of is tangent to the constraint curve.

Why: if they crossed rather than touched, you could slide along the constraint and move to a higher contour — so you would not be at the optimum. Tangency means their gradients are parallel:

is the Lagrange multiplier. Solve that vector equation together with the constraint.

Worked example

Maximise subject to .

With the constraint : , giving a maximum product of 25.

The general result — for a fixed sum, the product is largest when the parts are equal — falls straight out, and is more valuable than the number.

also has a meaning: it is the rate at which the optimal value would improve if the constraint were relaxed. In economics that is the shadow price — what one more unit of the constrained resource is worth.

5. Practice — problems

  1. Find and classify the critical points of .
  2. Find and classify the critical points of .
  3. Show that has a saddle at the origin, using the discriminant.
  4. Use Lagrange multipliers to find the rectangle of maximum area with perimeter 20.
  5. Minimise subject to . Interpret the answer geometrically.
  6. Explain why a random critical point of a 1,000,000-parameter function is almost certainly a saddle, in one sentence about curvature directions.
Answers — open only after an attempt
  1. Critical point . , , , so and → local minimum, value .
  2. and give and , so , giving or . Points and . , , . At : → saddle. At : with → local minimum.
  3. , , , so → saddle ✓
  4. Maximise subject to , i.e. — the same problem as the worked example. : a square of area 25. The general result is that the optimal rectangle for a fixed perimeter is always a square.
  5. , so and , giving . With : , , minimum value 5. Geometrically: this is the closest point on the line to the origin, and the line from the origin to is perpendicular to the constraint — which is exactly what says.
  6. Because being a minimum requires upward curvature in all 1,000,000 directions simultaneously, and any single downward direction makes it a saddle — so minima are exponentially rare among critical points.

Before moving on

  • Find critical points by setting the gradient to zero.
  • Classify them with the discriminant, including saddles.
  • Explain what a saddle point is and why one variable has no analogue.
  • Use Lagrange multipliers and explain the tangency argument.

Recap: Critical points of a multivariable function occur where the gradient vanishes, but in two or more variables a new possibility appears — the saddle, which is a minimum in one direction and a maximum in another. The discriminant, which is the determinant of the Hessian, distinguishes the cases, and a negative value means opposite curvatures. In high dimensions saddles vastly outnumber minima, which reshaped how neural network training is understood. Constrained optimisation uses Lagrange multipliers, resting on the observation that at the optimum the objective’s contour is tangent to the constraint.

This is the end of Calculus 3, and of the calculus sequence.

↑ Back to Mathematics