[Intermediate] — The two quantities used on every page of the ML notes and explained on none of them. What your loss function actually measures.

This is the note the rest of the vault was missing. ai-ml/ is 98 notes; cross-entropy loss appears throughout and KL divergence underlies VAEs, RLHF and distillation. Here’s what they are.

Cross-entropy

Entropy assumes you know the true distribution . What if you use a model instead?

Cross-entropy is the average number of bits needed to encode data from using a code optimised for .

You pay for the mismatch. Code as though the distribution were , but reality draws from , and your messages are longer than they had to be.

The key inequality:

Your model can never beat the true distribution. The best possible model is the truth, and cross-entropy is minimised exactly there.

Which is why cross-entropy is a loss function. Minimising over drives toward . It has a unique minimum at the truth, and that’s precisely what you want a loss to do.

KL divergence

The excess — how many bits you waste using instead of :

Cross-entropy = the irreducible entropy of the data + the penalty for your model being wrong.

Properties:

, zero iff (Gibbs’ inequality).

Not symmetric. — so it is not a distance, despite constant informal use as one.

Not a metric — no triangle inequality either.

The asymmetry matters

This is the part that has real consequences, and it’s frequently glossed over.

Forward KL, — “mean-seeking” / mass-covering.

The term blows up wherever is large and is near zero. So is heavily penalised for putting no mass where has some — it must cover everything, even if that means spreading over regions doesn’t occupy.

Reverse KL, — “mode-seeking” / zero-forcing.

Now the expectation is over . Wherever is near zero, that region contributes nothing regardless of . So can safely ignore parts of — it will lock onto one mode and stay there.

 True p is bimodal:      ╱╲      ╱╲
                        ╱  ╲    ╱  ╲

 Forward KL fit:      ╱‾‾‾‾‾‾‾‾‾‾‾‾╲     covers both, mass in between
 Reverse KL fit:        ╱╲                picks one mode, ignores the other
                       ╱  ╲

Where each shows up:

Forward KL — maximum likelihood, which is what standard supervised training does. You want to cover the data.

Reverse KL — variational inference (the VAE’s ELBO), and RLHF’s KL penalty, where you want the policy to stay near the reference model without being forced to cover its whole distribution.

Mode collapse in generative models is partly this. A reverse-KL objective is content to model one mode well. Understanding which direction you’re minimising tells you what failure to expect. → Autoencoders and GANs

Jensen–Shannon divergence symmetrises it — with the average. Bounded, symmetric, and its square root is a true metric. The original GAN objective minimises JS divergence, and its vanishing gradients when distributions don’t overlap is why Wasserstein GANs were proposed.

Why your loss function is cross-entropy

The connection that makes this note load-bearing.

For classification with one-hot labels, puts probability 1 on the true class and 0 elsewhere. Cross-entropy collapses to:

Just the negative log-probability your model assigned to the correct answer.

Averaged over a dataset:

That is exactly the negative log-likelihood.

All three are the same objective. Whichever framing you started from — information theory, statistics, or “make the model assign high probability to the data” — you arrive at the same loss. That convergence is why cross-entropy is the classification loss rather than one option among many.

Note is a constant with respect to your parameters (it’s a property of the data), so minimising cross-entropy and minimising KL are identical optimisation problems. People say “cross-entropy loss” and “minimising KL to the data distribution” interchangeably for this reason.

Binary cross-entropy is the two-class case:

Perplexity

The language-modelling metric, and it’s just exponentiated cross-entropy:

Interpretation: the effective number of equally-likely choices the model is deciding among at each step.

Perplexity 1 — perfect prediction. Perplexity 50,000 — no better than uniform over the vocabulary.

Why report perplexity instead of loss: it’s on a meaningful scale. Going from loss 4.0 to 3.0 sounds modest; going from perplexity 55 to 20 is obviously a large improvement.

The caveat that makes cross-model comparison treacherous: perplexity depends on the tokeniser. A model with a larger vocabulary predicts fewer, larger tokens, and its perplexity isn’t comparable to one with a smaller vocabulary. Bits-per-character normalises this and is the honest comparison. → How LLMs Work

Compression is prediction

The equivalence that ties this track together.

A language model outputs . Feed that to an arithmetic coder and you have a compressor achieving bits per token.

They are the same quantity. Training a language model is training a compressor, and the loss curve is a compression-ratio curve in different units.

This isn’t a cute analogy — it’s exploited directly:

The Hutter Prize pays for compressing 1 GB of Wikipedia, explicitly on the premise that compression is equivalent to intelligence.

Large language models are excellent general-purpose compressors — DeepMind’s 2023 result showed Chinchilla compressing ImageNet patches better than PNG and audio better than FLAC, despite being trained only on text. The model learned enough structure to predict, and prediction is compression.

And it’s the strongest argument for what the pretraining objective does: to predict the next token well, a model must learn syntax, facts, arithmetic, and reasoning patterns — because those are what make text predictable. Compression forces modelling.

Practical notes

Never implement cross-entropy naively.

loss = -np.log(softmax(logits)[y])      # ✗ overflows / underflows
loss = F.cross_entropy(logits, y)       # ✓ fused, numerically stable

softmax then log overflows for large logits and underflows for small probabilities. The fused implementation uses the log-sum-exp trick:

PyTorch’s cross_entropy takes raw logits and does this internally. Passing it post-softmax probabilities is a common and silent bug — you apply softmax twice, gradients shrink, and training just underperforms without erroring. → Numerical stability

Zero probabilities give infinite loss. . Clamp, or use label smoothing.

Label smoothing replaces one-hot targets with on the true class and elsewhere. Prevents the model driving logits to infinity chasing certainty, improves calibration, and usually generalises slightly better.

Watch your units. ML libraries use nats (); information theory conventionally uses bits (). Factor of .

KL in VAEs has a closed form for Gaussians — no sampling needed:

Estimating KL from samples is high-variance. For the reverse direction, use the low-variance estimator with , rather than the naive — this is what modern RLHF implementations use and it materially reduces gradient noise.

Cross-entropy is not calibration. A model can have low cross-entropy and still be overconfident. Check reliability diagrams and expected calibration error separately. → Model Evaluation


↑ Back to Information Theory