[Intermediate] — What one variable tells you about another. The toolkit, and the one Venn diagram worth memorising.
Joint entropy
The uncertainty in two variables together:
Bounded on both sides:
The upper bound holds with equality exactly when and are independent. If knowing one tells you nothing about the other, the uncertainties simply add.
The lower bound is reached when one determines the other — means adds nothing.
Conditional entropy
The uncertainty remaining in once you know :
The chain rule follows immediately:
Read it as: total uncertainty = uncertainty in , plus whatever’s left in once you know . It’s just decomposing the joint uncertainty in stages, and it generalises to any number of variables.
Two facts worth carrying:
On average, information never hurts. Knowing something can only reduce your uncertainty, or leave it unchanged.
But note “on average”. A particular observation can increase your uncertainty — can exceed for some specific . It’s the expectation over that’s guaranteed to decrease. This trips people up, and it’s the difference between “this measurement surprised me and left me more confused” and “measurements are useless”.
Mutual information
The central quantity in the subject.
How much does knowing one reduce your uncertainty about the other?
Equivalent forms:
The Venn diagram, which is genuinely worth memorising:
┌───────────────────────────┐
│ H(X,Y) │
│ ┌──────────┬──────────┐ │
│ │ H(X|Y) │ H(Y|X) │ │
│ │ ┌───┼───┐ │ │
│ │ │ I(X;Y)│ │ │
│ │ └───┼───┘ │ │
│ └──────────┴──────────┘ │
│ H(X) H(Y) │
└───────────────────────────┘
Every identity in this note is a statement about that picture, which is why it’s the fastest way to remember them.
The properties that make it useful:
Symmetric. — what tells you about equals what tells you about . Not obvious, and it falls straight out of the joint form.
Non-negative. , with equality iff and are independent.
A general measure of dependence. Unlike correlation, it captures nonlinear relationships.
This is mutual information’s main practical advantage over correlation. For with symmetric about zero, the correlation is exactly 0 and the mutual information is large. Correlation measures linear association; MI measures any association.
The cost: MI is much harder to estimate from finite samples, needs binning or a density estimator for continuous variables, and has no natural sign or scale.
Where these get used
Feature selection. Rank features by — sklearn.feature_selection.mutual_info_classif. Catches nonlinear relevance that a correlation filter misses.
The known weakness: it’s marginal. Two features individually uninformative can be jointly predictive (XOR is the standard example), and pairwise MI ranking will discard both. mRMR (max-relevance, min-redundancy) partially addresses this.
Decision trees. Information gain is mutual information — at each split, choose the feature maximising , i.e. the one that most reduces label entropy. ID3, C4.5, and the entropy criterion in scikit-learn. → Classical ML
(Gini impurity is the common alternative and behaves similarly — it’s a second-order approximation to entropy, and cheaper because there’s no logarithm.)
Channel capacity. — capacity is the mutual information between input and output, maximised over input distributions. → Channel Capacity
Representation learning. The information bottleneck frames learning as compressing while preserving information about : minimise . InfoNCE and contrastive methods maximise a lower bound on mutual information between views. → Autoencoders
Image registration. Mutual information between two images is the standard alignment metric in medical imaging — it works across modalities (aligning an MRI to a CT) where pixel intensities aren’t comparable at all but are statistically dependent.
Side-channel analysis. quantifies exactly how much a timing channel leaks. Zero mutual information is what constant-time code is trying to achieve, and this makes “how bad is this leak” a measurable question. → Cryptographic Attacks
Conditional mutual information
What tells you about that didn’t already.
The key subtlety, and it’s a real trap:
can be larger than . Conditioning can create dependence.
The classic case: and are independent fair coins, and . Then — they’re independent. But — given the XOR, knowing one tells you the other exactly.
This is the “explaining away” effect, and it’s why causal inference can’t be done by looking at correlations alone. Conditioning on a common effect (a collider) induces dependence between independent causes. → Causal Inference
Practical consequence: “controlling for more variables is always more rigorous” is false. Conditioning on a collider introduces bias rather than removing it, and this is a live problem in applied statistics.
The data processing inequality
A short result with wide consequences.
For a Markov chain (where depends on only through ):
Post-processing cannot create information. Once you’ve mapped to , no further computation on recovers anything about that didn’t already contain.
Where this bites:
No feature engineering creates information. It reshapes what’s there into a form a model can use more easily — genuinely valuable — but the information ceiling is set by the raw data. If your features don’t contain the signal, no transformation will conjure it.
Every layer of a neural network can only lose information about the input. (Which sounds alarming and isn’t — the goal is to discard the irrelevant information while preserving what predicts . That’s the information bottleneck view of what training does.)
Lossy compression is irreversible. Obviously, but now it’s a theorem.
Anonymisation is one-way, which is the reassuring version — though the inequality says nothing about how much information survives, which is why re-identification attacks work.
Entropy for continuous variables
Differential entropy replaces the sum with an integral:
And it behaves badly, in ways worth knowing before you use it:
It can be negative. A uniform distribution on has bit. Not a bug — differential entropy isn’t the limit of discrete entropy, which diverges.
It’s not invariant under change of variables. Rescaling changes by . So it has no absolute meaning — measure in metres versus centimetres and you get a different number.
But mutual information is well-behaved for continuous variables — it’s a difference of differential entropies, and the scaling terms cancel. is invariant under invertible transformations of either variable, which is exactly what you want from a dependence measure. So use MI, not differential entropy, when working with continuous data.
Two useful facts:
The Gaussian maximises entropy for a given variance. . This is the information-theoretic justification for the Gaussian as the “least assumptive” distribution given a known mean and variance — the maximum-entropy principle.
For a fixed range, the uniform distribution maximises entropy. Same principle, different constraint.
Practical notes
Estimating entropy from samples is biased downward. With few samples you systematically underestimate — you haven’t seen the rare events. Miller–Madow and other corrections exist; the naive plug-in estimator is optimistic.
Binning choices dominate continuous MI estimates. Too few bins loses structure; too many makes everything look dependent. Prefer k-NN estimators (Kraskov) or kernel methods over naive histograms.
Units matter. Bits () or nats (). ML libraries almost always use nats; be consistent, since a factor of silently corrupts comparisons.
Sanity-check against the bounds. . A negative MI estimate means your estimator is broken, since the true value can’t be.
Related
- Cross-Entropy and KL Divergence — the ML-facing measures
- Channel Capacity — mutual information, maximised
- Causal Inference — where conditioning-on-colliders bites
- Information theory map
Software engineering
Infrastructure & security
Computer science
Hardware & engineering