[Intermediate] — The payoff note. One measure, applied across nine domains already in this vault.
The argument for this track being connective tissue rather than a new silo: almost everything below is already written elsewhere in the vault. This note is the thread running through it.
Cryptography — entropy as security
Key strength is entropy, measured in bits.
| Secret | Entropy |
|---|---|
| 4-digit PIN | 13.3 bits |
| 8-char random alphanumeric | 47.6 bits |
| 12-char random printable ASCII | 78.7 bits |
| 5-word Diceware passphrase | 64.6 bits |
| AES-128 key | 128 bits |
The comparison that matters: a 5-word passphrase (65 bits) is stronger than an 8-character random password (48 bits) and vastly easier to remember. Entropy is what makes that comparison possible rather than a matter of opinion — and it’s why the xkcd 936 argument is correct.
Two things people get wrong:
Entropy is a property of the generation process, not the string. Tr0ub4dor&3 has ~28 bits if generated by “take a word and substitute characters” — the substitution rules are known to attackers. The string looks random; the process wasn’t. → Attacks and Threats
Randomness sources must be cryptographic. rand() seeded from the clock has maybe 20 bits of real entropy regardless of output length. Use /dev/urandom or the OS CSPRNG — and the Debian OpenSSL disaster of 2008 (a patch that reduced key entropy to ~15 bits) is what happens when this goes wrong.
Perfect secrecy has an information-theoretic definition: . The one-time pad achieves it — and Shannon proved it requires a key at least as long as the message, which is why it’s unusable in practice. → Symmetric Encryption
Machine learning — the loss function
Cross-entropy loss, KL divergence, perplexity. Fully covered in note 04 — the short version:
Minimising cross-entropy = maximising likelihood = minimising KL to the data distribution. Three framings, one objective.
Elsewhere in ML:
Decision trees split on information gain, which is mutual information. → Classical ML
Feature selection by mutual information catches nonlinear relevance that correlation misses.
Mutual information neural estimation (MINE), InfoNCE, and contrastive learning all optimise bounds on mutual information.
The information bottleneck frames a good representation as one that compresses the input while preserving what predicts the target.
Active learning picks the sample with maximum expected information gain — which is just “ask the question whose answer you can least predict”.
Compression — and the LLM connection
A model that predicts well compresses well, because they’re the same quantity. → Source Coding
The striking demonstration: DeepMind showed in 2023 that Chinchilla, trained only on text, compresses ImageNet patches better than PNG and audio better than FLAC. Learned structure transfers, and compression measures it.
And practically: gzip + k-NN is a surprisingly competitive text classifier. Classify a document by which training document it compresses best with. No training, no embeddings, and it beat several neural baselines on low-resource benchmarks — a 2023 result that caused some healthy embarrassment.
Networking — capacity and coding
Shannon–Hartley sets the physical layer’s ceiling. → Channel Capacity
Every layer of the stack is information theory:
- Physical — modulation and coding rate chosen against measured SNR
- Link — CRC for error detection, LDPC/turbo for correction → Link Layer
- Transport — TCP infers congestion from loss, which is inference from a very low-bandwidth signal → Congestion Control
- Application — HTTP compression, Brotli’s built-in dictionary
Adaptive bitrate streaming is a rate–distortion problem solved live: estimate the channel, choose the quality level whose rate fits.
Databases and storage
Compression in column stores works because a single column is homogeneous — low entropy given the context, which is exactly why run-length and dictionary encoding get 10× ratios there and not on rows. → Storage and Page Layout
Bloom filters are an information-theoretic trade: accept one-sided error to store far less than the set itself. The optimal bits-per-element for a target false-positive rate is — a hard bound, derived from counting. → Hash Maps
HyperLogLog estimates cardinality in ~1.5 KB regardless of set size, by tracking the maximum number of leading zeros in hashed values. The intuition is pure information theory: seeing leading zeros suggests you’ve drawn about items.
Erasure coding for durability at 40–50% overhead instead of 200%. → Error-Correcting Codes
Security — leakage as a channel
A side channel is a communication channel from a secret to an attacker, and its capacity is measurable in bits per query.
This makes “how bad is this leak” quantitative:
- Timing — a secret-dependent branch leaks; constant-time code targets zero capacity → Spectre
- Compression + encryption — CRIME and BREACH extract secrets by watching compressed length. The leak is real bits, and you can count them → Compression
- Padding oracles — one bit per query, and one bit per query is enough
- Differential privacy bounds information leakage explicitly: the in -DP is a bound on how much any individual’s data can shift the output distribution
Differential privacy is the cleanest example of information theory as a design tool rather than an analysis tool — you choose a leakage budget, then add exactly enough noise to enforce it.
Physics and biology
Landauer’s principle — erasing a bit costs at least of dissipated energy. Measured experimentally in 2012. Information is physical.
Maxwell’s demon is resolved by it: the demon must erase its measurements to keep operating, and that erasure pays back the entropy it appeared to remove.
Black hole entropy is proportional to horizon area, not volume — the origin of the holographic principle, and of the still-unresolved information paradox.
DNA carries ~2 bits per base pair, with error correction: proofreading and mismatch repair reduce the raw error rate by several orders of magnitude. The genetic code is redundant (64 codons, 20 amino acids), and the redundancy is arranged so that common point mutations are more likely to be synonymous — error-tolerant by design.
Neural coding — how many bits does a spike train carry? An active research area using exactly these tools.
Language and cognition
English is ~1.1 bits/character with full context, against 4.7 for uniform letters. ~75% redundant.
That redundancy is functional, not waste: it’s what lets you read text with typos, understand speech in a noisy room, and autocomplete reliably. Natural language has built-in error correction.
Zipf’s law — word frequency inversely proportional to rank — has an information-theoretic reading: it’s close to the distribution minimising average code length given a communication cost, which is why it recurs across languages.
The uniform information density hypothesis proposes that speakers unconsciously spread information evenly across an utterance — contracting predictable words (“I’m” for “I am”) and spelling out surprising ones. There’s decent empirical support, and it’s a nice example of an information-theoretic prediction about behaviour.
Decision-making and inference
Bayesian updating is information acquisition. The expected reduction in entropy from an observation is exactly the mutual information between the observation and the parameter.
Optimal experiment design — choose the experiment maximising expected information gain. → Experiment Design
Twenty questions is optimal binary search, and an optimally-played game extracts 1 bit per question — which is why 20 questions distinguishes ~ possibilities.
Model selection — AIC and MDL (minimum description length) are information-theoretic. MDL’s framing is the cleanest: the best model is the one that most compresses the data, counting the model’s own description length. Occam’s razor as a compression argument, and it’s a genuinely satisfying formalisation.
The thread
One measure — — applied consistently:
| Domain | What it measures |
|---|---|
| Compression | bits needed to store |
| Communication | bits transmittable |
| ML | bits wasted by a wrong model |
| Cryptography | bits of secrecy |
| Security | bits leaked |
| Physics | energy cost of erasure |
| Biology | fidelity of replication |
| Inference | bits gained per observation |
Shannon’s paper was titled A Mathematical Theory of Communication. It turned out to be a mathematical theory of a great deal more, and the reason is that “how surprised should I be” is a question with the same structure everywhere.
Related
- Cross-Entropy and KL Divergence — the ML-facing half
- Channel Capacity — the communications half
- Cryptography — entropy as security
- Information theory map
Software engineering
Infrastructure & security
Computer science
Hardware & engineering