[Intermediate] — The payoff note. One measure, applied across nine domains already in this vault.

The argument for this track being connective tissue rather than a new silo: almost everything below is already written elsewhere in the vault. This note is the thread running through it.

Cryptography — entropy as security

Key strength is entropy, measured in bits.

SecretEntropy
4-digit PIN13.3 bits
8-char random alphanumeric47.6 bits
12-char random printable ASCII78.7 bits
5-word Diceware passphrase64.6 bits
AES-128 key128 bits

The comparison that matters: a 5-word passphrase (65 bits) is stronger than an 8-character random password (48 bits) and vastly easier to remember. Entropy is what makes that comparison possible rather than a matter of opinion — and it’s why the xkcd 936 argument is correct.

Two things people get wrong:

Entropy is a property of the generation process, not the string. Tr0ub4dor&3 has ~28 bits if generated by “take a word and substitute characters” — the substitution rules are known to attackers. The string looks random; the process wasn’t. → Attacks and Threats

Randomness sources must be cryptographic. rand() seeded from the clock has maybe 20 bits of real entropy regardless of output length. Use /dev/urandom or the OS CSPRNG — and the Debian OpenSSL disaster of 2008 (a patch that reduced key entropy to ~15 bits) is what happens when this goes wrong.

Perfect secrecy has an information-theoretic definition: . The one-time pad achieves it — and Shannon proved it requires a key at least as long as the message, which is why it’s unusable in practice. → Symmetric Encryption

Machine learning — the loss function

Cross-entropy loss, KL divergence, perplexity. Fully covered in note 04 — the short version:

Minimising cross-entropy = maximising likelihood = minimising KL to the data distribution. Three framings, one objective.

Elsewhere in ML:

Decision trees split on information gain, which is mutual information. → Classical ML

Feature selection by mutual information catches nonlinear relevance that correlation misses.

Mutual information neural estimation (MINE), InfoNCE, and contrastive learning all optimise bounds on mutual information.

The information bottleneck frames a good representation as one that compresses the input while preserving what predicts the target.

Active learning picks the sample with maximum expected information gain — which is just “ask the question whose answer you can least predict”.

Compression — and the LLM connection

A model that predicts well compresses well, because they’re the same quantity. → Source Coding

The striking demonstration: DeepMind showed in 2023 that Chinchilla, trained only on text, compresses ImageNet patches better than PNG and audio better than FLAC. Learned structure transfers, and compression measures it.

And practically: gzip + k-NN is a surprisingly competitive text classifier. Classify a document by which training document it compresses best with. No training, no embeddings, and it beat several neural baselines on low-resource benchmarks — a 2023 result that caused some healthy embarrassment.

Networking — capacity and coding

Shannon–Hartley sets the physical layer’s ceiling. → Channel Capacity

Every layer of the stack is information theory:

  • Physical — modulation and coding rate chosen against measured SNR
  • Link — CRC for error detection, LDPC/turbo for correction → Link Layer
  • Transport — TCP infers congestion from loss, which is inference from a very low-bandwidth signal → Congestion Control
  • Application — HTTP compression, Brotli’s built-in dictionary

Adaptive bitrate streaming is a rate–distortion problem solved live: estimate the channel, choose the quality level whose rate fits.

Databases and storage

Compression in column stores works because a single column is homogeneous — low entropy given the context, which is exactly why run-length and dictionary encoding get 10× ratios there and not on rows. → Storage and Page Layout

Bloom filters are an information-theoretic trade: accept one-sided error to store far less than the set itself. The optimal bits-per-element for a target false-positive rate is — a hard bound, derived from counting. → Hash Maps

HyperLogLog estimates cardinality in ~1.5 KB regardless of set size, by tracking the maximum number of leading zeros in hashed values. The intuition is pure information theory: seeing leading zeros suggests you’ve drawn about items.

Erasure coding for durability at 40–50% overhead instead of 200%. → Error-Correcting Codes

Security — leakage as a channel

A side channel is a communication channel from a secret to an attacker, and its capacity is measurable in bits per query.

This makes “how bad is this leak” quantitative:

  • Timing — a secret-dependent branch leaks; constant-time code targets zero capacity → Spectre
  • Compression + encryption — CRIME and BREACH extract secrets by watching compressed length. The leak is real bits, and you can count them → Compression
  • Padding oracles — one bit per query, and one bit per query is enough
  • Differential privacy bounds information leakage explicitly: the in -DP is a bound on how much any individual’s data can shift the output distribution

Differential privacy is the cleanest example of information theory as a design tool rather than an analysis tool — you choose a leakage budget, then add exactly enough noise to enforce it.

Physics and biology

Landauer’s principle — erasing a bit costs at least of dissipated energy. Measured experimentally in 2012. Information is physical.

Maxwell’s demon is resolved by it: the demon must erase its measurements to keep operating, and that erasure pays back the entropy it appeared to remove.

Black hole entropy is proportional to horizon area, not volume — the origin of the holographic principle, and of the still-unresolved information paradox.

DNA carries ~2 bits per base pair, with error correction: proofreading and mismatch repair reduce the raw error rate by several orders of magnitude. The genetic code is redundant (64 codons, 20 amino acids), and the redundancy is arranged so that common point mutations are more likely to be synonymous — error-tolerant by design.

Neural coding — how many bits does a spike train carry? An active research area using exactly these tools.

Language and cognition

English is ~1.1 bits/character with full context, against 4.7 for uniform letters. ~75% redundant.

That redundancy is functional, not waste: it’s what lets you read text with typos, understand speech in a noisy room, and autocomplete reliably. Natural language has built-in error correction.

Zipf’s law — word frequency inversely proportional to rank — has an information-theoretic reading: it’s close to the distribution minimising average code length given a communication cost, which is why it recurs across languages.

The uniform information density hypothesis proposes that speakers unconsciously spread information evenly across an utterance — contracting predictable words (“I’m” for “I am”) and spelling out surprising ones. There’s decent empirical support, and it’s a nice example of an information-theoretic prediction about behaviour.

Decision-making and inference

Bayesian updating is information acquisition. The expected reduction in entropy from an observation is exactly the mutual information between the observation and the parameter.

Optimal experiment design — choose the experiment maximising expected information gain. → Experiment Design

Twenty questions is optimal binary search, and an optimally-played game extracts 1 bit per question — which is why 20 questions distinguishes ~ possibilities.

Model selection — AIC and MDL (minimum description length) are information-theoretic. MDL’s framing is the cleanest: the best model is the one that most compresses the data, counting the model’s own description length. Occam’s razor as a compression argument, and it’s a genuinely satisfying formalisation.

The thread

One measure — — applied consistently:

DomainWhat it measures
Compressionbits needed to store
Communicationbits transmittable
MLbits wasted by a wrong model
Cryptographybits of secrecy
Securitybits leaked
Physicsenergy cost of erasure
Biologyfidelity of replication
Inferencebits gained per observation

Shannon’s paper was titled A Mathematical Theory of Communication. It turned out to be a mathematical theory of a great deal more, and the reason is that “how surprised should I be” is a question with the same structure everywhere.


↑ Back to Information Theory