[Intermediate] — Shannon’s definition, why surprise is the right measure, and where the bit comes from.
Source: [reference] — see the domain note.
The question
1948. Claude Shannon, at Bell Labs, asks: how much information is in a message?
The answer had to be independent of meaning. A telephone company doesn’t care whether you’re proposing marriage or reading a shopping list — it needs to know how many bits it must carry.
Shannon’s move was to define information as surprise.
A message that tells you something you already knew carries no information. A message that tells you something wildly unexpected carries a lot. “The sun rose today” is worth nothing; “it snowed in Lagos” is worth a great deal.
So information is a property of the probability distribution, not of the content.
Self-information
How surprised should you be by an event of probability ?
Three requirements force this form, and it’s worth seeing that it’s not arbitrary:
1. Certain events carry no information. . ✓
2. Rarer means more informative. decreases as increases. ✓
3. Independent events add. If and are independent, learning both should give you . Since , you need a function turning multiplication into addition — a logarithm. ✓
That third requirement is the one that pins it down. Additivity over independent events is what makes information behave like a physical quantity you can measure and budget.
The base sets the unit:
| Base | Unit |
|---|---|
| 2 | bits (or shannons) |
| nats — natural in maths and ML | |
| 10 | hartleys / dits |
A fair coin flip is bit. That’s the definition of the bit, and it’s why the unit exists.
A fair die roll is bits — note it’s not an integer, which is the first sign that “bits” here means something more precise than “storage cells”.
Entropy
The average information content of a distribution:
Entropy is the expected surprise. Equivalently — and this is the operational meaning that matters:
is the average number of bits needed to encode a value drawn from , using the best possible code.
Not “could be” — the minimum, provably. That’s Shannon’s source coding theorem. → Source Coding
The properties:
when one outcome is certain — no uncertainty, no information.
when all outcomes are equally likely — maximum uncertainty. A uniform distribution is the most surprising one, which matches intuition.
Worked examples worth carrying:
| Distribution | Entropy |
|---|---|
| Fair coin | 1 bit |
| Biased coin, | 0.469 bits |
| Biased coin, | 0.081 bits |
| Fair die | 2.585 bits |
| English letters (frequency only) | ~4.1 bits |
| English text (with context) | ~1.1 bits/char |
That last pair is the whole subject in one comparison. Treating letters as independent gives 4.1 bits each. Accounting for the fact that
qis followed byu, thatthis common, and that words are real, drops it to ~1.1.Shannon measured this experimentally in 1951 by having people guess the next letter of a text. The redundancy is why English compresses ~4:1, why you can read text with vowels removed, and why autocomplete works.
The binary entropy function
For a two-outcome distribution:
H(p)
1 ┤ ╭───╮
│ ╱ ╲
0.5 ┤ ╱ ╲
│ ╱ ╲
0 ┼─────────┬─────────────
0 0.5 1 p
Maximum at , zero at both extremes, and symmetric. A coin biased 90/10 and one biased 10/90 are equally uncertain — which they obviously should be, and the formula gets it right.
The curve is flat near the top. still gives — a 40/60 split is nearly as uncertain as 50/50. This is why moderate class imbalance in ML barely reduces the information content, while extreme imbalance (, ) removes almost all of it.
Why “entropy”
The name is borrowed from thermodynamics, and the connection is real rather than metaphorical.
Boltzmann: , with the number of microstates consistent with the observed macrostate.
Shannon: .
For a uniform distribution over states, . Same formula, different constant.
The story goes that von Neumann suggested the name to Shannon, on the grounds that the function already had one in statistical mechanics — and that “nobody knows what entropy really is, so in a debate you will always have the advantage.” Probably embellished, and the underlying point stands: both measure uncertainty about a system’s exact state given partial knowledge.
Landauer’s principle makes it physical: erasing one bit of information dissipates at least joules of heat. Information has a thermodynamic cost, and it’s been measured experimentally. It’s also the resolution of Maxwell’s demon — the demon must erase its measurements, and that erasure pays back the entropy it seemed to remove.
What entropy is not
Three clarifications that prevent real confusion:
Not “amount of data”. A 1 GB file of zeros has almost zero entropy. Entropy is about the distribution, not the length.
Not “meaning” or “value”. A random string has maximum entropy and no meaning. Shannon deliberately excluded semantics — and said so explicitly in the 1948 paper, noting that the semantic aspects are “irrelevant to the engineering problem”.
Not a property of a single message. Entropy is a property of the source — the distribution messages are drawn from. Asking for the entropy of the string "hello" is not well-posed without saying what distribution produced it. (Kolmogorov complexity is the per-object alternative, and it’s uncomputable.)
Where this goes
The rest of the track builds on this one definition:
Multiple variables — joint, conditional and mutual information → note 02
Compression — entropy is the hard floor → note 03
Machine learning — cross-entropy loss and KL divergence are this, directly → note 04
Communication — how much can a noisy channel carry → note 05
Reliability — error-correcting codes → note 06
Prerequisites: probability — distributions, expectation, independence, conditional probability. Logarithms. Nothing else.
Related
- Joint, Conditional and Mutual Information — the toolkit
- Cross-Entropy and KL Divergence — the note the ML material needs
- Probability and Statistics — the prerequisite
- Information theory map
Software engineering
Infrastructure & security
Computer science
Hardware & engineering