[Intermediate] · Twelve exercises. Every idea in this course is a short script over real data.
This course’s own note says it’s “unusually cheap to make concrete” — and it is. Pure Python; no libraries needed except where marked.
Solutions with measured results in note 09.
Part A — Entropy (notes 01–02)
1. Measure the entropy of a real file.
Count byte frequencies in a file you own, compute in bits per byte, and multiply by the file size for a predicted floor. Then compress it with gzip -9 and xz and compare.
Done when: you have three numbers — and if the compressor beat your “floor”, you have the most important result in this course. Explain it → note 01.
2. Entropy of English.
Compute the order-0 entropy of letters in a large English text, and compare against for a uniform alphabet.
Done when: you can say how many bits of redundancy per letter the letter frequencies alone reveal, and why the true figure for English is far lower still.
3. Reproduce Shannon’s 1951 experiment.
Have someone guess the next letter of a sentence they haven’t seen, one letter at a time, recording how many guesses each took. 100 letters is enough.
Done when: you have a guess-count distribution and an entropy estimate. A genuinely fun hour, and it produces a number nothing else gets you.
4. Mutual information beats correlation.
Generate and . Compute Pearson correlation, then estimate by binning.
Done when: correlation is ~0 and mutual information is clearly positive. Two variables where one determines the other, and correlation sees nothing → note 02.
5. Conditioning creates dependence.
Let and be independent fair coins and . Verify but bit.
Done when: you can state why “control for more variables” is not automatically more rigorous. This is a collider, and it’s the mechanism behind a large class of bad statistical practice.
Part B — Compression (note 03)
6. Implement Huffman.
Build the tree, encode, decode, and verify round-trip. Compare bits used against the entropy floor from exercise 1.
Done when: your output is within 1 bit/symbol of → note 03.
7. Make Huffman lose badly.
Encode a binary source with , then , then . Compare bits/symbol against .
Done when: you can state how much worse Huffman is at each, and why it cannot do better without a change of approach.
8. Fix it with blocking.
Group the source into blocks of 2, 3, 4 symbols and Huffman-code the blocks.
Done when: bits/symbol approaches as block size grows, and you can explain what arithmetic coding does that makes blocking unnecessary.
9. Prove no compressor compresses everything.
Argue by counting that no lossless scheme can shorten every input of length .
Done when: the argument is airtight in three lines — and you can use it to dismiss any “compresses any file by 50%” claim in one sentence.
Part C — The ML connection and channels (notes 04–06)
10. Cross-entropy by hand.
Take a 4-class problem, five examples, hand-written predicted probabilities. Compute the cross-entropy loss manually, then confirm it matches your framework’s cross_entropy. Then compute perplexity.
Done when: your hand figure matches to several decimals, and you can state the relationship between loss and perplexity → note 04.
11. Forward vs reverse KL.
Fit a single Gaussian to a bimodal target by minimising and then . Plot both.
Done when: one covers both modes and one picks a single mode, and you can say which failure each direction produces.
12. Hamming code by hand.
Implement Hamming(7,4). Encode a nibble, flip one bit, and correct it. Then flip two bits.
Done when: single errors are corrected and you can show a double error is silently mis-corrected — which is why real systems use SECDED → note 06 · ECC memory.
Related
Source: [reference] — built from this course’s own gap-closing list. Results measured Aug 2026.
Software engineering
Infrastructure & security
Computer science
Hardware & engineering