The hardest domain to get reps in, because real architecture needs scale you don’t have. The workaround: build the small version and then break it deliberately. A three-node system you can partition teaches more than a diagram of a hundred.
Difficulty: 🟢 starter (hours–days) · 🟡 intermediate (a week or two) · 🔴 ambitious / portfolio-grade. ⭐ = highest signal.
The ladder
-
🟢 Write a design doc for something you already built — context, constraints, options considered, decision, trade-offs accepted. Done when: someone else could rebuild it from the doc, and you’ve named at least one thing you’d do differently. Exercises: fundamentals.
-
🟢 Do five system-design questions on a whiteboard, out loud, timed — 45 minutes each, using the cheat sheet only afterwards to check yourself. Done when: you stop drawing boxes and start naming numbers (QPS, storage, bandwidth).
-
🟡 Cache invalidation, for real — put Redis in front of a real read path in one of your apps. Done when: you’ve handled a stampede, chosen a TTL you can justify, and can describe the staleness window a user might see. Exercises: caching.
-
🟡 ⭐ Break your own system — take a service you built, run it with a dependency behind a proxy (Toxiproxy), and inject latency, packet loss, and hard failures. Done when: you’ve found at least one failure mode you didn’t know you had, and fixed it. Exercises: why this is hard.
-
🟡 Idempotency and exactly-once, honestly — build a payment-ish flow with an outbox, at-least-once delivery, and an idempotent consumer. Done when: you can explain to someone why exactly-once delivery doesn’t exist but exactly-once effect does. Exercises: distributed transactions.
-
🟡 Event sourcing something small — an append-only log plus a projection. Done when: you can rebuild current state from the log alone, and you’ve hit the first schema-evolution problem. Exercises: the log.
-
🔴 ⭐ A Raft key-value store — leader election, log replication, safety, then a replicated KV store on top. Test it against crashes and partitions. Done when: you kill the leader mid-write and the cluster stays consistent. (MIT 6.824 is the gold standard.) The best distributed-systems rep that exists — and the one that turns consensus from reading into knowledge.
-
🔴 Implement a CRDT and prove convergence — a G-counter, then an LWW-set, then a text CRDT. Done when: randomised concurrent operations in any order converge, verified by a property test. Exercises: CRDTs.
-
🔴 Rebuild one case study — pick something from case studies and actually build a toy of it end to end. Done when: it handles a load test and you’ve written up where your design diverges from the real one.
If you only do one
Raft. Every distributed-systems idea in this folder — leaders, terms, quorums, split brain, linearisability — stops being vocabulary the first time your own cluster elects a leader after you killed one.
The full build guide is Your Own Raft Key-Value Store — milestones from RPC through election, replication, the commit rule and snapshots, with the chaos-testing harness as its own step.
Related
- the architecture course · interview bank
- system design cheat sheet
- Project Ideas — the vault-wide index
Software engineering
Infrastructure & security
Computer science
Hardware & engineering