Long-form notes on backend systems: the problem, the numbers, the design, and the trade-offs I would defend in a review. Grounded in the systems I have built.
9 entries
A video goes viral and every viewer runs the same harmless line: views = views + 1. The CPU stays idle and the database still dies. Why one hot row serializes everything, the arithmetic that caps it near 1,000 writes a second, three ways out, and the one place you must never use them.
Read the log →Two patients tap the same 9:30 slot in the same millisecond. Only one of them may win, every time, even at 7 a.m. when a whole city opens the app. Requirements, numbers, schema, four ways to guard the invariant, and what breaks first when traffic spikes.
A harmless-looking DELETE froze a whole table because a foreign key had no index. Why a lock is only as narrow as the index behind it, what two-phase locking really means (it is not locking twice), how deadlocks are born and detected, and the retry loop every ORM user owes the database.
A 10-million account, a network cut in half, and two leaders who each approve a 10-million withdrawal. Why more followers do not scale writes, what the three steps of failover are, the three ways automatic failover goes wrong, and why a quorum is the way out.
I mixed these three up for years, and the mix-up was always the same: I treated three answers as if they were one. How a database writes to disk, how it searches, and where the row actually lives are separate questions. Plus the part that decides whether your query is fast: the optimizer, its statistics, and why a full scan can be the right plan.
The dashboard says 150 ms and the green light is on, yet users keep writing “the app is laggy”. Why averages hide the pain, what p50, p99 and p999 actually say, the 63% arithmetic that turns a healthy service into a slow page, how to compute percentiles without fooling yourself, and what to tell your boss.
