Two Hard Things: caching bugs · Part 3
One cache key expired. Your database went down.
A hot key expires and thousands of requests run the same two-second query at once: a cache stampede. Fix: one miss, one query. Take a lock and let one request recompute, or serve the old value while it refreshes in the background (stale-while-revalidate).
Comment below: What's the hottest key in your cache?
Two Hard Things: a series on the caching bugs you already ship, and how to fix them. "There are only two hard things in computer science: cache invalidation and naming things." (Phil Karlton) All code is real TypeScript on Node, typechecked before it went on screen, and every claim was reproduced against real Redis and Postgres.
Transcript
One cache key expired and your database went down. Your home page query takes 2 seconds, so you cache it for a minute. Thousands of requests per second hit that key. Then the minute is up. Every request in the next 2 seconds misses and every one of them runs the same 2 second query. Thousands of copies of one query all at once. That's a cache stampede. Fix. One miss, one query. The first request takes a lock and recomputes. Everyone else waits for that one result. Or better, they get the old value while it refreshes in the background. That's stale-while-revalidate. Thousands of misses become one query and nobody sees a spinner. Best practice. A hot key should never expire in front of your users. One hot key is bad. Now picture every key expiring in the same second. Part four. What's the hottest key in your cache?
