Two Hard Things: caching bugs · Part 2
Deleting the cache key is not enough
A slow reader fetches the old row, a writer updates Postgres and deletes the key, then the reader writes the old row back into the cache. Stale, and nobody comes to delete it. The fix from Facebook's memcache paper: leases. A delete voids every outstanding ticket, so the late write is rejected.
Comment below: Have you ever chased a stale cache in prod?
Two Hard Things: a series on the caching bugs you already ship, and how to fix them. "There are only two hard things in computer science: cache invalidation and naming things." (Phil Karlton) All code is real TypeScript on Node, typechecked before it went on screen, and every claim was reproduced against real Redis and Postgres.
Transcript
Deleting the cache key on write still serves stale data. Watch the timing. A reader misses and reads the old row from Postgres. Then it pauses. A GC pause, a slow network, anything. Meanwhile, a writer updates Postgres and deletes the key. There's nothing to delete. Then the reader wakes up and writes the old row into the cache. Stale. And nobody is coming to delete it. Facebook hit this at scale and their Memcache paper has the fix. Leases. On a miss, the cache hands the reader a ticket. A delete tears up every ticket for that key. When the slow reader comes back with its old row, its ticket is void. So the write is rejected. Stale data never lands. Best practice. Filling the cache is a write too. Make late writes lose. Leases fix one more thing. A thousand requests missing the same key at once. Part three. Have you ever chased a stale cache in prod?
