Two Hard Things: caching bugs · Part 4
Your outage has a schedule
Deploy at 2 PM, a million keys fill with a one-hour TTL, and at 3 PM they all expire together. Then again at 4. Fix: one line of jitter, one hour plus or minus ten percent per key. A million expiries spread over twelve minutes and the spike becomes a ripple.
Comment below: Do you jitter your TTLs?
Two Hard Things: a series on the caching bugs you already ship, and how to fix them. "There are only two hard things in computer science: cache invalidation and naming things." (Phil Karlton) All code is real TypeScript on Node, typechecked before it went on screen, and every claim was reproduced against real Redis and Postgres.
Transcript
Your whole cache expires in the same minute. You deploy at 2:00 p.m. The cache starts empty. So in the next minute a million keys get filled all with a TTL of 1 hour. At 3:00 p.m. they expire together. Hit rate drops. The database takes the load and slows down and the cache refills at the same moment again. Same spike at 4:00. Your spike has a schedule. Fix one line. Add jitter. 1 hour plus or minus 10% random per key. Now a million expiries spread across 12 minutes instead of one. The spike becomes a ripple. Best practice. Never give a million keys the same deadline. Your hit rate is fine now unless someone asks for keys that don't exist. Part five. Do you jitter your TTLs?
