Setup
A product catalog API serves about 4,000 requests per second at peak. Reads go through a cache cluster with a 97% hit rate. Cache misses go to a primary database with two read replicas.
Retries happen at two layers:
- Clients (the mobile app and the web backend) retry a failed request up to 3 times, 100 ms apart.
- The API’s database client retries a timed-out query up to 2 times. The query timeout is 2 seconds.
At 14:02 a cache node failed and the cluster failed over to a replica. The failover finished in about 40 seconds, but the new primary started with an empty cache.
At 14:03 you’re paged: API error rate 35%, p99 latency 9 seconds.
Stages
Stage 1: The first five minutes
- The cache reports healthy since 14:03. Hit rate is 12% and climbing very slowly.
- Database CPU is at 98% on the primary and both replicas.
- Database connections are at the pool maximum on every API instance.
- The API is receiving 11,000 requests per second, almost 3× normal peak.
- CDN logs show user traffic is flat: no spike in real users.
Stage 2: Where the extra traffic comes from
- About 65% of incoming requests carry a retry header (
attemptgreater than 1). - Database client logs show a steady stream of 2-second timeouts, each followed by a retry.
- The cache is filled only after a database query succeeds. Most queries are timing out, so most keys never get cached.
Stage 3: The first mitigation
At 14:15 the team scaled the API from 20 to 40 instances to absorb the load.
- Error rate went from 35% to 52%.
- Database connections doubled. The database started refusing new connections.
- Cache hit rate dropped back to 9%.
Outcome
The team recovered by taking load away, not adding capacity:
- They turned off client retries through the mobile app’s remote config and the web backend’s feature flag.
- They rate-limited the API at the load balancer to 4,500 requests per second, returning fast 503s above that.
- They scaled back to 20 instances to cut database connections.
With the database no longer saturated, queries succeeded, the cache filled, and the hit rate passed 90% at 14:41. Error rates were normal by 14:44, 42 minutes after a 40-second cache failover.
Why it stayed down: a cold cache multiplied database load roughly 30×. Retries at two layers multiplied it again: each user request could become up to 12 database queries (4 client attempts × 3 database attempts). Because the cache only filled on success and almost nothing succeeded, the system was stuck in a self-sustaining bad state after the original trigger was gone. This is a metastable failure.
Follow-ups: request coalescing on cache misses (one database query per key, not one per request), retries at one layer only with exponential backoff, jitter, and a retry budget, load shedding in front of the database, and a cache warm-up step in the failover runbook.
Senior commentary
“I’d ignore the cache at first. It says it’s healthy, and that’s probably true. The number that jumps out is 11,000 requests per second while user traffic is flat. Traffic that doesn’t come from users comes from us: retries, health checks, batch jobs. So my first question is ‘who is sending this?’, and the retry header answers it in stage 2.
“Then I’d ask why it isn’t recovering on its own. Something has to be holding it in this state. The cache only fills on success, and almost nothing succeeds, so it can’t warm up. That’s the loop to break. You don’t fix a metastable failure by fixing the trigger; the trigger is already gone. You remove whatever load is sustaining it.
“Scaling out in stage 3 is the instinct most people have, and it’s a trap when the bottleneck is downstream and shared. More API instances means more connections and more concurrent retries against the same saturated database. I’d want to know where the bottleneck is before adding capacity anywhere.”
Discussion questions
- Which signal in stage 1 best separated “the cache is broken” from “the system is stuck”?
- Why did doubling the API instances make things worse? When would scaling out have helped?
- If you could keep retries at only one layer, which layer would you keep and why?
- What would a cache warm-up step in the runbook actually do, and what could go wrong with it?