Order of operations under fire
Also in Chinese: 中文版
Symptom
One tenant’s query rate spiked far above baseline. Serving latency rose, error rates climbed on the dependency chain, and asynchronous recomputation began competing with the message queue and the database for the same headroom.
The sequence, and why it cannot be reordered
-
Stop the bleeding first. Split traffic 50/50 to a second cluster. Reversible in seconds, buys headroom everywhere downstream, changes no behavior.
-
Shed non-critical load second. Dynamically degrade the non-critical async recomputation — rate-limit it to zero through runtime configuration, no restarts. The system keeps serving its primary contract while background work yields.
-
Scale third, with caps. Serving and async pools scale out under explicit ceilings; the message queue’s partitions double; the database scales vertically 2×.
Scaling first is the intuitive move and the wrong one: adding consumers under uncontrolled inflow feeds the surge directly into the slowest stateful component. The database gets pushed over the cliff by the very capacity that was meant to save it, and capacity added under pressure arrives minutes late to a seconds-scale problem. Shedding before scaling is what makes the scaling safe.
Validation discipline
Each step had to prove itself before the next: error rate, P95/P99, consumer-lag slope reversal (lag must be falling, flat is not recovery), and database headroom — each with a stability window of at least 10–15 minutes. Rollback criteria were staged in advance, so de-escalation was a checklist, not a judgment call at 2 a.m.
Under overload, the order of operations is the decision. Everything on the list was correct; done in the wrong order, the same list takes the system down.