Alert Governance: Turning the Pager from Noise Back into Decision Signal
Also in Chinese: 中文版
Why
By the time this became a project, the pager had stopped being evidence of anything. The baseline: 367 alert rules carried the label that routes to OpsGenie, spread across 30 rule files, and OpsGenie was absorbing roughly 960 alerts a day — about 85% of them P3, because P3 was simply the default when no priority was set. The infra-team channel alone ran at ~12 alerts an hour, around 290 a day.
The texture was worse than the volume. One OOM-kill rule with for: 0m produced 80 alerts in three days from 18 crash-looping pods; the top three rules together accounted for 145 pages in that window. A single ALB QPS-zero alert fanned out to six Slack channels at once, then fired again two minutes later. One channel took 8 OpsGenie alerts in six minutes. The batch pipeline’s timeout alert paged the same stuck job hourly — three pages, three hours, one problem — and rated a job 28% over its average the same P1 as one 300% over; on the sampled day it produced 10+ P1 pages, all @-mentioning the subteam, none carrying a diagnostic link. Twelve infra alert channels existed; two had gone silent and nobody had noticed.
The cost was plain: alert fatigue from low-signal pages slows triage exactly when response quality matters most. The governing principle came first and stayed one sentence: an alert that cannot direct action gets turned off or demoted.
How
The method was a decision tree, applied per alert, backed by evidence — every judgment traced to a captured alert sample or a rule definition, not to opinion.
The audit surfaced four structural root causes: the OpsGenie label was a free one-line switch with no admission threshold; overlapping Alertmanager routes with continue: true fanned one alert into multiple receivers; no receiver set a dedup alias, so the same problem re-created itself as a new alert on every re-fire; and a bypass script posted straight to OpsGenie with hardcoded P3, invisible to all routing and inhibition.
Execution was phased so that measurement preceded change. Phase 0 built the baseline dashboard — the numbers above. Phase 1 was config stop-the-bleed, packaged as two reviewable PRs with pre-flight lint and a rollback path: debounce windows on the noisiest rules (for: 0m → 10m), a hard admission gate so only P1/P2 reaches OpsGenie, severity tiering for the batch alerts, and a dedup alias with update_alerts on every receiver so repeats increment a counter instead of paging. Phase 2 split the shared OpsGenie API key into per-team integrations, added causal inhibition chains, and set auto-close policies. Phase 3 made the fixes structural: the bypass sender rerouted through Alertmanager, and a team ownership label required on every rule. Each phase carried acceptance gates against the baseline: OpsGenie volume 960/day → under 400, then under 200; the top-3 noise sources 145 per 3 days → under 40, then under 15; P3 share 85% → under 50%, then under 30%; on-call watching three channels instead of twelve. The same pass delegated deploy-approval toil — 11 approval requests @-ing infra on-call in two days, all for preprod — to team-lead approval, since an existing auto-approve path already proved infra review added nothing there.
Why alerting systems decay
The analysis chapter of this project, and the part I consider its actual core, is a mechanism, not a list of bad rules. An alerting system increases in entropy unless external work is applied — and the audit shows exactly why.
The incentives are asymmetric. Adding an alert is cheap and visible: after an incident, one label line routes a new rule to the pager, there is no admission threshold, and the author looks diligent. That is how 367 paging rules accumulate. Deleting an alert is risky and invisible: delete wrong and you own the missed outage; delete right and nothing observable happens. So the flow runs one way.
Meanwhile every rule’s semantics silently expire as the system evolves under it, and expiry emits no signal. The audit is a catalog of this: a node-memory alert at 90%, a threshold that predates 90% being a normal operating level for a Kubernetes node; a 300Mbps traffic page, though high traffic alone is not a failure and gave on-call nothing to do; a QPS-zero alert built on the assumption that traffic is always present, firing on endpoints that legitimately have none. Priority decayed the same way: with P3 as the unexamined default, 85% of alerts shared one level and the field carried no information — trivial and severe deviations paged identically. Even the topology decayed: two of twelve channels had simply died, and the system grew a bypass path around its own governance. None of this was anyone’s mistake. It is the default trajectory of any system where adding is free, deleting is punished, and failure is silent.
Keeping it clean: anti-entropy mechanisms
A one-time cleanup only resets the entropy to a low point; preventing recurrence means structures that apply work continuously. Three came out of this project: admission control — the P1/P2 gate in front of OpsGenie, a required ownership label on every rule, and a planned CI lint rejecting any paging rule without a runbook_url; standing measurement — the baseline dashboard kept as a permanent alert SLO dashboard, plus an automated weekly toil report, making pager quality itself a monitored metric; and a review loop — a quarterly, owner-assigned alert review with explicit severity thresholds, proposed upward as a team-level process. The intent: decay now has to fight the system instead of riding its defaults.
Takeaways
- An alerting system obeys a second law: without continuous external work it accumulates noise, because adding alerts is cheap and visible while removing them is risky and invisible. Governance is that work, institutionalized.
- Cleanup is a state change; admission control is a rate change. The gate in front of the pager (priority threshold, owner, runbook) is worth more than any single purge.
- Make the pager itself an SLO object: if alert volume, priority mix, and dedup ratio are not on a dashboard, their regression is invisible — which is how the entropy got in the first time.