Tenant-Aware Alerting on a Multi-Tenant Metrics Platform

Posted on 2026-08-03 Edited on 2026-09-18 In Observability

Designed a 3-tier alerting system with per-tenant SLI recording rules on a VictoriaMetrics platform serving 50 Kubernetes clusters, replacing resource-centric paging (about 80% of pages were not actionable) with admission control tied to observable user impact.

Also in Chinese: 中文版

Why

The global Prometheus at the top of our federation topology was OOMing two to three times a week. Its head block carried 1.2M series and cost 5–10 GB of memory to maintain; every crash-restart cycle left holes in the data, and those holes paged on-call for incidents that did not exist. That was the reliability half of the problem. The latency half was structural: federation stacks two scrape intervals — 15s at the cluster level plus 30s at the global tier — so the global view could trail reality by up to 45 seconds. During a P0, the person paged was looking at the past.

Scale made both problems terminal. At 50 clusters, the /federate endpoint was timing out under scrape load, producing gaps in Grafana. Every cluster maintained its own federation rules, and onboarding a new cluster meant hand-editing the global Prometheus scrape config. And the platform had no usable multi-tenant dimension: metrics aggregated across tenants, so a fault confined to one tenant disappeared into the average.

The fix for this was moving the metrics platform from federation to VictoriaMetrics. That migration was carried out by the platform team before I took on this scope: the engine choice (VictoriaMetrics vs. Thanos), the data lifecycle policy, and the migration execution were not mine. I can discuss why that choice makes sense from an SLO and architecture standpoint, but I cannot speak to the rough edges hit during the migration itself.

What I owned, end to end, was the alerting layer sitting on top of that new metrics store. An audit of 30 days of alert history showed that roughly 80% of pages were resource-centric: CPU or memory jitter that on-call would glance at and dismiss. And the metrics had no usable tenant dimension, so a fault confined to one tenant disappeared into a cluster-wide average.

How

The platform underneath is a push-based hub: one vmagent per workload cluster scrapes locally, injects cluster and cluster-group labels at relabel time, and remote-writes into a central VictoriaMetrics cluster (vminsert ×2 behind LB, vmstorage ×3 with replication factor 2, vmselect ×2). vmalert evaluates recording and alert rules against the same store; Alertmanager (HA pair) routes notifications; Grafana and Loki form the read-and-verify plane. That topology and its capacity planning (50 clusters × ~12 nodes ≈ 600 nodes at ~2,000 series per node, giving ~1.2M active series and ~80,000 samples/s at a 15s scrape interval; hot storage holding 3 months in ~250 GB on SSD; cold storage as 5-minute downsampled data on S3, 180 days in ~25 GB) were already in place when I started working on top of it.

Platform topology (inherited context, not my build).

flowchart TB subgraph FLEET2["50 clusters"] A["vmagent remote_write + persistent queue"] end A --> VI["vminsert ×2"] VI --> VS[("vmstorage ×3 · repl=2")] VS --> VQ["vmselect ×2"] VQ --> VA["vmalert recording + alerting rules"] VQ --> GF["Grafana / Loki"] VA --> AM["Alertmanager 3-tier + inhibition"]

On top of that platform, I designed the alerting layer. Recording rules materialize an SLI family first: QPS, error ratio, P95/P99 latency, saturation, and per-tenant SLI, keyed by {tenant, cluster group}; dashboards and alert rules both consume the precomputed series instead of running heavy queries live. Severity is three-tiered: PAGER routes to PagerDuty and is reserved for confirmed user impact, HIGH goes to Slack, MEDIUM becomes a ticket. A PAGER requires a multi-window burn-rate breach, both the 5m and the 30m windows over threshold, which filters transient jitter without losing sustained degradation. Inhibition rules suppress derived noise along explicitly causal chains only (NodeNotReady inhibits PodUnableToStart on the same node); speculative cross-service inhibition was deliberately excluded, because a wrong inhibition hides a real failure.

What I designed on top of it: the alerting layer.

flowchart TB VA["vmalert SLI recording rules by {tenant, cluster_group}"] --> SEV{"Severity admission control"} SEV -->|"observable user impact, 5m+30m burn-rate"| PAGER["PAGER → paging channel"] SEV -->|"degraded, not user-facing"| HIGH["HIGH → Slack"] SEV -->|"informational"| MED["MEDIUM → ticket"] PAGER --> INH["Inhibition: causal chains only"]

Tenant identity rides two paths. Applications emit the tenant label on their own metrics, since only the application knows whose request it is serving; vmagent relabeling injects cluster and cluster-group context at scrape time. Before this project, relabel configs had drifted per cluster — inconsistent cluster naming, some teams labeling client instead of tenant — which silently broke alert routing, drill-down, and inhibition’s label-equality matching. I wrote one relabel template and distributed it to all clusters, and drove the application-side convergence on tenant.

Hard parts

Multi-tenant cardinality governance. Adding a tenant dimension multiplies series, and series growth after the label rollout was fast. The controls: tenant-level recording rules exist only for core SLIs; infrastructure metrics carry no tenant label at all; retention is tiered (tenant SLA series 90 days, troubleshooting series 30 days, infrastructure 15 days); and periodic cardinality reviews catch accidentally high-cardinality labels. The honest retrospective note: a cardinality budget should have existed from day one, not after the growth curve forced the issue.

Making severity mean something. An audit of 30 days of alert history showed roughly 80% of pages were resource-centric — CPU or memory jitter that on-call would glance at and dismiss. The fix was admission control on the PAGER tier: a page must indicate observable user impact (SLO burn rate, SLA latency breach, or a QPS cliff on a critical flow), everything else is demoted to HIGH or MEDIUM. Static thresholds were replaced by multi-window burn-rate. Two feedback signals tuned the system after launch: action rate per PAGER, and user-reported incidents that produced no page.

Production

The severity redesign and SLI recording rules rolled out cluster by cluster on top of the already-migrated metrics platform, so there was no separate cutover risk for the alerting layer itself. New rules were validated against firing history before their routing went live. Inhibition rules were validated in staging by manually triggering upstream failures and checking which downstream alerts were suppressed, with firing history retained for postmortem review. Ongoing verification runs on two signals: PAGER action rate, and missed-incident tracking (user-reported incidents that produced no page). The tenant-label convergence is enforced going forward by the shared relabel template rather than by per-cluster review.

Takeaways

Federation’s 45s lag and weekly OOMs were properties of the architecture, not of the configuration. That is a judgment I can defend even though I did not execute the migration off it.
Alert quality is an admission-control problem: define what earns a page, demote everything else, and measure action rate per page as the regression test.
Label consistency is platform infrastructure. Routing, drill-down, and inhibition all depend on label equality, so it must be enforced once at the relabel layer, not negotiated per team.