Tenant-Aware Alerting on a Multi-Tenant Metrics Platform

发表于 更新于 分类于
Designed a 3-tier alerting system with per-tenant SLI recording rules on VictoriaMetrics, replacing resource-centric paging (~80% of pages were not actionable) with admission control tied to observable user impact — on a platform serving 50 Kubernetes clusters.

Also in Chinese: 中文版

Why

Ownership boundary, stated up front: the metrics platform itself — VictoriaMetrics replacing an earlier Prometheus Federation topology across 50 Kubernetes clusters — was migrated before I took on this scope. That migration’s architecture decision, engine selection (VictoriaMetrics vs. Thanos), data lifecycle policy, and execution belong to the platform team, not to me. I can discuss why Federation’s failure mode (weekly OOM, up to 45s data lag) was architectural rather than a tuning problem, because that’s a judgment call I can defend, but I don’t own the migration’s implementation details and can’t speak to the rough edges it hit along the way.

What I did own, end to end, was what sits on top of that metrics store: the alerting layer. An audit of 30 days of alert history showed that roughly 80% of pages were resource-centric — CPU or memory jitter that on-call would glance at and dismiss — and the metrics had no usable tenant dimension, so a fault confined to one tenant disappeared into a cluster-wide average.

How

The platform underneath is a push-based hub: one vmagent per workload cluster scrapes locally and remote-writes into a central VictoriaMetrics cluster (vminsert ×2 behind LB, vmstorage ×3 with replication factor 2, vmselect ×2); vmalert evaluates rules against the same store; Alertmanager (HA pair) routes notifications; Grafana and Loki form the read-and-verify plane. That topology and its capacity planning (~1.2M active series, ~80,000 samples/s at a 15s scrape interval; ~250 GB hot storage for 3 months; 180-day downsampled cold storage on S3) were already in place when I started working on top of it.

On top of that platform, I designed the alerting layer. Recording rules materialize an SLI family first — QPS, error ratio, P95/P99 latency, saturation, and per-tenant SLI — keyed by {tenant, cluster group}; dashboards and alert rules consume the precomputed series instead of running heavy queries live. Severity is three-tiered: PAGER is reserved for confirmed user impact, HIGH goes to Slack, MEDIUM becomes a ticket. A PAGER requires a multi-window burn-rate breach — both the 5m and the 30m windows over threshold — which filters transient jitter without losing sustained degradation. Inhibition rules suppress derived noise along explicitly causal chains only (NodeNotReady inhibits PodUnableToStart on the same node); speculative cross-service inhibition was deliberately excluded, because a wrong inhibition hides a real failure.

Tenant identity needed its own fix. Relabel configs had drifted per cluster before this project — inconsistent cluster naming, some teams labeling client instead of tenant — which silently broke alert routing, drill-down, and inhibition’s label-equality matching. I wrote one relabel template, distributed it to all clusters, and drove the application-side convergence on tenant (applications emit their own tenant label since only the application knows whose request it is serving; vmagent relabeling injects cluster and cluster-group context at scrape time).

Platform topology (inherited context, not my build).

flowchart TB subgraph FLEET2["50 clusters"] A["vmagent<br/>remote_write + persistent queue"] end A --> VI["vminsert ×2"] VI --> VS[("vmstorage ×3 · repl=2")] VS --> VQ["vmselect ×2"] VQ --> VA["vmalert<br/>recording + alerting rules"] VQ --> GF["Grafana / Loki"] VA --> AM["Alertmanager<br/>3-tier + inhibition"]

What I built on top of it: the alerting layer.

flowchart TB VA["vmalert<br/>SLI recording rules by {tenant, cluster_group}"] --> SEV{"Severity admission control"} SEV -->|"observable user impact,<br/>5m+30m burn-rate"| PAGER["PAGER → paging channel"] SEV -->|"degraded, not user-facing"| HIGH["HIGH → Slack"] SEV -->|"informational"| MED["MEDIUM → ticket"] PAGER --> INH["Inhibition: causal chains only"]

Hard parts

Getting tenant identity right. Before the relabel template, routing, drill-down, and inhibition all silently broke whenever a team’s label convention drifted from the norm — client instead of tenant, inconsistent cluster naming. Fixing this meant treating label consistency as platform infrastructure rather than a per-team convention: one relabel template, enforced at the layer where metrics enter the system, not negotiated case by case.

Multi-tenant cardinality governance. Adding a tenant dimension to SLIs multiplies series, and growth after the label rollout was fast. The controls: tenant-level recording rules exist only for core SLIs; infrastructure metrics carry no tenant label at all; retention is tiered (tenant SLA series 90 days, troubleshooting series 30 days, infrastructure 15 days); periodic cardinality reviews catch accidentally high-cardinality labels. Honest retrospective: a cardinality budget should have existed from day one, not after the growth curve forced the issue.

Making severity mean something. The 30-day alert audit showing ~80% resource-centric, non-actionable pages was the forcing function. The fix was admission control on the PAGER tier: a page must indicate observable user impact (SLO burn rate, SLA latency breach, or a QPS cliff on a critical flow), everything else is demoted to HIGH or MEDIUM. Static thresholds were replaced by multi-window burn-rate. Two feedback signals tuned the system after launch: action rate per PAGER, and user-reported incidents that produced no page.

Production

The severity redesign and SLI recording rules rolled out cluster by cluster behind the existing (already-migrated) metrics platform, so there was no separate cutover risk for the alerting layer itself — new rules were validated against firing history before their routing went live. Ongoing verification runs on two signals I track: PAGER action rate, and missed-incident tracking (user-reported incidents that produced no page). The tenant-label convergence is enforced going forward by the shared relabel template rather than by per-cluster review.

Takeaways

  • Federation’s 45s lag and weekly OOMs were properties of the architecture, not of the configuration — that’s a judgment I can defend even though I didn’t execute the migration off it.
  • Alert quality is an admission-control problem: define what earns a page, demote everything else, and measure action rate per page as the regression test.
  • Label consistency is platform infrastructure. Routing, drill-down, and inhibition all depend on label equality, so it must be enforced once at the relabel layer, not negotiated per team.