Replacing Prometheus Federation: Rebuilding a Multi-Tenant Monitoring Platform
Also in Chinese: 中文版
Why
The global Prometheus at the top of our federation topology was OOMing two to three times a week. Its head block carried 1.2M series and cost 5–10 GB of memory to maintain; every crash-restart cycle left holes in the data, and those holes paged on-call for incidents that did not exist. That was the reliability half of the problem. The latency half was structural: federation stacks two scrape intervals — 15s at the cluster level plus 30s at the global tier — so the global view could trail reality by up to 45 seconds. During a P0, the person paged was looking at the past.
Scale made both problems terminal. At 50 clusters, the /federate endpoint was timing out under scrape load, producing gaps in Grafana. Every cluster maintained its own federation rules, and onboarding a new cluster meant hand-editing the global Prometheus scrape config. And the platform had no usable multi-tenant dimension: metrics aggregated across tenants, so a fault confined to one tenant disappeared into the average.
I owned the observability domain end-to-end within a 3–4 person SRE team — design, evaluation, rollout. For the engine choice I ran the VictoriaMetrics-vs-Thanos POC (write throughput, compression, operational complexity) and the team ratified the result.
How
The replacement is a push-based hub: one vmagent per workload cluster scrapes locally, injects cluster and cluster-group labels at relabel time, and remote-writes into a central VictoriaMetrics cluster (vminsert ×2 behind LB, vmstorage ×3 with replication factor 2, vmselect ×2). vmalert evaluates recording and alert rules against the same store; Alertmanager (HA pair) routes notifications; Grafana and Loki form the read-and-verify plane.
Capacity was designed from measured inputs: 50 clusters × ~12 nodes ≈ 600 nodes at ~2,000 series per node gives ~1.2M active series and ~80,000 samples/s at a 15s scrape interval. Hot storage holds 3 months in ~250 GB on SSD at roughly 4× compression — the same window cost ~930 GB under federation. Cold storage is 5-minute downsampled data on S3: 180 days in ~25 GB.
Alerting is part of the platform, not an afterthought. Recording rules materialize an SLI family first — QPS, error ratio, P95/P99 latency, saturation, and per-tenant SLI — keyed by {tenant, cluster group}; dashboards and alert rules both consume the precomputed series instead of running heavy queries live. Severity is three-tiered: PAGER routes to PagerDuty and is reserved for confirmed user impact, HIGH goes to Slack, MEDIUM becomes a ticket. A PAGER requires a multi-window burn-rate breach — both the 5m and the 30m windows over threshold — which filters transient jitter without losing sustained degradation. Inhibition rules suppress derived noise along explicitly causal chains only (NodeNotReady inhibits PodUnableToStart on the same node); speculative cross-service inhibition was deliberately excluded, because a wrong inhibition hides a real failure.
Tenant identity rides two paths. Applications emit the tenant label on their own metrics, since only the application knows whose request it is serving; vmagent relabeling injects cluster and cluster-group context at scrape time. Before this project, relabel configs had drifted per cluster — inconsistent cluster naming, some teams labeling client instead of tenant — which silently broke alert routing, drill-down, and inhibition’s label-equality matching. I wrote one relabel template and distributed it to all clusters, and drove the application-side convergence on tenant.
Before: federation.
After: VictoriaMetrics platform.
Hard parts
Cutting over without losing an alert. VictoriaMetrics speaks MetricsQL, which has boundary differences from PromQL, so “the rules still fire” could not be assumed. We ran both stacks in parallel for two weeks in a dual-write configuration, evaluated every alert rule on both sides, diffed firing behavior, and fixed divergences before flipping notification routing. The platform also had to fail loudly: vmalert emits a continuous heartbeat, and a deadman’s switch plus an out-of-band probe convert “monitoring went silent” into an explicit page within minutes, rather than a quiet blind spot.
Data lag: 45s to under 5s. The improvement is architectural, not tuned. Federation’s lag is two stacked scrape periods by construction; remote_write pushes samples as they are scraped, so lag drops below 5 seconds. What actually needed engineering was loss, not speed: push means a network blip can drop data, so every vmagent runs a persistent queue on local disk, buffering through remote outages and draining on reconnect.
Multi-tenant cardinality governance. Adding a tenant dimension multiplies series, and series growth after the label rollout was fast. The controls: tenant-level recording rules exist only for core SLIs; infrastructure metrics carry no tenant label at all; retention is tiered (tenant SLA series 90 days, troubleshooting series 30 days, infrastructure 15 days); and periodic cardinality reviews catch accidentally high-cardinality labels. The honest retrospective note: a cardinality budget should have existed from day one, not after the growth curve forced the issue.
Making severity mean something. An audit of 30 days of alert history showed roughly 80% of pages were resource-centric — CPU or memory jitter that on-call would glance at and dismiss. The fix was admission control on the PAGER tier: a page must indicate observable user impact (SLO burn rate, SLA latency breach, or a QPS cliff on a critical flow), everything else is demoted to HIGH or MEDIUM. Static thresholds were replaced by multi-window burn-rate. Two feedback signals tuned the system after launch: action rate per PAGER, and user-reported incidents that produced no page.
Production
The cutover gate was the two-week dual-write window: alert rules had to fire equivalently on both stacks and dashboard queries had to return equivalent results under MetricsQL before routing moved. The federation stack stayed live as the fallback until that gate passed. Inhibition rules were validated in staging by manually triggering upstream failures and checking which downstream alerts were suppressed, with firing history retained for postmortem review. After the switch, new-cluster onboarding collapsed from “write federation rules, edit the global scrape config” to “deploy vmagent, point it at the remote_write address.” Ongoing verification runs on three signals: the deadman heartbeat, PAGER action rate, and missed-incident tracking.
Takeaways
- Federation’s 45s lag and weekly OOMs were properties of the architecture, not of the configuration; no tuning budget would have fixed them.
- Alert quality is an admission-control problem: define what earns a page, demote everything else, and measure action rate per page as the regression test.
- Label consistency is platform infrastructure. Routing, drill-down, and inhibition all depend on label equality, so it must be enforced once at the relabel layer, not negotiated per team.