Audited a pager emitting ~960 OpsGenie alerts a day, ~85% of them default-P3, from 367 paging rules — traced four structural root causes, and rebuilt admission, dedup, and ownership so decay has to fight back against the system.
How the nine scores were derived from evidence density, what each one means, and a behavioral record that independently cross-checks the shape.
Five design decisions behind a heavy-query compute pool that rests at zero: physical isolation instead of a workload class, discrete capacity instead of per-query autoscaling, sizing by bottleneck instead of by size, a controller reduced to patching one field, and a readiness path built on the assumption that every health signal eventually lies. Spot pricing is a per-pool consequence, not the point.
A self-authored, continuously running systemd-level daemon that turned Intel's Best Known Configuration from prose-and-shell documentation into a git-declared, continuously audited desired state across ~30 bare-metal servers spanning multiple clusters.
A new SQL statement inside the Apache Doris frontend that classifies every query heavy or light in 6–8 ms before execution — built on a fork, mechanism prepared for upstream, policy deliberately kept internal.
An Upgrade Safety System (check → plan → apply + evidence) turned an 18–21 hour, two-person manual ritual into a 6–8 hour single-operator procedure, executed across two production fleets with zero customer-impacting downtime.
A cross-tablet point-query index that prunes 566 tablets to about one before any tablet is opened, and a wide-read prefetch that removes serialization rather than IO — chosen because the cost of each was mostly avoidable fan-out, and shipped fail-open because neither may ever change a result. Includes the tradeoff that actually cost us, and the silent incident it caused.
ClickHouse → Doris compute-storage separation: memory engineering, engine-level routing, two compaction incidents, and numbers that survived re-measurement.
Two words that halve the search space, and two root causes found by walking eight hops.
Moved 50 Kubernetes clusters off a weekly-OOMing federation topology onto a VictoriaMetrics platform — 1.2M active series, data lag 45s → <5s, zero alert gaps through the cutover.