Tech & AI

Five design decisions behind a heavy-query compute pool that rests at zero: physical isolation instead of a workload class, discrete capacity instead of per-query autoscaling, sizing by bottleneck instead of by size, a controller reduced to patching one field, and a readiness path built on the assumption that every health signal eventually lies. Spot pricing is a per-pool consequence, not the point.

A cross-tablet point-query index that prunes 566 tablets to about one before any tablet is opened, and a wide-read prefetch that removes serialization rather than IO — chosen because the cost of each was mostly avoidable fan-out, and shipped fail-open because neither may ever change a result. Includes the tradeoff that actually cost us, and the silent incident it caused.

An Upgrade Safety System (check → plan → apply + evidence) turned an 18–21 hour, two-person manual ritual into a 6–8 hour single-operator procedure, executed across two production fleets with zero customer-impacting downtime.

Split, shed, then scale — and why the intuitive order takes the database down.

Two words that halve the search space, and two root causes found by walking eight hops.

A new SQL statement inside the Apache Doris frontend that classifies every query heavy or light in 6–8 ms before execution — built on a fork, mechanism prepared for upstream, policy deliberately kept internal.

A restore drill that corrected its own premise and made RPO explicit for the first time.

Designed a 3-tier alerting system with per-tenant SLI recording rules on a VictoriaMetrics platform serving 50 Kubernetes clusters, replacing resource-centric paging (about 80% of pages were not actionable) with admission control tied to observable user impact.

ClickHouse → Doris compute-storage separation: memory engineering, engine-level routing, two compaction incidents, and numbers that survived re-measurement.

What changed, and what still needs an engineer