Designing the Elastic Compute Tier: Discrete Capacity, One Write Primitive, Readiness That Cannot Lie
Also in Chinese: 中文版
The architecture, as five decisions
The pool itself is unremarkable; the design is in what it refuses to do. Five decisions carry the whole thing.
1 — Physical isolation, not a workload class. The heavy tail does not get a priority label inside the serving pool; it gets its own compute group. In shared-data mode that is affordable in a way it would not be in shared-nothing: tablets live in S3, so a pool is a metadata construct. Scaling the pool live from 2 to 4 backends moved zero data — ownership rebalanced to 131/128/128/125 as a metadata operation. Isolation is therefore the cheap primitive here, and memory isolation is the property the serving pool actually needs.
2 — Discrete capacity, not per-query autoscaling. The pool scales in fixed increments between a floor of zero and a small maximum, and deliberately does not compute a precise per-query target. Scale-up is minutes-scale (~66 s for the node, ~2 min for backend registration) while heavy queries run 60–500 s: many finish before a new backend is ready, so per-query elasticity is mostly theater. What elasticity buys is a floor of zero, not a fitted curve.
3 — Sizing by bottleneck, not by data volume. The heavy tail splits in two. Scan-bound shapes want scale-out — more backends, more S3 fetch parallelism. Memory-bound shapes — high-cardinality distinct aggregation, windows, join blow-ups — want scale-up plus spill, because their state has to fit inside one backend’s RAM. Adding backends to a memory-bound query buys nothing; that is a capacity decision that has to be made per shape, not per gigabyte.
4 — One write primitive. The scaler is an internal controller I rebuilt into a pure HPA role. Its first design deployed backends itself, via a Helm install plus ALTER SYSTEM ADD BACKEND; I killed that before implementation, because it applied a shared-nothing mental model to shared-data mode, where backends register with the MetaService and any compute group not declared in the operator’s custom resource is erased on the next reconcile. Bespoke deployment logic was not merely redundant — it built state the platform would actively destroy. The rewrite does exactly one thing: read the CR, locate the target compute group’s index, JSON-patch spec.computeGroups[i].replicas.
5 — Readiness that assumes every signal lies. Scale-from-zero fails silently by construction, so the readiness path is layered and adversarial: controller RUNNING is only a pod-count check, the real gate parses SHOW BACKENDS for the target group’s Alive column, and then a canary query touches storage. A SELECT 1 probe is constant-folded by the frontend and never dispatched — it reports an empty compute group as healthy.
The trigger comes from the layer above: a routing verdict at plan time says heavy, and that verdict is what launches capacity. Evaluate, route, then run — elasticity is the third step, not the first.
Why a floor of zero: cost points the opposite way from frequency
Production query-log mining settled the shape of the problem. Point-serving traffic is ~97% of captured executions on the event table; the heavy classes — cross-date aggregations, big GROUP BYs, window queries — are ≤3% of executions but carry a concentrated, outsized share of compute. The per-query cost ratio is ~1,240×: the monthly cumulative-window query averaged 196,354 ms while the highest-frequency point lookup averaged 158 ms. Starker still, 18 executions of that one window query roughly equal the total compute of 21,755 point queries. A pool sized for the light majority optimizes for throughput and concurrency; the heavy tail fits none of it, and cannot share the serving pool’s machines.
The economics then argue against a resident heavy pool. A static 2-node heavy pool costs ~$772/month on-demand. The measured burst pattern — averaging 2 heavy nodes for ~2 hours a day — costs ~$63/month on-demand, ~$24 on spot: a ~92% saving on-demand, ~97% on spot. The price of elasticity is a cold start of ~66 s for the node plus ~2 min for backend registration, plus a cold-S3 first scan — acceptable for a latency-tolerant <3% tail. That arithmetic is the entire justification for the pattern.
How: a controller reduced to one patched field
The scaler is an internal controller I rebuilt into a pure HPA role. Its first design deployed backends itself, via a Helm install plus ALTER SYSTEM ADD BACKEND. I killed it before implementation: that is a shared-nothing mental model applied to shared-data mode, where backends register with the MetaService and any compute group not declared in the operator’s custom resource is erased on the next reconcile. Bespoke deployment logic was not just redundant; it built state the platform would actively destroy. The rewrite does exactly one thing: read the CR, locate the target compute group’s index, and JSON-patch spec.computeGroups[i].replicas — a single write primitive, merged with 34/34 tests passing.
The full loop runs live on a preprod cluster at production scale: a routing verdict says heavy → the controller patches replicas 0→N → the ASG boots a node in ~66 s → the backend registers with the frontend in ~2 min → layered readiness checks pass → the query executes in the heavy pool → an idle reaper returns the pool to zero. Readiness is layered: controller RUNNING is only a pod-count check; the real gate parses SHOW BACKENDS for the target group’s Alive column, then runs a canary scan. Scaling the pool live from 2 to 4 backends moved zero data — tablets live in S3, and ownership rebalanced to 131/128/128/125 as a metadata operation.
Capacity is discrete, not continuous. The pool scales in fixed increments between a floor of zero and a small max, rather than computing a precise per-query target — scale-up is minutes-scale and many queries finish before a new backend is ready; per-query elasticity is mostly theater. Sizing splits the heavy tail by bottleneck: scan-bound shapes want scale-out (more backends, more S3 fetch parallelism), memory-bound shapes — high-cardinality distinct aggregations, windows, join blow-ups — want scale-up plus spill, because their state must fit one backend’s RAM.
The hard parts
Two ways cluster-autoscaler silently refuses to scale from zero. First, the backend requested cpu=8 on an 8-vCPU node whose allocatable was ~7.9 after kube-reserved — the scheduler simulation reports Insufficient cpu forever, with no event pointing at the 0.1-core gap; requesting cpu=7 scaled immediately; “requests strictly below node vCPU” became a P0 checklist item. Second, at literal zero nodes the autoscaler has no real node to inspect; it builds a virtual node purely from the ASG’s node-template label and taint tags. With those missing, it concludes the group can never host the pod and skips it — the pod stays Pending with no error mentioning the tags. Both failures are silent by construction — what makes scale-from-zero its own discipline.
Probes that lie. A SELECT 1 readiness probe is constant-folded by the frontend and never dispatched to any backend — it happily reports an empty compute group as healthy. The real data-plane probe is SHOW BACKENDS Alive plus a canary query that touches storage.
Red that means healthy. A compute group parked at zero replicas makes the operator’s aggregate cluster-health field read red. That is a monitoring quirk, not a fault: scale-to-zero is the design working. Sharper still: a group scaled to zero can leave an orphaned lease in the metadata service, blocking other groups’ compaction until it expires — learned the hard way; idle states need their own operational review.
Spot versus on-demand, per pool. The serving pool is never spot: a reclaim is a query outage plus a cache dump. The heavy pool tolerates interruption by construction — the frontend detects a dead backend via ~2 s heartbeats, locks release in ~7 s, and a failed query retries once, all inside spot’s 2-minute reclaim window — and spot pricing runs ~62% below on-demand for the node class. But floor-zero already makes the absolute on-demand cost small, and a mid-query reclaim on a 60–500 s query wastes the work plus a ~3–4 min re-scale. The decision: on-demand until a node-termination handler is installed, then a spot-first mixed ASG with on-demand fallback — tied to measured node-hours, not dogma.
Productionization: pull the decisions out of the client
The first working integration left too much intelligence in the query-execution client, which probed the frontend, chose replica counts, and judged queue pressure itself. The production evolution converges four capabilities into the controller. Launch is formalized as a declarative, raise-only, idempotent target — repeated launches with the same N are no-ops, and concurrent launches coalesce onto a single handle at the max requested target. Scale-up becomes a fixed increment rather than precise scale-to-N. The status endpoint exposes alive-backend counts and a serve-ready bit, so clients stop probing the frontend directly. And the controller’s monitor reads waiting_query_num from SHOW WORKLOAD GROUPS, scaling up on sustained queueing, capped at the pool max. The client’s only conversation is with the controller.
Takeaways
- Elasticity is an economics argument before it is an infrastructure pattern: measure the compute concentration and the burst shape first; ~92–97% of a static pool’s cost was simply idle time.
- When a platform operator owns the state, bespoke tooling should shrink — the correct size for this scaler was one patched field on one custom resource.
- The characteristic failure mode of scale-from-zero is silence: schedulers, probes, and health gauges that report nothing wrong while nothing happens. Design the readiness path assuming every signal will eventually lie, because each one did.