Observability for a Shared-Data Doris From Zero: Two-Tier Triage and Twelve Alerts

Posted on In
Three dashboards and 115 panels organized so an on-call engineer can go from "is it broken?" to "which layer?" in two clicks, plus an alert set deliberately cut from 27 candidates to 12 — each kept because a real failure had gone unnoticed for 35 hours, 16 days, or five repeats.

Also in Chinese: 中文版

Starting point: nothing, and a platform that fails quietly

When I started, the shared-data Doris deployment had no dashboards and no alerts. That is a normal state for a platform that has not shipped yet, and a dangerous one for a platform whose failure modes are mostly silent. Compute groups scale from zero, metadata lives in a separate service backed by a distributed key-value store, and a control-plane component decides replica counts. Most of the ways this stack breaks do not produce errors; they produce a pool that quietly stops doing its job.

Three real incidents from the build-out set the design brief, because each one went unnoticed for a long time:

  • A warm-up job pointed at a compute group that had been deleted and failed for 35 hours.
  • Platform CI overwrote the compute platform’s state 5 times before anyone connected the repeats.
  • The metadata store’s single health signal sat frozen for 16 days while looking perfectly green.

None of these would have shown up on a latency or error-rate panel. That observation drove everything below.

Dashboards: two tiers of triage, not one big wall

There are three dashboards, 115 panels in total, with exactly one job each.

Dashboard Panels Question it answers
Summary 31 Is anything broken, and in which layer?
Details 60, in 13 sections Within that layer, where and why?
Autoscaler health 24 Is the control plane alive, deciding, and acting?

The organizing idea is two-level triage. Summary is built to be read in under a minute by someone who just got paged and knows nothing: it is ordered by layer, so the first screen already narrows the search. Details is split into 13 sections along the same lines, so a drill-down from Summary lands in a named section instead of a search through 60 graphs. The autoscaler gets its own dashboard because its failure mode is different in kind: when it breaks, every other panel can stay green while capacity stops tracking demand.

The tradeoff is that this is more panels than anyone wants to read. I accepted it because Details is a lookup table, not a dashboard to watch, and the cost of a missing panel during an incident is far higher than the cost of an extra one.

Honest boundary: the dashboards are deployed and every panel’s query was checked against the source JSON, but the data comes from preprod and sandbox environments, and I did not do a visual render check of each panel. They are untested against production traffic shape.

Alerts: 27 candidates, 12 kept

I wrote 27 candidate rules and validated each expression against real metrics in VictoriaMetrics. Then I cut 15. The filter was one sentence: keep a rule only if it catches a silent failure or an irreversible one. Anything that a human would notice anyway within minutes, or that is a symptom of something already covered, was deferred.

The 12 that survived fall into three groups:

  • Availability (4). The always-on head backends of each pool, the frontend and metadata service, and a query error-rate rule that is suppressed by the first three so one outage produces one alert, not a storm.
  • Control plane (4). Autoscaler tick stalled, tick series absent entirely, a scaling decision made but never acted on, and replica writes failing. These are the rules that would have caught the 5-times-overwritten and 35-hour incidents.
  • Metadata safety (4). Distributed store fault tolerance exhausted, coordinator unreachable, metrics absent, and a delete-bitmap failure counter. The metrics-absent rule exists because of the 16 days.

Every rule has a hedge against its own silence. “Metrics absent” and “tick series gone” are rules about the monitoring going blind, because an alert that depends on a series that vanished will never fire. That is the lesson from the frozen health signal: an always-green line is not evidence of health.

Rules do not page anyone yet. They route to a test channel with no on-call attached, with the intent to upgrade to paging at production cutover. I verified routing with a small simulator that replayed nine non-Doris label sets before and after the change and confirmed existing routes did not move.

What this is not

Being exact about scope, because the title says observability and a reader will assume more:

  • The 12 rules are designed, not deployed. The change is in review. Number of alerts that have ever fired in production: zero, by construction.
  • They are not SLO burn-rate alerts. They guard against silent and irreversible failure, which is a narrower and arguably more important set than latency objectives. A real SLI/SLO layer for query latency is still to build, and I would rather say so than rename these rules to fit.
  • I did not run amtool check-config or mutation tests (deliberately breaking a thing to see whether the rule fires). The expressions are verified against real data; the firing behavior is not verified end to end.

What I would defend

Cutting from 27 to 12 was the useful work. Writing 27 alerts takes an afternoon. Deciding that 15 of them would only teach people to ignore the channel took longer, and it is the part that transfers. The design rule — alert on silence and irreversibility, and pair every rule with its own blindness check — came from reading three incidents backward, not from a template.