Doris Capacity and Disaster Recovery: What a Drill Is Actually For
Also in Chinese: 中文版
Two questions, one habit
Before a data platform takes production traffic, two questions get asked in the wrong order: how big does it need to be, and what happens when it breaks. Both have the same failure mode. People answer with a number that looks authoritative and was never checked against the thing it describes. This page is about two places where I checked, and what I found.
A scope note up front. Everything here was measured on pre-production environments (a production-scale preprod cluster and a smaller dev cluster), before the platform went live. I do not claim a validated recovery time objective or recovery point objective, and the capacity model was built for one region of production only.
Capacity: the 2.5x error was a unit confusion
An earlier sizing document planned storage and pool depth against 15 TB. That figure was an EBS volume size, the disk attached to a node, not the amount of data. I redid the plan from the measured source: three wide tables holding roughly 7 billion rows and 6.4 TiB on the source system. Replacing volume size with data size removed about 2.5x of over-provisioning. A plan that overbuys by a factor of two and a half does not fail loudly. It simply costs that much every month.
The second half was pool depth, which is a concurrency question, not a storage one. I took 90 days of hourly peak concurrency and removed 35 pile-up hours, where a backlog had inflated the number rather than demand. The distribution:
| Percentile | Hourly peak concurrency |
|---|---|
| p50 | 9 |
| p90 | 23 |
| p99 | 60.8 |
| max | 138 |
The gap between p50 and max is 15x, and that gap is the whole design problem. Sizing for the max buys capacity that sits idle nearly all of the time; sizing for the median guarantees queueing at every peak. p99 is a reasonable place to anchor a provisioned floor, with the tail handled by elastic capacity rather than by owning it permanently. I wrote it as a basis for procurement, not as a final number: several inputs in the model are marked inferred, and the traffic-replay A/B that would test the model against real load is incomplete. I do not claim a cost saving from this work.
Disaster recovery: the toolchain
I built a toolchain for backup and restore around the shared-data architecture, where table data lives in object storage and what must be protected is the metadata. It does four things: bundles metadata to S3 on a 12-hour cycle, restores in place in five phases with destructive operations isolated inside one CLI, copies a dataset between clusters through an API, and runs precheck, teardown and guard commands around all of it.
Design intent was layered: object-store versioning, metadata volume snapshots, a point-in-time layer for the metadata key-value store, and logical export. I evaluated the point-in-time layer and dropped it in late September. Also worth saying plainly: object-store versioning is off, because this was a test-only environment at the time. That is a gap to close before relying on any of this.
What the drills measured
| Drill | Environment | Result |
|---|---|---|
| In-place restore, 2.74 TB / 1,799 tablets | preprod, production scale | 55 min clear-to-promote, including about 25 min lost to a failed attempt and retry; object check passed and row counts matched baseline |
| Cross-cluster copy, same dataset | preprod | restore step 33.5 min; redesigned copy about 35 min vs about 65 min before |
| Fault injection, 17 scenarios | dev cluster | 15 scenarios verified with zero data loss; e.g. metadata volume deleted, restored from snapshot, back to Ready in about 43 s |
| Frontend plus key-value store joint recovery | dev cluster | data byte-identical after recovery |
| Full end-to-end DR run | dev cluster | clear to promote in about 22 min |
Read the first row carefully. A 55-minute number with 25 minutes of failure inside it is not a recovery time I would put in an SLA. It is a restore path plus about 25 minutes of what a single failed attempt costs. I report it with the failure left in.
The argument: the point of a drill is the gaps
The full end-to-end run took 22 minutes and, by itself, that is a pleasant number. What it produced that mattered was something else: 8 gaps in the runbook and 1 code bug. Steps that were missing, ordered wrongly, or that assumed knowledge the person running them would not have at 3 a.m.; and one bug in the tooling that only a full-path run could reach.
That is the lesson I would push on any team: a drill is not a benchmark for a recovery time, it is an audit of the runbook. A tidy RTO from a rehearsed run says how fast the author can execute their own procedure. The number that predicts a real incident is how many places a different person would get stuck, and a drill that finds none either was not realistic or was not looked at honestly. Eight gaps in a first full run is the normal, healthy result, and each is cheaper to find on a dev cluster than during an outage.
For the same reason I do not publish an RTO. One drill at one scale on a non-production cluster, with one failed retry, supports a range and a list of things to fix. It does not support a commitment.
Where it stands
The platform is live in production. The recovery numbers above predate that and come from the environments named. What is not yet done: object-store versioning enabled, a recurring drill schedule (these were one-off), and a repeat of the full run by someone other than the author, which is the real test of whether the eight gaps stayed closed.