Restoring 5.2 billion rows without touching prod
Also in Chinese: 中文版
Correcting the premise
The initial plan assumed rows expired by the database’s TTL could be recovered from the live volume. That assumption was wrong, and proving it wrong was the first deliverable: this engine’s TTL delete physically unlinks the data parts, and no S3 tiering existed to catch them. Nothing recoverable remained on the live system — and experimenting on a live disk to confirm that would have been risk without payoff.
The recovery architecture
What did exist was an immutable daily EBS snapshot series of the production volumes. The restore path: materialize a snapshot to a brand-new, independent volume, attach it to a separate database instance, and verify there. No attach/detach on any live disk, no connection to any production system at any point. The blast radius was not “managed” — it was structurally zero, provable from the design before execution.
Verified, with numbers
The restored dataset came up live and queryable: 5.198 billion rows / 4.07 TiB. A full count(*) completed in 50 s; a full-table cold aggregation in 83 s. A restore is not a restore until you have queried it — a mounted volume proves storage, not data.
Honest boundaries
Two limits, named on purpose. The implied RPO is roughly 24 hours — that is what a daily snapshot series gives you, and nobody had written it down as an SLA before this drill made it visible. And this was a one-off drill proving the path works at full scale, not yet a scheduled DR program. Both gaps are now explicit, which is the difference between a gap and a surprise.
The most useful output of a restore drill is rarely the data. It is the corrected assumptions and the numbers (RPO, restore time) that nobody had been forced to state before.