← Back to Blog

The disaster recovery plan that already worked — we just couldn't prove it

The ask

The CTO of a financial trading platform sent over a filled-in disaster recovery questionnaire and a third-party compliance assessment of AWS's backup service against SEC recordkeeping rules. The message behind it was clear: "before we scale further, are we actually protected against a disaster?"

The questionnaire had real numbers in it. Production database recovery: under an hour, no more than 60 minutes of data loss. Staging and dev environments: a four-hour window was fine. A break-glass process, restricted to three named people. Slack-only alerting — no pager, no on-call escalation. A retention answer of "two weeks" to a yes/no compliance question, sitting oddly next to a compliance report that kept citing multi-year regulatory retention rules.

The obvious next step looked like: go build backups that hit those numbers.

The investigation

First stop was the infrastructure-as-code repo itself: grep for anything backup-related.

$ grep -rl "aws_backup\|backup_vault\|backup_plan\|vault_lock" .
(no matches)

Zero. No AWS Backup vaults, no plans, nothing. On paper, this looked like a green-field build.

Then the actual database configs told a different story. Every database instance, across every environment, had the same three settings:

multi_az            = true
backup_retention_period = 7   # days
deletion_protection = true

multi_az = true gives automatic failover in the 60–120 second range. Native automated backups already give five-minute-granularity point-in-time recovery. Run the math against the CTO's own numbers — sub-1-hour recovery, sub-60-minute data loss for production; sub-4-hour, sub-24-hour for the rest — and the stated targets were already being met, today, with zero custom tooling.

That reframed the whole problem. This wasn't "we have no disaster recovery." It was "we have disaster recovery, we just have no retention depth beyond a week, no tamper-proof records, no cross-region copy, no break-glass role, and no way to prove any of it to an auditor." A very different, and much more interesting, problem.

The first landmine: a security control that would silently reject the "obvious" fix

The platform ran validator nodes for the underlying blockchain network — infrastructure holding private key material considered too sensitive to ever leave the account. A defense-in-depth control, rolled out independently and already partly live, denied the exact snapshot/copy API calls a backup tool needs — specifically for anything tagged as validator infrastructure.

Which meant: a lazily-scoped "back up everything with this tag pattern" backup selection would, at best, silently skip the validator volumes if scoped correctly — or generate a wall of failed-job alerts every night if scoped by wildcard. Either way, relying on the security control as a silent backstop was the wrong call. The backup selections needed their own explicit exclusion, scoped on their own terms, not "the other team's control will save us." Two independent layers, deliberately redundant — the same shape the existing security work already used elsewhere.

The second landmine: building the vault at the wrong moment

Digging into recent history turned up something adjacent but critical: an in-flight migration to enable storage encryption on the production database, not yet cut over. The live, currently-serving instance was still unencrypted; an encrypted replacement existed in parallel, empty, waiting.

Here's the trap: if you stand up an immutable, tamper-proof backup vault right now and point it at the live database, the first "permanent, provably untampered" recovery point you produce is a snapshot of an unencrypted database — the exact thing the encryption migration exists to retire. Lock that in, and you've cemented the wrong artifact.

The fix wasn't technical, it was sequencing: finish the encryption cutover first, then point the backup plan at the encrypted instance. If backup coverage was judged too urgent to wait, the fallback was to build against the current instance with an explicit "interim, will be repointed" comment in the config — never silently let the target drift across the cutover.

The third landmine: "permanent" doesn't mean what you think it means

The compliance report pointed at a specific mechanism for tamper-proof retention: a locking feature with two modes. One is reversible by an authorized person, logged, auditable. The other is not reversible — not by the account owner, not by the cloud provider's own support team, once a grace period lapses. No feature permits privileged delete. Full stop.

Given that the CTO's own retention answer read like an incomplete response to a yes/no question — "two weeks" sitting next to citations of multi-year regulatory retention windows — turning on the irrevocable mode now, with any number, was a bet nobody could take back. Lock in two weeks and the real requirement turns out to be seven years? No undo button exists.

So the recommendation was: reversible mode everywhere, for now. Logged, audited, gated behind the same break-glass process as everything else. That's a legitimate, defensible position under the compliance framework being targeted — not a weaker stand-in for the "real" answer. It just isn't the irreversible one, and it doesn't become the irreversible one until the retention number and the exact record sets requiring it come back from whoever owns that call.

The fourth landmine: one word hiding a structural surprise

The questionnaire asked about staging and dev as a single bucket, and the CTO's answer said they should "share a vault." Reasonable-sounding, until you check: staging and dev live in two separate cloud accounts. A "shared vault" between two accounts is a cross-account sharing problem — access policies, cross-account copy jobs — not "point two things at the same shared resource within one account." Nothing like it existed anywhere in the infrastructure yet.

Rather than silently building the more complicated version because the questionnaire's phrasing implied it, this got kicked back as an open question: does "shared" really mean cross-account, or would two identically-configured vaults — one per environment — satisfy the actual intent more simply? Sometimes the fastest way to avoid over-engineering is just to ask what a word meant.

The twist: what "surviving a regional outage" actually needed

The CTO had explicitly flagged uncertainty about whether the production environment needed to survive an entire cloud region going down, and leaned toward "probably not." The tempting answer was to just agree and move on. But the honest analysis landed somewhere in between:

Full failover — a live standby cluster and database replica in a second region, ready to take traffic instantly — wasn't worth building. The platform's actual uptime guarantee came from dozens of independently-hosted validator nodes distributed outside the company's own infrastructure entirely; a regional outage wouldn't take the network down, only the company's own application layer. And running a partial second-region app stack that still can't reach a healthy version of the thing it depends on solves half a problem at real ongoing cost.

But stopping at zero cross-region protection left a real gap: a genuine single-region data-loss event would mean the backups themselves are gone too. The cheap middle ground: copy each backup's recovery point into a second region automatically. Storage cost only — no standby compute, no cross-region networking to maintain. It doesn't improve the recovery-time number (restoring into freshly-built infrastructure in a new region is slow), but it does mean a regional disaster is "eventually recoverable" instead of "unrecoverable." Framed explicitly as a restore guarantee, not a failover guarantee, so nobody mistakes it for something stronger than it is.

The last gap: there's no "3 people must approve" button

The break-glass design called for three named people to sign off before anyone could restore or delete a production backup. Turns out cloud identity systems have no built-in primitive for that at all — role assumption is a single-principal action, period. No "require N co-signers" condition exists to configure.

The answer, again, was procedural rather than technical: restrict the role's trust policy to exactly those three individual identities, require multi-factor authentication on every assumption, and require a documented, logged sign-off from all three in an incident channel before anyone actually assumes the role — audited after the fact by the platform's activity log, not gated by a technical mechanism. Weaker than a true quorum system in theory (nothing stops one person from skipping the wait), but it matches how break-glass processes are meant to work in practice, and avoids building bespoke approval infrastructure for what is fundamentally a human process.

One tension got flagged rather than resolved: requiring all three, with no fallback, means the restore is blocked entirely if even one person is unreachable during a real incident — which could itself blow the one-hour recovery target the rule was meant to protect. That's not an infrastructure team's call to make unilaterally; it went back to the CTO as an open question.

The takeaway

Nobody wrote a line of infrastructure code in this pass. The whole exercise was investigation and sequencing: check what's actually true today before building anything, distinguish "the numbers are already met" from "the numbers are unproven," and refuse to let an ambiguous questionnaire answer, a tempting shortcut, or an irreversible setting get resolved by assumption.

The generalizable pattern, past this one incident: