The Case of the Database That Looked Busy
A short detective story about a dead server that wouldn't stop talking, a chart that told the truth badly, and a message nobody ever received.
┌────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├────────────────────────────────────────────────────────────┤ │ │ │ [ delete the old cluster? IRREVERSIBLE ] │ │ │ │ │ ┌─────────┴─────────┐ │ │ ▼ ▼ │ │ ┌───────────┐ ┌───────────┐ │ │ │ EVIDENCE │ │ EVIDENCE │ │ │ │ daily: │ │ hourly: │ │ │ │ "still │ │ "dead │ │ │ │ busy" │ │ since │ │ │ │ │ │ 08:00" │ │ │ └─────┬─────┘ └─────┬─────┘ │ │ │ same data │ same data │ │ └─────────┬─────────┘ │ │ ▼ │ │ ┌───────────────────────┐ │ │ │ ONE OF THESE WAS │ │ │ │ ABOUT TO DECIDE │ │ │ │ A PRODUCTION DELETE │ │ │ └───────────────────────┘ │ └────────────────────────────────────────────────────────────┘
The Complaint
It started as the most boring kind of ticket: a compliance finding.
The Company ran a product-catalogue search cluster — a managed AWS OpenSearch domain — and the cloud provider's security scanner had flagged it, at Critical, with a one-line complaint:
OpenSearch domains should not be publicly accessible.
The cluster had a public endpoint. It was fenced off by an IP-restriction policy, so in practice nothing unwanted could reach it, but the control doesn't grade on a curve: the endpoint existed on the internet, so it failed.
The fix is well-trodden and mildly annoying. AWS won't move a live domain inside a VPC in place — the API simply refuses. You stand up a second, VPC-only cluster next to the first, migrate, point the application at the new one, and then delete the old.
That last clause is the whole story. Delete the old one. Three environments — development, staging, production — each with its own live cluster, each holding real indices, each deletion irreversible and each taking every index inside it along for the ride.
Everything before that step is reversible. That step is a cliff.
Chapter 1 — The Quiet Good Decision
Before any of the interesting mistakes, someone made a decision worth stealing.
One Terraform codebase served all three environments. The obvious way to retire the old cluster is to delete its module block from the code. Do that, and the next apply in every environment destroys its cluster — including the two that hadn't migrated yet.
So instead of deleting the block, they put a switch in front of it:
# was: count = var.enable_search ? 1 : 0 count = var.enable_search && var.enable_search_legacy ? 1 : 0
Now retiring a cluster is one line in one environment's config file. Each environment moves on its own schedule.
Then came the part that's easy to get backwards. What should that switch default to?
House convention across the Company's repos was firm: feature flags default to false, opt-in, no surprises. Every other flag in the file followed it.
This one defaulted to true, deliberately, against convention — because the two directions fail very differently:
DEFAULT SOMEONE FORGETS IT RESULT
──────────────────────────────────────────────────────────────────
false (convention) resolves to false a live production
cluster is SILENTLY
DESTROYED
──────────────────────────────────────────────────────────────────
true (chosen) resolves to true cluster kept; the
retirement just
doesn't happen yet
Normal flags gate whether something gets created, and false is the harmless default. This one gated whether something gets destroyed, so the harmless default inverted with it.
The rule underneath: a boolean's default should be whichever value does nothing when someone forgets it. Consistency is a fine goal right up until it starts deleting things. They wrote the reasoning into the variable's description, so the next person to come along and "fix the inconsistency" would find an argument waiting for them.
Development went first. One line, one apply, cluster gone. Staging followed. Both uneventful.
Then production.
Chapter 2 — The Corpse With a Pulse
Before flipping the switch on production, the sensible move: check that nothing is still using the thing you're about to delete. The migration runbook said production had been cut over days earlier. But a runbook saying an environment migrated is a claim, and the cost of that claim being wrong is a production outage, not an inconvenience.
So: pull the traffic metrics for both clusters, old and new, and confirm the old one is dead.
It wasn't dead. Over the previous week, the cluster that was supposed to be retired had handled more search traffic per day than its replacement.
Read plainly, that's a stop sign. You do not delete the busier of two databases.
And this is the point where the story could have ended badly in either direction. Believe the number and the retirement stalls indefinitely on a cluster that's actually dead. Ignore the number because the runbook says otherwise and you're deleting a live production database on vibes.
The number wasn't wrong. It was being read at the wrong resolution.
Chapter 3 — Zoom In
The daily figures were daily totals — one number per cluster per day. Same data, pulled per hour instead, told a completely different story.
OLD CLUSTER (per hour) NEW CLUSTER (per hour)
before cutover: before cutover:
26 . 26 . 26 . 26 (night) 2 . 2 . 2 . 2 . 2 . 2
60 . 77 . 95 .140 (workday) 2 . 2 . 2 . 2 . 2 . 2
108 . 62 . 26 . 26 (evening) 2 . 2 . 2 . 2 . 2 . 2
▲ a human-shaped day ▲ nothing home
─────────────── 08:00, cutover ───────────────
after: after:
26 . 26 . 26 . 26 54 . 47 . 27 .126
26 . 26 . 26 . 26 28 . 63 . 19 . 23
26 . 26 . 26 . 26 2 . 2 . 33 . 2
▲ flat. all day. all night. ▲ a human-shaped day
There it is. At one specific hour, the two clusters swapped shapes.
The old cluster's daily rhythm — quiet overnight, busy through the working day, tapering in the evening — picked itself up and moved to the new cluster. What stayed behind was a flat line: exactly the same number, every hour, through the afternoon, through the night, into the next morning.
That's not usage. Nothing a human does is that regular. The workday curve is people; the flat line is a machine talking to itself.
Daily totals had hidden this completely, because a daily total adds the flat line and the human curve together and hands you one number. The old cluster's flat line ran all twenty-four hours; the new cluster's real traffic only happened during working hours. Sum both over a day and the corpse comes out ahead.
Compare the shape, not the total.
Chapter 4 — Idle Is Not Zero
Which raises the obvious question: why is a cluster with no users doing anything?
Because managed datastores are never truly quiet. Internal system indices, health checks, the housekeeping the service does on its own behalf — all of it registers as traffic on exactly the same metrics your application uses. There is no flag separating "my code did this" from "the service breathed."
The scale of it was the surprise. Checking the already-retired environments turned up something that looked like a typo:
indexing operations per day, post-cutover, no users:
development environment ....... 92
staging environment ........... 92
▲
two different AWS accounts.
two different clusters.
identical to the digit.
Ninety-two a day, in both, forever. Meanwhile the search-side floor sat at about 26 an hour — and that number showed up again in production, in a third account.
These aren't coincidences, they're the service's resting heart rate. Which quietly demolishes the most natural rule anyone would reach for here:
"If the metric isn't zero, something's still using it."
That rule would have blocked all three retirements. Permanently. The metric is never going to be zero, because the cluster is never going to stop breathing until you kill it.
Before you use a metric to prove absence, find out what that metric reads when nothing is there.
Chapter 5 — The Control Group You Already Have
Even knowing the flat line was baseline, one question stayed uncomfortable: is this particular flat line small enough to be safe?
Asked that way it has no good answer. Is 58 a day too much? 634? Compared to what? You can talk yourself into either verdict, and neither survives a reviewer asking "how do you know?"
The way out was to stop asking in the absolute.
Development's cluster had already been examined, judged safe, deleted, and not missed. That verdict was made, accepted, and proven. So the question could be re-asked as a comparison: is staging's residual floor bigger or smaller than the one already signed off?
Smaller. Meaningfully so. Which meant staging was safe by a standard the team had already agreed to and already lived with — no new judgement call required.
This is the underrated bonus of rolling anything out one environment at a time. Everyone does staged rollouts to limit blast radius. But the environment that went first also becomes a control group — a worked example of "this was fine," available for free, that converts unanswerable absolute questions into answerable relative ones.
Production got the same treatment, and passed the same way.
The switch was flipped. The plan came out exactly as predicted: one cluster destroyed, two access policies trimmed to drop a permission pointing at something that would no longer exist, and — the assertion that actually mattered — zero changes to the replacement. For a teardown, the meaningful safety check isn't what the plan does. It's what the plan leaves alone.
Chapter 6 — The Message Nobody Got
With the deletion underway, all that remained was writing it up on the ticket: what was verified, what the evidence showed, and one genuinely open question for the team — production data isn't disposable, so do we take a snapshot before deleting, or are the old indices genuinely dead weight?
Two people needed to see that question. Both were tagged, by name, in the comment.
The comment posted. It rendered beautifully — headings, tables, code blocks, the two names in the familiar highlighted style that means you have been mentioned.
Except they hadn't been.
The tooling used to post the comment could write the tracker's rich-text format perfectly for everything except mentions. Reading an existing comment back showed mentions in what looked like a reusable syntax — so the same syntax went back in. But the identifier inside it wasn't a user's account ID at all; it was a positional placeholder, an artefact of how the tracker had translated its own data on the way out. Sending it back stored plain text wearing a mention's clothes.
No error. No warning. The comment looked, to the person who wrote it, exactly like a comment that had notified two colleagues. The only way to find out otherwise was to read the stored comment back in its raw structured form and check whether the object was a mention or merely text.
It was text.
And notice what that is. It's the same failure as the chart in Chapter 2, wearing a different hat: something that looks like a signal, isn't one. A metric that reads as usage and isn't. A mention that reads as delivered and isn't.
The half-fixed version is worse than the broken one, too. A missing mention is visible — someone notices the question went nowhere and asks again. A fake mention is invisible: the comment reads as though the right people were looped in, so nobody chases it, and an open question about deleting production data quietly sits unanswered until the day of the deletion.
The fix was unglamorous: write the names as plain text, and tag them by hand.
Summary — Hints for Fellow Investigators
- Make a destructive flag default to the safe value, even against house convention. A flag that gates creation is safe defaulting to off. A flag that gates destruction is not. Pick whichever default does nothing when someone forgets it — and write down why, or someone will "fix" it later for consistency.
- Never let a daily total decide anything. Aggregation sums the baseline and the real signal into one number and hands you the wrong answer with total confidence. Zoom in until you can see the daily rhythm.
- Compare the shape, not the total. Flat around the clock is a machine. A curve that follows office hours is people. A dip on Saturday is the most convincing evidence you'll get, because nothing synthetic knows it's the weekend.
- Find out what your metric reads when nothing is there. Idle is not zero. Managed services keep talking after the last client leaves, and "non-zero means in use" will block every retirement you ever attempt.
- The environment that went first is a control group — use it. "Is this number small enough?" has no defensible answer. "Is it smaller than the one we already approved and deleted?" does. Staged rollouts hand you this for free and almost nobody cashes it in.
- For a teardown, the safety check is what the plan doesn't touch. One destroy is easy to eyeball. Zero diff on the replacement is the assertion that actually protects you.
- A runbook saying "this environment was migrated" is a claim, not evidence. Verify per environment, not just before the first one. The check that matters is the one you'd rather skip because the previous two were fine.
- Verify delivery, not appearance. Whether it's a metric, a config annotation, or a message to a colleague — the failure mode that costs you is never the one that looks broken. It's the one that looks perfect.
WHAT THE DAILY CHART SAID WHAT ACTUALLY HAPPENED
┌────────────────────────────┐ ┌────────────────────────────┐
│ old cluster ...... busier │ │ old cluster .... breathing │
│ new cluster ...... quieter │ ═══════►│ new cluster .... serving │
│ │ │ │
│ verdict: DO NOT DELETE │ │ verdict: safe to delete │
└────────────────────────────┘ └────────────────────────────┘
same data, different resolution ────────┘