The Alarm That Was Red On Day One
We built a monitor to catch backups that quietly stop happening. The next morning it was screaming. It took a while to work out that the monitor was the thing that was broken — and longer to notice it had been right about something else entirely.
The gap we were closing
The Project is a trading platform. Its database is backed up nightly and copied into two separate AWS accounts — one for account isolation, one in a second region for a genuine regional disaster.
Failure alerts were already wired up: if a copy job fails, an event fires and lands in a chat channel. That covers a copy that ran and broke.
It does not cover a copy that never ran at all.
Delete the schedule, detach the selection, revoke a permission in the right way, and no job is ever created. No job, no failure, no event, no alert. The channel stays quiet — and quiet is exactly what a healthy night looks like. The two states are indistinguishable from the alerting channel alone, and the silent one is the dangerous one.
So we added a presence check: an alarm that fires when the count of completed copy jobs drops below one over a long window, with missing data explicitly treated as breaching. Because the metric only exists when work happens, "no data" is the whole point.
We deployed it. The next morning, every environment was red.
First question: is this real?
The instinct on a red backup alarm for a trading database is to assume the worst. The instinct is wrong often enough that it's worth one minute of checking.
Two facts arrived quickly:
Copy jobs were completing. Listing them showed clean daily runs to both destinations for the previous three days. Nothing had stopped.
All three environments were red, not just one. Same config pattern, deployed everywhere, failing everywhere. Infrastructure failures don't usually respect a config boundary that neatly. That's the signature of a bug in the thing you just shipped.
The alarm's own state reason said it plainly:
Threshold Crossed: no datapoints were received for 2 periods and 2 missing datapoints were treated as [Breaching].
Not "the count was zero." No datapoints at all. The alarm was watching a metric series that didn't exist.
The twist: the label points the other way
The alarm was scoped by vault name, and we'd scoped it to the vault the backups are copied from. That felt obviously right — the metric is published by the source account, so surely it's tagged with the source vault.
Listing what the platform actually publishes settled it:
NumberOfBackupJobsCompleted vault = <the local source vault> NumberOfRecoveryPointsCompleted vault = <the local source vault> NumberOfCopyJobsCompleted vault = <destination A> NumberOfCopyJobsCompleted vault = <destination B>
Copy-job metrics are labelled with the destination. The source vault name appears only on backup jobs and recovery points. Our alarm asked for a combination that is never emitted, got silence, and dutifully treated silence as breaching — which is exactly what we told it to do.
The twist that was worth more than the fix
Weeks earlier, working from that same wrong assumption, we had written down a limitation.
The reasoning went: both copy destinations share one source vault, so both land on one metric series. An alarm on that series stays green if one destination silently dies, because the other keeps the count above zero. Detecting that would need monitoring inside the destination account, which had no alerting stack. We recorded it as an accepted gap and moved on.
Every part of that was downstream of the wrong label.
Because the metric is keyed on the destination, each destination has its own series. Which means you can simply run one alarm per destination — and the failure we'd written off as undetectable becomes the easy case. The second-region copy going quiet while the first keeps working is precisely the scenario that copy exists to protect against, and now it pages.
A documented limitation dissolved into two lines of config. It had never been a limitation. It was a wrong assumption wearing a limitation's clothes.
We wrote a note next to the new alarms: do not merge these back into one. An aggregate across both destinations goes green whenever either one alone still works.
Green, but not healthy
We fixed the label, redeployed, and everything went green. Job done.
Except one alarm's state reason read:
Threshold Crossed: 1 datapoint [1.0 (05:36:00)] was not less than the threshold (1.0) and 1 missing datapoint was treated as [Breaching].
Green — while already counting one breaching datapoint out of two. One nudge from firing.
Chasing that turned up the last surprise. When you query a metric, the platform buckets it on clean clock boundaries — midnight, noon. When an alarm evaluates that same metric, it does not. Its windows are anchored to the moment the alarm was created or last updated.
We had deployed at 05:34. The copies finish between 05:23 and 05:45. We had placed a bucket boundary directly on top of the event we were watching, so a few minutes of ordinary jitter drops a run into the neighbouring bucket and manufactures a gap.
deployed 05:34 ──┐
▼
bucket A ┤───────┼──────────────────────┤ bucket B
05:34 │ 17:34
│
copies finish ~05:23 – 05:45 ── some land left of the
boundary, some right
Nothing in the configuration says this. Two identical definitions deployed six hours apart behave differently, and the difference is invisible in code review. The fix is to deploy at a time far from the watched event — which is real, load-bearing state living nowhere but in the deploy timestamp.
The ceiling nobody mentions
One more thing worth knowing before you build this.
The natural design is a window slightly longer than the schedule: for a daily job, "alert if nothing completed in about 26 hours." That gives some slack for a late run.
You can't build it. The platform caps the alarm window at 24 hours — period × evaluation_periods ≤ 86400. For a daily job the maximum window is exactly the job's interval, leaving at most one bucket of slack however you arrange the periods.
That's not a tuning problem. If an alarm like this starts flapping, don't retune it — the shape can't be made comfortable. Move to something that takes an age threshold directly ("no recovery point newer than 2 days"), which has no such ceiling.
Hints for the reader
Failure alerts and absence alerts are different monitors. Having one and believing you have both is the actual risk. Any scheduled, invisible, valuable job — backups, replication, archival, reconciliation — needs a presence check, because nobody notices absence until they need the thing.
Before you trust an alert, spend one minute on "is this real?" Check whether the underlying work actually happened, and whether the alert failed everywhere at once. A pattern failing uniformly across independent environments is a config bug, not an outage.
A wrong configuration that screams on day one is a gift. Ours was unmissable because a nonexistent metric plus treat-missing-as-breaching pins the alarm red from birth. The alternative — a monitor that sits quietly green while watching nothing — is the same bug with a worse ending. When you choose defaults, choose the ones that fail loudly.
Read the alert's reasoning, not its colour. "Green" and "healthy with margin" are different claims, and only the detailed state distinguishes them. Ours was green and one datapoint from firing.
Check which way the label points. Metrics about a transfer can be tagged with the source or the destination, and the intuitive guess is a coin flip. List what the platform actually publishes instead of inferring it — one command, and in our case it also demolished a limitation we'd been living with.
Watch for assumptions that have already been promoted to documentation. Our wrong guess didn't just break an alarm; it had been written down as an architectural constraint and accepted. When you correct a fact, go back and check what else you built on top of it.