The Disk That Was Always Sixty Days From Full
A short detective story about a disk alert that fired, fixed itself, and turned out to be measuring nothing.
An alert landed in the channel: a production node’s disk was “predicted full within 60 days”. A little later, without anyone touching anything, it said resolved. Leadership asked the obvious question: so, will it be full in 60 days or not?
The honest answer turned out to be: the alert had no idea either.
The message
The Project runs a private blockchain network on Kubernetes. Every node keeps the whole chain on a persistent volume. A family of alerts watches those volumes and warns if one is heading towards full in 90, 60 or 30 days.
The warning read, roughly:
Summary: node volume predicted full within 60 days
Description: Based on the last 6h trend, the volume is projected
to run out of space within 60 days.
Then, a while later, it cleared.
An alert that fires and clears on its own is one of two things. Either a real problem came and went, or the alert reacts to something that isn’t a problem. Sixty days of headroom doesn’t come and go in an afternoon. So we started with the second option.
Reading the rule instead of the message
The rule behind it, in Prometheus terms:
predict_linear(volume_available_bytes[6h], 60 * 24 * 3600) < 0 for: 1h
predict_linear fits a straight line through a window of samples and extends it forward. Here the window is 6 hours and the forecast is 60 days. That’s 240 times longer than the data it saw.
Some quick arithmetic with round, illustrative numbers. Take a 100 GiB volume with 60 GiB free. To “run out within 60 days”, free space has to fall by more than 1 GiB a day. Over the 6-hour window, that’s about 250 MiB.
So the alert fires whenever the node writes a net quarter of a gigabyte in any six-hour stretch, and keeps it up for an hour.
The rule never asks how full the disk is. A volume at 40% and one at 95% get exactly the same forecast if the last six hours looked the same.
What writes a quarter-gig in six hours
Blockchain nodes, like most modern databases, sit on an LSM-tree storage engine (RocksDB here). Free space doesn’t fall smoothly on these. It moves in a sawtooth: new data piles up, then compaction merges files and gives space back.
free space │╲ ╲ ╲ │ ╲ ╱╲ ╲ ╱╲ ╲ ╱ │ ╲╱ ╲ ╲╱ ╲ ╲╱ long-run trend: almost flat │ ╲╱ ╲╱ │ │ ├──6h──┤ ← fit a line here and extend it 60 days: │ ╲ │ ╲ "full in 60 days!" └────────────────────────────────────────────── time
Any six-hour window on a falling edge produces a steep line. Extend it far enough and it crosses zero. Then compaction runs, the next window is flat or rising, and the alert resolves. That explains both halves of what we saw: the fire and the clear.
There was also a more specific suspect. A few days earlier the network had been upgraded to a new node software version, which converts its on-disk database format the first time it starts. Nodes were restarted one at a time, and this one came near the end. A one-off migration burst on a single node, extended over two months, is a very convincing-looking catastrophe.
The twist: crying wolf hides the real wolf
It would be easy to conclude “noise, ignore it”. But there’s a catch.
This kind of node doesn’t prune its history, so its disk really does grow over time, just slowly. A real trend exists. It’s measured in weeks, and the alert can’t see it through the noise. Every false alarm trains people to ignore the next one, including the one that’s eventually true.
So the fix couldn’t be “turn it off”. It had to be “make it measure what it claims to measure”.
The fix
Three changes, applied the same way in all three environments:
- 90 days (info): window 6h → 3 days, only when the disk is more than half full, must last 1h → 6h.
- 60 days (warning): window 6h → 3 days, only when the disk is more than half full, must last 1h → 6h.
- 30 days (critical): window 6h → 1 day, only when the disk is more than half full, must last 1h → 3h.
Why each part:
- A window measured in days. A three-day fit covers many compaction cycles, and they average out. The forecast is now 20–30× the data, not 240×. We checked first that Prometheus keeps enough history: the chart’s default retention is 10 days, so a three-day window has data.
- A fill gate.
and (1 - available / capacity) > 0.5. Below half full, plain “80% used” and “90% used” alerts already exist, and they’re the ones that matter for a disk that suddenly fills fast. - Hours, not an hour. One burst shouldn’t page anybody.
And a way to answer the original question properly, instead of trusting any alert:
available / -deriv(available[7d]) / 86400 # days until full, from a week of data
A large number, or a negative one (the disk isn’t shrinking), means relax.
The one-sentence version
Leadership didn’t need any of the above. They needed this:
The alert projected a few hours of normal, temporary database write activity on that node 60 days into the future, as if that pace would continue nonstop, so it warned about a full disk that was never really coming; we’ve changed it to look at several days of data and to fire only once a disk is already more than half full.
Coda: the fix that almost shipped in someone else’s pull request
The change itself was ten minutes of YAML. Getting it merged cleanly took longer.
Another automated session was working in the same local clone of the repository at the same time. Between creating our branch and committing to it, the other session switched the checkout back to its branch. Edits to files it didn’t touch carried over silently, and our commit landed on its branch. Our own branch was still identical to main, so opening the pull request failed with “no commits between main and branch”.
By the time we noticed, the other session had committed on top of ours and pushed. Its pull request had been merged, but by luck only with its earlier commit.
The recovery was to copy our commit onto our own branch in a separate git worktree, ship it, and leave the other branch alone. Rewriting someone else’s branch to take your commit back is the kind of tidy-up that causes the next incident. Since then, every change goes through its own worktree.
We also couldn’t run the rule linter locally. Fetching it was blocked by a safety policy. We relied on the fact that the Prometheus Operator rejects an invalid rule at admission, which shows up as a failed GitOps sync rather than silently missing alerts, and wrote that check into the PR.
Hints for the reader
- Compare the window with the horizon. A
predict_linearthat extends its data more than about 30× is mostly measuring noise. Days of data for weeks of forecast. - Storage engines breathe. LSM databases (RocksDB, LevelDB, Cassandra and friends) free space in bursts. Short-window trends on them are sawtooth artefacts.
- Gate predictions on fill level. “Full in 30 days” on a 10%-full disk is almost always an artefact. Let usage-percentage alerts cover the low end.
- Upgrades and migrations make one-off bursts. Expect trend alerts to fire during rollouts, and make sure the rule’s window outlasts them.
- An alert that resolves itself is evidence too. If nobody acted and it cleared, ask what changed in the data, not in the system.
- Fix noisy alerts, don’t mute them. The slow, real trend behind the noise is the thing you’ll eventually need to see.
- One working copy per agent. If several sessions or tools share a clone, give each change its own worktree.