The Case of the Pod That Died Exactly on Schedule
A short detective story about a safety net that was the murder weapon, a graceful shutdown that made everything worse, and a killing announced two days in advance to an audience of nobody.
┌──────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├──────────────────────────────────────────────────────────────┤ │ │ │ [ live demo dies mid-call, 06:00-ish ] │ │ │ │ │ ┌────────────┴────────────┐ │ │ ▼ ▼ │ │ ┌────────────┐ ┌────────────┐ │ │ │ SUSPECT │ │ SUSPECT │ │ │ │ "the node │ │ "someone │ │ │ │ hit its │ │ deployed │ │ │ │ max age" │ │ at 3am" │ │ │ └─────┬──────┘ └─────┬──────┘ │ │ │ innocent │ innocent │ │ ▼ ▼ │ │ ┌────────────────────────────┐ │ │ │ ACTUAL CULPRIT │ │ │ │ the "grace period" we │ │ │ │ added to be *kind* to │ │ │ │ this exact workload │ │ │ └────────────────────────────┘ │ │ │ │ evidence: 5,585 log entries saying "I did NOT do it" │ │ ...followed by one saying "ok, I did it" │ └──────────────────────────────────────────────────────────────┘
The Complaint
The Company runs a real-time media service on Kubernetes — the kind that carries live video and audio sessions between remote hardware and a browser. One morning, during a live demo to an external audience, the media stopped.
The report that landed with the platform team was not "the service is down." It was stranger than that:
Two edge devices lost their connection to the media service at the same time. The demo on production is broken. High priority.
Nothing was crash-looping. No pod was restarting. Dashboards were green. The service was up, answering, healthy — and the media wouldn't flow.
Those are the good cases. A dead process tells you what it is. A healthy process that isn't working makes you actually think.
Chapter 1 — The Suspect With a Perfect Alibi
The media pods carried a well-known annotation. Every autoscaler has some version of it: a marker that says do not disrupt this pod. The team had put it there deliberately, years of instinct behind it: this thing holds live sessions, don't move it.
So the first theory was the obvious one. The annotation must have failed. Somebody must have put it in the wrong place in the config — a mistake this team had made before on a different service, where the marker had been attached to the deployment object instead of the pod, where the autoscaler never looks. It rendered, it showed up in describe, and it had been decorative for the life of the service.
Cheap to check. Read the annotation off the live pod, not off the config file.
It was there. Correctly placed. On the pod.
Then came the alibi. The cluster's own event stream had been recording the autoscaler's decisions, and for this one node it had written the same line, over and over, roughly every seventeen minutes, for eleven days:
DisruptionBlocked: Pod has "do-not-disrupt" annotation (Pod=<the media service>) ...occurrence 5,585
Five and a half thousand times, the autoscaler had walked up to that node, looked at the protected pod, and walked away. The annotation wasn't broken. The annotation was the most thoroughly proven-working piece of configuration in the entire cluster.
Which raised a much better question: what killed a pod that had been successfully protected 5,585 times?
Chapter 2 — A Confession, Filed Two Days Early
Here's where I got it wrong first, and it's worth admitting because it's the mistake the docs invite you to make.
My opening theory was node age. This autoscaler has a setting for maximum node lifetime — 30 days by default — and expiry is a forceful action: it doesn't care about your do-not-disrupt marker. That fits. That's the textbook answer.
So I checked the ages of every node in the cluster. The oldest autoscaler-managed node was eleven days old. Not one of them was anywhere near thirty days. The theory was dead on arrival, killed by one query.
Wrong turns are cheap when you check them. They're expensive when you write them into the incident report.
Back to the event stream. The node that had hosted the pod was already gone — deleted, unrecoverable, no object left to inspect. But its events survived, and they told the whole story with timestamps. Reading them in order:
Day 0, 06:16:32 DisruptionTerminating: Disrupting Node: Drifted
Day 0, 06:16:32 FailedDraining: Failed to drain node, 19 pods are waiting to be evicted
Day 0, 06:16:32 TerminationGracePeriodExpiring:
All pods will be deleted by Day 2, 06:16:32
Read that third line again.
Two days before the demo broke, the cluster wrote down the exact second it intended to kill the pod. Not a warning. Not a risk. A scheduled execution, with a timestamp, in a stream nobody was watching. Between that line and the outage, the same node emitted 517 more "failed to drain" events. No alert fired for any of them.
And the reason was right there in the first line: Drifted.
Chapter 3 — The Safety Net Was the Trap
Drift means the node no longer matches what the config says it should be. In this case the node pool was configured to always use the latest base OS image, so every time the cloud provider published a new one, every node in the pool instantly became "wrong" and eligible for replacement. Routine. Healthy, even — it's how security patches land.
But drift is supposed to be a polite operation. It's supposed to respect the do-not-disrupt marker. That's the whole point of the marker.
The documentation, read slowly, says something more precise. The marker excludes a pod from consolidation unconditionally, and from drift conditionally. And here's the condition, verbatim:
ConfiguringterminationGracePeriodaffects disruption eligibility forDrift. When set, a node can be disrupted via drift even if pods have blocking disruption budgets or the do-not-disrupt annotation.
The node pool had a terminationGracePeriod of 48 hours.
Somebody had added that setting for the kindest possible reason. It reads exactly like a safety net: give pods up to two days to finish what they're doing before you kill them. Who wouldn't want that for a service holding live video calls?
What it actually does is two things, and both of them remove protection:
- It makes drift ignore the do-not-disrupt marker entirely.
- It converts "this drain is blocked" into "this drain has a deadline," after which pods are deleted regardless of any protection.
The setting that looked like the seatbelt was the thing that unbuckled it. For eleven days the marker had held off consolidation — the other kind of disruption, the one it does block unconditionally — and that unbroken streak of 5,585 successes is precisely what made everyone trust it.
Nothing lies to you quite as convincingly as a safeguard that works for weeks against the wrong threat.
Chapter 4 — The Five-Hour Ghost
There's still a hole in the timeline. The execution was scheduled for Day 2, 06:16. The demo broke around 06:00. Close — but the replacement pod had actually started five hours earlier, at 01:16.
Why would the pod be deleted five hours early?
Because of its own grace period. The autoscaler is careful about this: if a pod asks for time to shut down cleanly, it starts the deletion early enough that the pod gets its full allowance before the hard deadline. So:
pod deletion = node deadline − pod's own grace period
01:16 = 06:16 − 5 hours
The media service's chart shipped a default grace period of 18,000 seconds — five hours. Presumably so a long call could finish in peace.
And here is the second twist, the one that turned a pod restart into an outage.
The instant a pod is marked for deletion, Kubernetes removes it from the service's list of healthy endpoints. That's correct behaviour: stop sending new traffic, let it finish. But this media service isn't reached through ordinary cluster networking. A relay gateway sits at the edge and forwards media directly to specific pod addresses — and it learns those addresses from exactly that endpoint list.
So at 01:16 the pod left the endpoint list, and the relay immediately stopped being willing to carry media to it. Meanwhile the pod kept running. It kept its existing sessions. A shared session registry kept routing participants to it, because as far as the application was concerned, it still owned those rooms.
01:16 ──────────────────────────────────────────► 06:16
│ │
│ pod: RUNNING, owns live sessions │
│ relay: "I don't know that address" │
│ result: sessions exist and cannot carry media │
│ │
└──── five hours of live-but-unreachable ─────────┘
▲
the demo was here
Five hours of a service that was up, healthy, holding real sessions, and structurally unable to move a single packet of media. The grace period intended to protect calls in flight didn't save one of them. It extended the broken state from seconds to hours, and put the demo squarely inside the window.
A graceful shutdown is only graceful if everything that routes to you agrees you still exist.
Chapter 5 — The Fix That Would Have Done It Again
The fix looked easy: take that terminationGracePeriod off the node pool, give the media service its own dedicated nodes, and stop the whole class of problem.
One more detail nearly turned the repair into a repeat performance.
Changing the node template on a pool — lifetime, grace period, machine types, any of it — drifts every node already running in that pool. The docs say so plainly. And existing nodes keep their own copy of the old grace period, immutable, for as long as they live.
Put those two facts together. Editing the grace period to stop drift-driven evictions would mark all ten production nodes as drifted — while they were still governed by the very setting that lets drift bypass the protection. The fix would have re-run the incident on every node in the cluster, on a 48-hour fuse.
The way out is ordering. Some parts of that config are node identity, and touching them replaces machines. Other parts — the disruption rules themselves — are pure policy and change nothing about the nodes. So:
- Freeze drift first, using a budget that allows zero drift-driven replacements. It's policy, not identity: nothing drifts, it takes effect immediately.
- Create the new pool. A brand-new pool drifts nothing.
- Move the workload deliberately, at a chosen hour, with someone watching.
- Then edit the old pool, and unfreeze drift inside a maintenance window.
One more trap sat in the isolation half of the fix. A dedicated node pool sounds like it dedicates nodes. It doesn't — it's a template describing machines the autoscaler is allowed to create. It has no opinion whatsoever about which pods land there. Getting real isolation takes two matching pairs: a label and a taint on the pool, and a selector and a toleration on the pod. Miss the selector and your workload cheerfully stays on the shared nodes while the shiny new pool sits empty, looking like it works. Miss the taint and everything else moves in.
And one small piece of poetry in the wreckage: freezing drift means those nodes stop receiving OS patches. The fix for "our nodes get replaced when we don't want them to" is, temporarily, "our nodes never get replaced." That has to be a scheduled window, not a permanent state, or in six months it's the next story.
Hints for the Reader
- A safeguard that has worked a thousand times may be working against the wrong threat. The unbroken streak is not evidence of protection; it's evidence that the thing you feared hasn't been tried yet. Ask which failure modes your safeguard doesn't cover, and write them down next to it.
- "Do not disrupt" markers are not finalizers. In every scheduler I've met, they stop the system from voluntarily choosing your workload. They do not survive forced operations — expiry, hardware interruption, node repair, a human with
delete— and sometimes, as here, a neighbouring setting quietly downgrades them. - Read the conditional clauses in the docs. "Unconditionally excluded from A, conditionally excluded from B" is the single most important sentence in this whole story, and it's the kind of sentence people skim.
- A long shutdown grace period is a liability, not a courtesy, if anything outside normal cluster networking resolves your pods through the service. The pod leaves the endpoint list the moment it's marked for deletion, but keeps serving. Size that window to how fast the work actually drains, not to how long the work could last.
- Your infrastructure may already be telling you the exact second it plans to break you. This cluster wrote a scheduled deletion timestamp two days ahead and repeated a failure event 517 times. Every ingredient of the postmortem existed before the incident. Alert on drains that don't finish, and on nodes stuck deleting — the events are usually free and already there.
- Before fixing a replacement problem, find out which fields cause replacement. In any declarative system, some fields are the desired identity of the thing and editing them replaces it. Learn which half of the config you're touching before you touch it, or the repair becomes a slower copy of the incident.
- When a controller's own decisions are recorded somewhere, read them before theorising. Config tells you intent. The event log tells you what actually happened, with reasons attached, and it very rarely agrees with the tidy theory you formed at 9am. My first theory here was node age; one query killed it. That's the cheapest kind of wrong to be.