The Case of the Nodes That Wouldn't Sit Still
A short detective story from the world of Kubernetes autoscaling.
┌────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├────────────────────────────────────────────────────────────┤ │ │ │ [ node-a ] [ node-b ] [ node-c ] │ │ age: 74d age: 10m age: 3m │ │ | | | │ │ | '--------.---------' │ │ | | │ │ | ┌─────▼─────┐ │ │ | │ SUSPECT │ │ │ | │consolidate│ │ │ | │ After: 1m │ │ │ | └─────┬─────┘ │ │ | | │ │ | ┌─────▼─────┐ │ │ '-------------------->│ MOTIVE │ │ │ │ $0.23/hr │ │ │ │ "saved" │ │ │ └───────────┘ │ └────────────────────────────────────────────────────────────┘
The Complaint
It started with a vague, unsettling report from the Company's platform team:
"Something's wrong with our nodes. They keep... disappearing. And reappearing. And disappearing again."
No alarms were firing. No outage had been declared. But something felt off — the kind of low-grade wrongness that experienced engineers learn to trust. Pods were being rescheduled far more often than they should be. Small hiccups here and there. Nothing was down, but nothing felt stable either.
The suspect: Karpenter, the autoscaler responsible for provisioning and removing nodes on the Project's Kubernetes cluster based on real-time demand.
The nickname the team gave the symptom said it all: node flapping.
Chapter 1 — Reading the Scene of the Crime
Before touching anything, the first move was to read the configuration as it stood — no assumptions, just the facts on paper.
Two files told two halves of the story:
- The Karpenter controller configuration — how the autoscaler itself was deployed.
- The NodePool / NodeClass configuration — the rules Karpenter uses to decide when a node should be created, and when it should be torn down.
Buried in the disruption rules was a setting that looked almost too simple to be suspicious:
consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m
Translated into plain English: "If a node looks underused, wait just one minute — then start tearing it down and repacking its pods somewhere else."
One minute. That's barely enough time for a freshly booted node to catch its breath, let alone reach a stable, representative level of utilization.
It was a lead. But a lead isn't proof. Time to go where the real evidence lives: the cluster itself.
THE FILE SAID... THE CLUSTER SAID... ┌───────────────────────────┐ ┌───────────────────────────┐ │ consolidationPolicy: │ │ $ kubectl get nodes │ │ WhenEmptyOrUnderutilized│ vs. │ │ │ consolidateAfter: 1m │ ───────► │ NAME AGE │ │ │ │ node-a 74d │ │ "looks harmless enough" │ │ node-b 10m │ │ │ │ node-c 3m │ │ │ │ node-d 50s <-- ??? │ └───────────────────────────┘ └───────────────────────────┘
Chapter 2 — Following the Trail Live
With read-only access to the cluster, the investigation moved from theory to observation.
First clue: node ages.
NAME AGE node-a 74d node-b 10m node-c 3m node-d 50s
Some nodes had been quietly running for months. Others were barely old enough to have a name. That's not a healthy autoscaling pattern — that's a revolving door.
Second clue: the autoscaler's own logs. And this is where the story turned from "suspicious" to "an open-and-shut case."
The log timeline read almost like a confession:
08:07— a brand-new node finishes booting and joins the cluster.08:08— a different, barely-4-minutes-old node gets deleted for being "underutilized."08:09— the autoscaler decides another node should be disrupted too — for a projected saving of $0.23 per hour.- Within seconds, a replacement node is requested, launched, and joined.
- Repeat. Every one to two minutes.
Third clue, and the one that made it personal: cluster events showed roughly a dozen pods per cycle — monitoring agents, backend workers, background processors — being evicted with the reason Underutilized, over and over, every couple of minutes. One workload in particular, a stateful database-like service backed by a persistent disk, kept failing to reschedule cleanly, because its storage volume was pinned to a specific availability zone and the churn kept sending it to nodes that didn't match.
The picture was now complete: the autoscaler wasn't reacting to real load. It was chasing its own tail — consolidating, evicting, rebuilding, and immediately reconsidering its own decision, for the sake of pocket change in savings.
┌─────────────────────────────┐
│ 1. New node boots up │
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
┌────►│ 2. Wait... one minute │
│ └──────────────┬──────────────┘
│ ▼
│ ┌─────────────────────────────┐
│ │ 3. "Underutilized!" │
│ │ node marked for removal │
│ └──────────────┬──────────────┘
│ ▼
│ ┌─────────────────────────────┐
│ │ 4. Pods evicted, │
│ │ node deleted │
│ └──────────────┬──────────────┘
│ ▼
│ ┌─────────────────────────────┐
└─────┤ 5. Replacement requested... │
│ back to step 1 │
└─────────────────────────────┘
Chapter 3 — The First "Fix" That Wasn't
Two changes were proposed to break the loop:
- Give nodes room to breathe — raise
consolidateAfterfrom 1 minute to 10 minutes, so Karpenter only acts on sustained underutilization, not a momentary dip. - Protect the sensitive workload — mark the stateful, zone-pinned service as "do not disturb," so the autoscaler would never pick its node as a candidate for removal.
The second change was applied — a small annotation added to the workload's configuration, based on how similar annotations were written elsewhere in the same repository.
It looked right. It reviewed cleanly. It even got merged.
And it did nothing.
The person who owns that workload caught it: the annotation had been placed one level too shallow in the configuration tree. It turns out that "looks consistent with the neighboring files" is not the same as "matches what the underlying Helm chart template actually reads." Two similar-looking workloads (a web service and a stateful database) used two different nested paths for the exact same setting — and only reading the chart's source template revealed which one applied here.
Lesson learned, on the record: when in doubt, read the template, not just the neighboring examples. The fix was corrected, verified against the actual chart source line by line, and shipped properly the second time.
X WRONG (silently ignored) OK RIGHT (chart reads this one) ┌────────────────────────────────────────────┐ ┌────────────────────────────────────────────┐ │ │ │ statefulset: │ │ podAnnotations: │ │ podAnnotations: │ │ karpenter.sh/do-not-disrupt: "true" │ │ karpenter.sh/do-not-disrupt: "true" │ │ │ │ │ │ # top level — the chart's StatefulSet │ │ # nested under `statefulset:` — exactly │ │ # template never reads this key │ │ # what the template.yaml expects │ └────────────────────────────────────────────┘ └────────────────────────────────────────────┘
Chapter 4 — Case Closed (With a Bonus Find)
With both fixes live, the cluster was watched for the next stretch of time. The results spoke for themselves:
- Node lifetimes stopped being measured in seconds and minutes — they stabilized.
- The wave of "Underutilized" pod evictions disappeared from the event stream.
- The stateful workload's node stopped being offered up as a sacrifice for fractional cost savings.
As a bonus, the investigation surfaced a second, unrelated but long-standing nuisance: one of the deployment pipelines was permanently flagging a stateful workload as "out of sync" — a cosmetic false alarm caused by Kubernetes always reporting an empty timestamp field that the source configuration never included. It's a well-known quirk, and a fix for it already existed elsewhere in the same project — it just hadn't been copied over to this pipeline yet. Ten minutes of comparison work closed that gap too.
Summary — Hints for Fellow Investigators
If you run workload autoscaling on Kubernetes (Karpenter or otherwise), a few hard-earned takeaways:
- A very short "consolidate after" window is a false economy. Saving a few cents per hour isn't worth destabilizing your workloads every couple of minutes. Give your autoscaler room to observe sustained underutilization before it acts.
- Stateful and zone-pinned workloads need explicit protection. If a pod's storage can't just move to any node, tell your autoscaler to leave its node alone — don't rely on the general disruption policy to "figure it out."
- Read live cluster state, not just the config. Configuration tells you intent. Logs and events tell you what's actually happening. The smoking gun in this story ($0.23/hour saved for cluster-wide chaos) only showed up in the logs.
- Don't trust "it looks like the file next to it." Copying a pattern from a similar-looking config is a great starting point, and a terrible final source of truth. When a setting doesn't seem to take effect, go straight to the template or documentation that actually consumes it.
- While you're in there, look around. Some of the best fixes aren't the one you went looking for — they're the ones you trip over on the way.
BEFORE: the revolving door AFTER: a quiet, stable cluster ┌───────────────────────────┐ ┌───────────────────────────┐ │ node-a ............ 50s │ │ node-a ............ 74d │ │ node-b ............ 3m │ ═════► │ node-b ............ 74d │ │ node-c ............ 10m │ │ node-c ............ 62d │ │ node-d ............ 2m │ │ node-d ............ 41d │ │ churn every 1-2 minutes │ │ boringly stable │ └───────────────────────────┘ └───────────────────────────┘