The Case of the Cluster That Refused to Grow
A short detective story about one GPU, four ghosts, and a number nobody had checked in months.
┌────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├────────────────────────────────────────────────────────────┤ │ │ │ [ one pod, Pending, 6 hours ] │ │ │ │ │ ┌─────────┴─────────┐ │ │ ▼ ▼ │ │ ┌───────────┐ ┌───────────┐ │ │ │ SUSPECT │ │ SUSPECT │ │ │ │ "the flag │ │ "AWS is │ │ │ │ someone │ │ out of │ │ │ │ grepped" │ │ GPUs" │ │ │ └─────┬─────┘ └─────┬─────┘ │ │ │ innocent │ innocent │ │ ▼ ▼ │ │ ┌───────────────────────┐ │ │ │ ACTUAL CULPRIT │ │ │ │ limit: 1 GPU │ │ │ │ usage: 4 GPUs │ │ │ │ physical GPUs: 1 │ │ │ └───────────────────────┘ │ └────────────────────────────────────────────────────────────┘
The Complaint
A monitor went critical on the Company's development Kubernetes cluster:
CRITICAL: the autoscaler has reported unschedulable pods continuously for 15 minutes. The autoscaler sees pods it cannot place and is not resolving it by adding nodes.
Fifteen minutes was the alert threshold. The actual number, once someone looked, was six hours. One pod. Pending. Not crash-looping, not erroring — just sitting there, patiently, while an autoscaler whose entire job is to add capacity did nothing at all.
The alert helpfully listed four suspects in order of likelihood, ending with the reassuring note that only the last one is an actual AWS problem. The engineer who picked up the ticket had already run a grep across the config repo and found a candidate of their own:
$ grep -rs "do-not-disrupt" . ./utils/<a media service>/values.yaml: karpenter.sh/do-not-disrupt: "true" ./apps/<a worker service>/values.yaml: karpenter.sh/do-not-disrupt: "true" ./apps/<a backend service>/values.yaml: karpenter.sh/do-not-disrupt: "true"
Three workloads marked "do not disturb." A stuck cluster. It's a tidy theory, and it's the kind of theory that feels right at 9am.
It was wrong. But hold that thought — it was wrong in an interesting way.
Chapter 1 — Ask the Suspect Directly
Before theorising, there was a much cheaper move available: the autoscaler writes logs, and it had been writing the same one every two seconds for six hours.
It wasn't a hint. It was a signed confession:
could not schedule pod NodePool=gpu
error: no instance type has enough resources, ... nodepool In [default],
instance-type In [<general-purpose types>], resources={"cpu":"1600m",
"memory":"5020Mi","nvidia.com/gpu":"1", ...};
all available instance types exceed limits for nodepool (NodePool=gpu)
Two separate rejections in one line, which is why it took a moment to read properly.
The general-purpose node pool said: I can't help, none of my machine types have a GPU. Fair enough.
The GPU node pool said something much stranger: all available instance types exceed limits. Translated: I am already over my own resource budget. I will not launch anything. Ever.
That killed the AWS theory outright — the metric for cloud provider launch failures had no data points at all. Amazon was never asked for a machine. It also killed the do-not-disrupt theory, because that flag governs whether the autoscaler may remove an existing node. It has no vote whatsoever on whether a new one gets created.
So the question sharpened nicely: why does a node pool with exactly one node think it's over budget?
Chapter 2 — One GPU, Four Ghosts
The pool's configuration was short enough to read in one breath:
limits: cpu: "8" memory: 32Gi nvidia.com/gpu: "1"
And the pool's live status reported its usage as:
nodes: 1 cpu: 4 memory: ~15Gi nvidia.com/gpu: 4 ← ?
One node. One physical graphics card. Four GPUs of measured usage, against a limit of one.
The explanation was sitting two directories away, in the GPU device driver's configuration:
sharing:
timeSlicing:
resources:
- name: nvidia.com/gpu
replicas: 4
Time-slicing. A perfectly sensible technique: one physical GPU is advertised to Kubernetes as four logical ones, so four modest workloads can take turns on it instead of each demanding a whole card. The team had enabled it deliberately, and it was working exactly as designed.
Here's the trap. The autoscaler doesn't count physical hardware. It counts what nodes advertise. The moment time-slicing was switched on, that one node started declaring four GPUs — and the pool's limit of one went from "a cap allowing a single node" to "a wall permanently four times breached."
WHAT THE HUMANS MEANT WHAT THE AUTOSCALER MEASURED ┌───────────────────────────┐ ┌───────────────────────────┐ │ limit: 1 GPU │ │ limit: 1 GPU │ │ = "allow one GPU node" │ vs. │ usage: 4 GPUs │ │ │ │ = "400% over budget, │ │ [ gpu-node-1 ] ✓ │ │ refuse everything" │ └───────────────────────────┘ └───────────────────────────┘
And the sting: this was not an accident. A comment in the repo explained it plainly — the cap was implemented using this over-limit behaviour, on purpose, to hold the expensive GPU fleet at exactly one machine. Someone had understood the mechanism perfectly and built a cage out of it.
They'd just built a cage with no door. The pool could never grow — not for a legitimate need, not in an emergency, not ever — and nothing in the system distinguished "capped on purpose" from "genuinely out of capacity." Every capacity dashboard for that pool had been reading 400% exhausted for weeks, which meant it had quietly stopped being a signal at all.
Chapter 3 — The Room That Wasn't There
That explained why no new node appeared. It didn't explain the more basic question: why did this pod need one? There was a GPU node already running, with two of its four GPU slots free.
The node was a 4-CPU machine with roughly 3.92 usable cores. On paper:
digital simulation renderer ... 1500m (declared) worker replica #1 ... 1000m system daemons ... 500m ──────────────────────────────────────── committed 3000m free 920m ← "plenty of room"
The pending pod wanted 1000m. It missed by eighty millicores. Eighty. A rounding error away from a six-hour outage.
But the interesting number wasn't 920. It was what the renderer was actually doing. It had declared 1500m. Measured over 24 hours, it was consuming a flat, unwavering 3.5 cores.
The node wasn't at 77% committed. It was at 92% actually used. The scheduler had been doing careful arithmetic on numbers that bore no relationship to reality, and it had packed a workload onto a machine with essentially nothing left.
Nobody had lied. Requests in Kubernetes are a declaration — a promise about future behaviour — and absolutely nothing in the system ever goes back and checks whether the promise was kept.
Chapter 4 — Two Wrong Turns and a Correction
Three things nearly sent this investigation into a ditch. All three are worth confessing.
Wrong turn one: the memory leak that wasn't. Graphing the renderer's memory produced a shape every engineer recognises instantly — a clean linear climb from 4.4 GB to 5.8 GB over fifteen hours, then a cliff. Textbook leak, textbook restart. Except the pod had zero restarts, and two other metrics were flat as a table: the working set moved 80 MB in those fifteen hours, and RSS barely moved at all. The climb was page cache, and the cliff was the kernel reclaiming it under pressure. The metric that looked like a smoking gun simply counts something most people don't expect it to.
Wrong turn two: the autoscaler that appeared to be insane. The observability platform's summary view reported the workload's horizontal autoscaler as sitting at memory: 29781770240% against a 75% target. Twenty-nine billion percent. That reads as a catastrophically broken component and it was very tempting to stop there and file a bug. Reading the raw object instead showed memory at 69% against a 75% target, scaling normally, entirely healthy. The summary was rendering a byte count as a percentage. The CPU figure on the very same line was perfectly sensible, which is exactly what made the memory one convincing.
The correction: arithmetic is not measurement. The proposed fix was to shrink the pending workload's CPU request so it would fit. The first number — 700m — came straight from arithmetic: make it fit, leave a little slack. Then someone asked the obvious question nobody had asked yet: what does this pod actually use?
Nine millicores. Idle, sustained. Peaking around 300m at startup.
It had been reserving 1000m — over a hundred times its actual appetite — on the single most contended machine in the cluster. The arithmetic answer of 700m wasn't wrong, it was just needlessly tight for no benefit. The measured answer, 500m, was equally safe and handed 400m back to the node.
Chapter 5 — The Innocent Suspect Was Guilty of Something Else
Which brings us back to that grep.
The do-not-disrupt flag had nothing to do with the outage. But while checking it off the list, someone compared the flag in the config file against the annotations on the live pod.
It wasn't there.
The chart offered two similar-looking places to put annotations: one that attaches them to the deployment object, and one that attaches them to the pods it creates. The flag had been sitting in the first. It rendered correctly. It showed up in the deployment. It reviewed cleanly, presumably more than once.
And the autoscaler never saw it, because the autoscaler only reads that flag on pods. For the entire life of that service, a workload everyone believed was protected had been fully disruptible, and nothing had ever said a word about it.
X DECORATIVE OK LOAD-BEARING ┌──────────────────────────────────┐ ┌──────────────────────────────────┐ │ deployment: │ │ deployment: │ │ annotations: │ │ podAnnotations: │ │ do-not-disrupt: "true" │ │ do-not-disrupt: "true" │ │ │ │ │ │ # lands on the Deployment. │ │ # lands on the Pod, which is │ │ # The autoscaler never looks │ │ # the only place the autoscaler │ │ # there. No error. No warning. │ │ # actually reads it. │ └──────────────────────────────────┘ └──────────────────────────────────┘
Two sibling services in the same repo had it in the right place. This one didn't. Both configurations are valid; both render; neither errors. The only way to know which one you have is to look at the running pod.
Chapter 6 — Case Closed, Sort Of
The fix that shipped was small: drop the worker's CPU reservation from 1000m to 500m so both replicas genuinely fit on the existing node, and move the annotation to where something reads it. Both replicas came up. The pod that had been Pending for six hours started in seconds.
It would be satisfying to end there. But the honest version has a sting in the tail, and the sting is the most useful part of the whole story.
The real defect is still there. The renderer declares 1500m and uses 3500m. That gap is what allowed this to happen and it will allow it to happen again.
And it cannot be fixed on its own. Write the honest number, and the budget becomes 3500 + 500×2 + 500 against 3920 available — the workloads that were just successfully placed become unschedulable, and the exact same alert fires again. Correcting an under-declared resource request on a saturated machine doesn't fix an outage; it relocates it. It isn't a config edit at all. It's a decision about whether to buy another machine.
There was one last thing worth checking before blaming anyone: was that constant 3.5-core burn even legitimate? It looked deeply suspicious — dead flat through the night, at 3am, with no users anywhere near the development environment. Classic stuck-process behaviour.
It wasn't. Reading the container's actual startup command revealed a 3D game engine running with a scene loaded, alongside a resident asset-processing service. A game engine's main loop renders continuously whether or not a single human is watching. There is no idle state to fall back to. The workload was behaving exactly as designed — it had simply never told the scheduler the truth about what that design costs.
Summary — Hints for Fellow Investigators
- Ask the component directly before working through the runbook. The autoscaler had been logging its exact reason, twice a second, for six hours. The runbook's four ranked hypotheses were a slower path to the same answer — and its top-ranked suspect happened to be right, which would have been pure luck if the log hadn't confirmed it.
- Anything that inflates a resource count inflates the number your autoscaler enforces limits against. GPU time-slicing, virtual CPUs, over-provisioned extended resources — the scheduler counts what the node advertises, not what's physically installed. Check your limits after enabling any of them.
- If you cap something by making it permanently over-limit, you've also blinded the monitor that watches it. "Capped on purpose" and "genuinely exhausted" become indistinguishable. Cap with a mechanism that still lets you tell the difference.
- Resource requests are a promise nobody audits. A pod requesting a third of what it uses corrupts every scheduling decision on its node, silently, forever. When something won't fit on a node that "has room," compare declared against actual before anything else.
- You cannot correct one dishonest request in isolation on a full machine. An honest number evicts the neighbours. Measure, decide whether to buy capacity or reduce consumption, then write the number.
- A metric that looks like a leak might be counting page cache. Confirm against working-set and RSS before you go hunting for the leak. And when a monitoring dashboard shows you a number like twenty-nine billion percent, suspect the dashboard before you suspect the system.
- Measure before you size. The first fix came from arithmetic and was defensible. The measured answer was better, and took ninety seconds to obtain.
- For anything a controller consumes, verify on the live object. Not the config file, not the rendered template, not the parent resource. A valid setting in the wrong place renders perfectly, reviews cleanly, and does absolutely nothing.
BEFORE AFTER
┌────────────────────────────┐ ┌────────────────────────────┐
│ renderer .......... 1500m │ │ renderer .......... 1500m │
│ worker #1 ......... 1000m │ │ worker #1 ......... 500m │
│ worker #2 ...... PENDING │ ═══════► │ worker #2 ......... 500m │
│ daemons ........... 500m │ │ daemons ........... 500m │
│ │ │ │
│ pool can add a node? NO │ │ everything fits. │
│ (limit 1, usage 4) │ │ pool still can't grow. │
└────────────────────────────┘ └────────────────────────────┘
the real fix is still open ──────────┘