The idle metric that almost got a validator downsized
The setup
A monthly cloud bill for two environments of a blockchain-backed trading platform came in around $2,500 for compute alone. Not alarming, but enough to ask the obvious question: are we paying for capacity nobody's using? Two groups of machines stood out as candidates — a handful of general management nodes, and a larger group of beefier machines dedicated to running the platform's blockchain validator nodes. The validators were the ones worth real money: nine large instances per environment, each with eight virtual CPUs and 32GB of memory.
The instinct was to just look at CPU usage and decide from there.
The discovery
Thirty days of CPU metrics came back almost embarrassingly low. The management nodes idled along at a reasonable 10-20% with occasional bursts — fine, expected. The validator nodes, though, were barely breathing: 1-5% average CPU, spiking to a whole 13% under the heaviest load seen in a month. On paper, that's eight expensive virtual CPUs sitting almost completely idle, month after month, across eighteen machines.
Every instinct said: shrink them. Half the CPU, half the price, obvious win.
The turn
Before making that change, one more thing seemed worth checking: what does the actual workload running on these nodes ask for? The validator pods' own resource configuration told a different story than the CPU graph did. Each pod requested 2 CPUs but a full 8GB of memory, with headroom up to 20GB under load — a profile that reads like a workload that's memory-hungry, not CPU-hungry. A blockchain validator keeps a running cache of chain state in memory; that's exactly the kind of thing that would look idle on a CPU chart while still needing every gigabyte it was given.
Downsizing on the strength of the CPU number alone would have been optimizing for the wrong resource — and if it turned out wrong, the failure mode wasn't "the app runs a bit slower." It was a production validator falling behind consensus, or getting killed outright for running out of memory. That's a much worse Tuesday than an oversized invoice.
So: get real memory numbers before touching anything. That should have been the easy part.
The locked door
The cluster's own dashboard tooling — already deployed, already collecting exactly the metrics needed — seemed like the fastest path. But the read-only credentials in hand hit an unexpected wall trying to reach it directly through the cluster's own command-line tooling: the request reached the cluster fine, and the identity behind it was valid, but the response came back as a flat "you must be logged in."
That phrasing is its own clue. A network problem looks like a timeout. A truly invalid identity gets rejected before it ever reaches the API. "You must be logged in," from a request that visibly made it all the way to the server, means something narrower and easier to fix conceptually, if not in the moment: the cluster's access list simply had no entry mapping this particular identity to any permission at all. Two administrative identities were on the list. A read-only one, added later for exactly this kind of investigation, wasn't.
Rather than widen cluster access just to pull one metric — a bigger, harder-to-reverse change than the question warranted — the better move was routing around it entirely. The cluster already ran a metrics dashboard with its own login, reachable over the web, no cluster-level access needed at all.
The second wall
Getting an AI coding assistant wired up to query that dashboard directly meant a small piece of new tooling: a locally-run connector process, authenticated with a scoped, read-only access token, talking to the assistant over a standard plugin protocol.
It connected. And then it didn't do anything.
Every request into it just hung until it timed out, with no error message on either side to explain why. The token was confirmed valid — a plain web request to the dashboard's health endpoint using that same token came back healthy immediately. The network path was fine. The connector process itself was clearly running; it just wasn't talking back.
The connector's own built-in help text stated, in black and white, that it defaults to the exact communication mode being used to talk to it. That should have been the end of the investigation — except the actual behavior didn't match the documented default at all. Bypassing the assistant entirely and feeding the connector a raw handshake message by hand, then watching its own internal log output, surfaced the real story: it was quietly starting up in a completely different mode — an always-on network server — rather than the request-response mode its own --help text claimed was the default. Nothing about that mismatch was visible unless you went looking at the process's own logs directly; from the outside, both modes look identical — a running process that accepts a connection and says nothing back.
The fix was a single explicit flag, forcing the correct mode instead of trusting the tool's stated default. Once that flag was in place, the same handshake that had produced silence a minute earlier came back instantly, cleanly, fully formed.
The resolution
Two smaller findings came along for the ride while getting there: the authentication method the connector's setup guide suggested was already flagged internally as outdated in favor of a newer one — easy to miss if you follow the first example you find rather than checking for a deprecation note. And registering the finished connector with the coding assistant landed it, twice, under the wrong project entirely — the registration command scopes itself to whatever folder a terminal happens to be sitting in at the moment it runs, not to the project the assistant is actually working in. Both times, the fix was checking the tool's own status command rather than assuming the first "success" message meant it had gone where intended.
By the end of the session, the connector was live, correctly registered, and returning real data. The actual verdict on whether those validator nodes can shrink still hasn't been reached — that now depends on pulling real memory numbers through the newly working connector, not on the CPU chart that looked so persuasive at the very start.
Hints for the next investigation
- A resource that's idle on one dimension may not be idle at all. CPU utilization and memory utilization tell completely different stories for the same workload; check the metric the workload actually needs, not the one that's easiest to pull first.
- "Reaches the server but gets rejected" and "never reaches the server" are different failures with different fixes. A clean rejection message, even a confusing one, is often proof the hard part — connectivity, identity — already works, and the real gap is narrower than it first appears.
- Prefer the smaller, more reversible fix. Widening a production system's access list to answer one question was available, but routing through an existing, already-scoped tool avoided a change that would have outlived the question that prompted it.
- A tool's documented default and its actual default can quietly disagree, especially for anything distributed as a container image with its own baked-in startup behavior. When a documented default doesn't match observed behavior, trust the observed behavior, and verify by talking to the tool directly, bypassing whatever's wrapping it, before assuming the fix is somewhere else entirely.
- Don't let cost pressure shortcut a safety-relevant decision. The cheapest fast answer — "CPU usage is low, just shrink it" — was directionally wrong for the workload in question. Getting to the right answer took longer, but the alternative was a plausible-sounding number driving a change to infrastructure where being wrong is expensive in a completely different way than money.