Field Notes
Real incidents and investigations from cloud, AI and security work, written up as short detective stories. Every one is anonymized: names, systems and identifying details are generalized.
A Terraform apply hit a 409 on a resource its state had never seen — the twist was that the fix already existed, in a different environment's git history.
A build-once-promote-everywhere release pipeline hides two mysteries: a CI step that dies with zero output, and a production deploy role that won't assume.
An ArgoCD ingress with no address led to a self-signed cert with no IP SAN, then to a shared IngressClass object deleted by an unrelated GitOps prune, during a live ingress-controller migration.
A Kubernetes autoscaler kept tearing down and rebuilding nodes every 1-2 minutes over pocket-change savings — a "detective story" account of diagnosing and fixing it.
A Python dependency source switch pointed internal packages at an artifact-curation proxy that only ever mirrored public PyPI — with a near-miss force-push and a 404-vs-403 mixup along the way.
A financial trading platform's disaster-recovery audit finds its recovery-time targets were already met natively — the real gaps were retention depth, tamper-proof records, cross-account ambiguity, and an irreversible setting nobody should flip yet.
A commit message stating a financial platform's production database was already destroyed in AWS turns out to be entirely true and entirely benign — confirmed only by checking the live infrastructure directly, not by re-reading the same alarming sentence.
A near-idle CPU graph almost justifies shrinking a blockchain platform's validator nodes, until the workload's own memory profile raises doubt — and getting real numbers instead means routing around a locked-down cluster and a monitoring connector silently starting in the wrong mode.
A targeted Terraform apply quietly skipped the NAT gateway, walling a brand-new Kubernetes cluster in until its nodes timed out fifteen minutes later — and then a saved plan file turned out to be carrying the wrong identity's credentials.
A pod sat Pending for six hours because GPU time-slicing made one graphics card count as four against a node-pool limit of one — and underneath it, a neighbouring workload had been declaring a third of the CPU it actually used.
A search cluster scheduled for deletion showed more daily traffic than its live replacement — until the same data, read hourly, showed the two had swapped day-and-night shapes at the cutover hour and the leftover flat line was the service talking to itself.
A monitor built to catch backups that silently stop was red the morning after deployment — the backups were fine, the metric label pointed the opposite way to the obvious guess, and correcting it dissolved a limitation that had already been written down and accepted.
A second high-speed interconnect looked obviously necessary for an LLM split across a pair of NVIDIA DGX Sparks — until the first measurement read zero, the second read impossibly high, and the honest numbers showed 13% peak utilisation.
Container tooling promised 180 GB of reclaimable disk and delivered about one, because layered storage reports logical sizes — the real consumer was found by two filesystem commands, plus a near-miss deletion that a hard name guard stopped.
Freeing up one of two machines looked like it cost 4.4× the speed, until running the same model at both node counts split that into 1.65× hardware and 2.7× model — and a flat throughput line across five very different prompts revealed why.
A request to make an S3 bucket public turned out to need a CDN edge (Origin Access Control) with the bucket kept private — then a 403 traced to a module default, a wildcard cert in the wrong region, and a 'DNS doesn't resolve' that was only a stale VPN cache.
An LLM died for fifteen hours while its containers cheerfully reported "Up 46 hours" — a vendor's OOM policy had named it the preferred victim, and every restart then failed because of a setting about logging out.
In a CI-to-cloud migration, a "read-only" plan role couldn't actually plan — it couldn't read secret values, a single encrypted secret wouldn't decrypt, the cluster didn't recognize the role, and a stale re-run faked a green build.
A "do not disrupt" marker protected a live media pod 5,585 times, then didn't — because the 48-hour grace period added to be kind to that exact workload is what let node drift ignore it. The cluster announced the execution two days early and alerted nobody.
An analytics database crash-looped on an XML parse error while reading a YAML file — and the culprit was an unprintable terminal-bell byte hiding in a password that every tool in the chain displayed as perfectly correct.
One app out of twenty stopped reconciling because the cluster grew a status field the deploy tool had never heard of — then the "conservative" version choice turned out to be the unsupported one, and the upgrade lit up a CRD invalid since 2021 and a batch of jobs that hadn't run in ten months.
A validator operator's wallet disagreed with the address a setup script told them to register. The "address" was the first 40 characters of the public key, cut out of a log line by an unanchored grep, and a client who had already staked successfully had quietly routed around it.
Every test passed, then the hand-off to the GitOps repository failed with "401 Bad credentials". A personal access token had died during an eleven-month quiet spell, and the staging and production workflows depended on it too.