Field notes from production
A Terraform apply hit a 409 on a resource its state had never seen — the twist was that the fix already existed, in a different environment's git history.
A build-once-promote-everywhere release pipeline hides two mysteries: a CI step that dies with zero output, and a production deploy role that won't assume.
An ArgoCD ingress with no address led to a self-signed cert with no IP SAN, then to a shared IngressClass object deleted by an unrelated GitOps prune, during a live ingress-controller migration.
A Kubernetes autoscaler kept tearing down and rebuilding nodes every 1-2 minutes over pocket-change savings — a "detective story" account of diagnosing and fixing it.
A Python dependency source switch pointed internal packages at an artifact-curation proxy that only ever mirrored public PyPI — with a near-miss force-push and a 404-vs-403 mixup along the way.
A financial trading platform's disaster-recovery audit finds its recovery-time targets were already met natively — the real gaps were retention depth, tamper-proof records, cross-account ambiguity, and an irreversible setting nobody should flip yet.
A commit message stating a financial platform's production database was already destroyed in AWS turns out to be entirely true and entirely benign — confirmed only by checking the live infrastructure directly, not by re-reading the same alarming sentence.
A near-idle CPU graph almost justifies shrinking a blockchain platform's validator nodes, until the workload's own memory profile raises doubt — and getting real numbers instead means routing around a locked-down cluster and a monitoring connector silently starting in the wrong mode.
A targeted Terraform apply quietly skipped the NAT gateway, walling a brand-new Kubernetes cluster in until its nodes timed out fifteen minutes later — and then a saved plan file turned out to be carrying the wrong identity's credentials.
A pod sat Pending for six hours because GPU time-slicing made one graphics card count as four against a node-pool limit of one — and underneath it, a neighbouring workload had been declaring a third of the CPU it actually used.
A search cluster scheduled for deletion showed more daily traffic than its live replacement — until the same data, read hourly, showed the two had swapped day-and-night shapes at the cutover hour and the leftover flat line was the service talking to itself.
A monitor built to catch backups that silently stop was red the morning after deployment — the backups were fine, the metric label pointed the opposite way to the obvious guess, and correcting it dissolved a limitation that had already been written down and accepted.
A second high-speed interconnect looked obviously necessary for an LLM split across a pair of NVIDIA DGX Sparks — until the first measurement read zero, the second read impossibly high, and the honest numbers showed 13% peak utilisation.
Container tooling promised 180 GB of reclaimable disk and delivered about one, because layered storage reports logical sizes — the real consumer was found by two filesystem commands, plus a near-miss deletion that a hard name guard stopped.
Freeing up one of two machines looked like it cost 4.4× the speed, until running the same model at both node counts split that into 1.65× hardware and 2.7× model — and a flat throughput line across five very different prompts revealed why.
A request to make an S3 bucket public turned out to need a CDN edge (Origin Access Control) with the bucket kept private — then a 403 traced to a module default, a wildcard cert in the wrong region, and a 'DNS doesn't resolve' that was only a stale VPN cache.
An LLM died for fifteen hours while its containers cheerfully reported "Up 46 hours" — a vendor's OOM policy had named it the preferred victim, and every restart then failed because of a setting about logging out.
In a CI-to-cloud migration, a "read-only" plan role couldn't actually plan — it couldn't read secret values, a single encrypted secret wouldn't decrypt, the cluster didn't recognize the role, and a stale re-run faked a green build.
A "do not disrupt" marker protected a live media pod 5,585 times, then didn't — because the 48-hour grace period added to be kind to that exact workload is what let node drift ignore it. The cluster announced the execution two days early and alerted nobody.