Two ghosts in a release pipeline
A short detective story from the world of Infrastructure as Code.
The setup
The Project's release pipeline followed a pattern worth having in the first place: build a Docker image exactly once, tag it with the immutable git commit it came from, and never rebuild it for later environments. Staging and production don't get fresh builds — they get the same image, redeployed via a registry-side retag. That guarantees production is byte-for-byte whatever was actually validated earlier, not a "should be identical" rebuild that quietly drifted because a base image tag moved underneath it.
Getting there took a couple of false starts. First naive version: only build when app-relevant paths changed, to avoid burning CI minutes on doc-only commits. That broke the moment it met an automated release tool — the tool's own merge commits (bumping a changelog and version file) never touch application code, so the exact commit every release pointed at never got an image built at all, and every promotion check failed closed. The next-naive fix — just build on every push — solved that but reopened the original problem: now those same changelog-only commits did get rebuilt, and if anything in the build isn't perfectly deterministic, "identical to what was validated" is no longer actually true.
The real fix: diff each new commit against its parent for the paths that matter. Unchanged → retag the parent's exact image forward via a plain registry API call, no rebuild, no drift possible. Changed → real build. Simple in hindsight. It's also where both of the following ghosts live.
┌────────────────────────────────┐
│ dev — automatic │
│ build, or retag forward │
└────────────────┬───────────────┘
│ same sha
▼
┌────────────────────────────────┐
│ testnet — automatic │
│ on release, retag (no rebuild) │
└────────────────┬───────────────┘
│ same sha
▼
┌────────────────────────────────┐
│ mainnet — manual │
│ human-gated, explicit dispatch │
└────────────────────────────────┘
Ghost #1: the step that failed with nothing to say
The retag step had one cosmetic wrinkle. The registry API throws an "already exists" error if you retag a tag onto a digest it already points at — harmless, but it printed an alarming-looking [ERROR] line before the pipeline's own "this is fine" message, so a green step still looked scary in the logs. The fix seemed straightforward: check the target tag's current digest first, skip the retag call entirely if it already matches.
existing_digest=$(aws ecr describe-images --image-ids imageTag="$tag" \ --query 'imageDetails[0].imageDigest' 2>/dev/null)
Shipped, reviewed, merged. And then, on a completely unrelated push a bit later, the retag step died. Not with an error — with nothing. No log line, no message, just a red step and silence.
The investigation started where it should: assume nothing, check the facts.
First move: reproduce the exact API calls locally against the real registry, using the same short-lived credentials the pipeline itself would use, to separate "the AWS CLI is behaving oddly" from "the script logic is wrong." That confirmed a real non-zero exit code with zero stdout or stderr once output was redirected — consistent with the pipeline's own symptom, but not yet an explanation.
Second move: pull the CI role's actual IAM policy and check it line by line. It already had every permission the step needed. That ruled out the tempting, boring explanation — "we're missing a permission" — and forced a harder look at the script itself.
The actual bug was hiding in the "fix" from the previous incident: 2>/dev/null on a command substitution only throws away the message — it does nothing to the exit status. The assignment still carries the failing command's non-zero return code. Under the shell mode GitHub Actions uses by default for run steps (fail immediately on any non-zero exit), that status still killed the entire script — just silently, because the one thing that would have explained why had already been piped into the void a moment earlier.
And it hit hardest in exactly the case the precheck was supposed to handle gracefully: the very first time a brand-new tag is checked, before it exists. describe-images correctly reports "not found" — which is a non-zero exit dressed up as an expected outcome. The precheck's own error handling was the thing making that expected case fatal.
Fix, once found: swallow the exit status explicitly, not just the message —
existing_digest=$(aws ecr describe-images ... 2>/dev/null) || true
One extra || true. The lesson underneath it is bigger than one line: hiding a command's message is not the same as handling its failure. If a non-zero exit is genuinely expected in some case, say so explicitly — don't just make it quiet.
Ghost #2: the role that wouldn't assume
Production promotion in this pipeline is deliberately not automatic. Staging promotes the instant a release is cut — merging the release tool's own PR is the promotion action. Production requires a human to explicitly trigger it, naming the exact validated artifact.
That's not an accident: an event-driven production trigger, combined with a required-approval gate, queues one pending run per release. If two releases go out in quick succession and someone approves the older, now-stale pending run, production silently rolls backward onto an older release. A manual, explicitly-targeted trigger sidesteps that entirely — there's only ever one run, and it points at whatever a human just chose.
Which made it more surprising when the manual production run failed at the credentials step: "not authorized to assume role via web identity" — the cloud-side rejection you get when the federated trust relationship doesn't accept the token you're presenting.
First suspect: a known trust-policy footgun already fixed elsewhere in the same pipeline, where two different flavors of subject-matching condition (an exact match and a wildcard match) had ended up in the same policy statement. Identity systems AND every condition inside one statement, even two testing the same field — so a token would've needed to satisfy an exact string and a wildcard pattern simultaneously, which is impossible. That bug was real, but it lived in a different job. The fix for it didn't touch this one.
Second attempt: add the subject shape you'd naturally expect for a manually-triggered run on the main branch. Redeploy. Still fails. At this point it's tempting to suspect a stale deploy — did the trust-policy change actually propagate? So the next move was pulling the live trust policy straight off the cloud side, not from the config repo, to rule that out directly. The new entry was there, live, confirmed. And it still failed. That's the moment to stop suspecting propagation delay and start suspecting that the subject shape itself was simply wrong.
It was. The job in question pins a named deployment environment — the exact mechanism that gates it behind human approval in the first place. Turns out that setting alone changes what identity token gets issued entirely: once a job specifies an environment, the token's subject becomes "this repo, this environment," full stop — regardless of what triggered the run, replacing the branch- or tag-based subject you'd otherwise expect. The trust policy had been told to expect a branch. GitHub was never going to send one.
EXPECTED subject ACTUAL subject GitHub sent
┌─────────────────────────┐ ┌──────────────────────────────┐
│ repo:org/app:ref: │ vs. │ repo:org/app:environment: │
│ refs/heads/main │ │ production │
└─────────────────────────┘ └──────────────────────────────┘
(what the fix (what actually got
trusted) presented — always,
once `environment:`
is set on the job)
Fix: trust the environment-shaped subject instead of the branch-shaped one. Once that landed, the manual production promotion assumed its role on the first try.
Hints for the next investigation
- Silencing a message is not the same as handling a failure. If you redirect stderr to hide an expected error, the exit status is still live ammunition under a fail-fast shell — swallow it on purpose, don't just make it quiet.
- Rule causes out with evidence, not assumption. Reproducing the exact failing calls locally, and pulling the live configuration from the platform itself rather than trusting the source repo, are both cheap ways to eliminate a wrong theory before you've sunk an hour into it.
- A fix for one job doesn't imply the same fix for a sibling job. Two jobs hitting the identical error message can have entirely different root causes — confirm each one lands, don't assume the first fix covers the pattern.
- Orthogonal-looking job configuration can silently change your identity token. A setting whose whole purpose is "gate this behind approval" quietly overrides what subject claim gets issued, independent of whatever triggered the run. When a trust-policy fix doesn't fix an auth failure, check what else about the job's configuration might be shaping the token before adding more policy entries.
- For your highest-stakes environment, prefer a human naming the exact artifact over an event automatically queueing one. It's a small amount of manual friction in exchange for guaranteeing there's never a stale, about-to-be-approved deployment sitting in the queue.