The Case of the Key That Was Active and Still Rejected
A short detective story about a release that failed with "invalid token" while the console swore the key was healthy. Both were telling the truth. They were talking about two different keys, in two different AWS accounts.
┌──────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├──────────────────────────────────────────────────────────────┤ │ │ │ VICTIM: a release pipeline, stopped at the image push │ │ WEAPON: "The security token included in the request │ │ is invalid." │ │ │ │ SUSPECT 1: a revoked key ........... ALIBI: active, │ │ used 8 minutes ago │ │ SUSPECT 2: a stray session token ... no evidence │ │ SUSPECT 3: someone edited the var .. no evidence │ │ │ │ CULPRIT: the right key name, the wrong account, │ │ and a cleanup that warned us in a comment │ └──────────────────────────────────────────────────────────────┘
The Setup
The Project is an SDK. Every release does two things: it publishes a Python package, and it publishes a multi-architecture base container image that every downstream service builds on. Once the image is out, bots open upgrade merge requests in the repositories that depend on it.
On release day, the package went out. The image didn't. The pipeline stopped at the very first AWS call of the publish job:
$ aws ecr get-login-password --region "$AWS_DEFAULT_REGION" An error occurred (UnrecognizedClientException) when calling the GetAuthorizationToken operation: The security token included in the request is invalid.
It failed three times in a row. No image, so no upgrade MRs, so the release was half-shipped.
The Obvious Answer
First check: did the job change? No. The same script had published the previous release a week earlier, line for line. The only CI commit in between touched an unrelated job.
So the script was fine and the credentials weren't. The job logged in with a pair of long-lived access keys stored as group-level CI variables. UnrecognizedClientException means AWS doesn't recognise the key ID at all. A wrong secret for a real key gives a different error, SignatureDoesNotMatch. So the key had probably been deleted or deactivated.
The conclusion wrote itself: rotate the key, update the variable, retry. Case closed.
Then a screenshot of the IAM console came back.
Twist One: The Alibi
IAM users ┌────────────────────────────────┬──────────────┬───────────────────┐ │ user │ access key │ key last used │ ├────────────────────────────────┼──────────────┼───────────────────┤ │ ... │ Active │ ... │ │ registry-push-user │ Active │ 8 minutes ago ✓ │ └────────────────────────────────┴──────────────┴───────────────────┘
The write user for the image registry had exactly one key. It was Active, and it had been used successfully eight minutes earlier. Something, somewhere, was pushing with it right now.
So AWS hadn't revoked the key. That pushed the investigation in a new direction. If the key works, the job must be sending something else.
Two candidates:
- A stray session token. The job's script had a fallback:
AWS_SESSION_TOKEN="${REGISTRY_SESSION_TOKEN:-${AWS_SESSION_TOKEN:-}}". If the runner or a higher-level variable injected a session token, AWS rejects even a valid static key with exactly this message. We checked the runner: the same one had published fine a week earlier. We checked the group's audit log for variable edits: nothing. - A different key ID in the variable than the one in the console.
The second check is the dull one. You open the CI variable, reveal it, and compare it with the key in the secrets store. Nobody wants to do it, because it feels like the console already answered the question.
It hadn't. The two IDs were different.
Twist Two: The Wrong Account
An AWS access key ID isn't random. The characters after the AKIA prefix encode the 12-digit account the key belongs to, and you can decode it offline with no credentials:
import base64
def account_of(key_id):
b = base64.b32decode(key_id[4:]) # all 16 chars after "AKIA"
n = int.from_bytes(b[:6], "big")
return "%012d" % ((n & 0x7fffffffff80) >> 7)
(Decode the whole remainder. On the first attempt we sliced it too short and got 10-digit nonsense. A result that isn't 12 digits is the tell.)
the variable named "REGISTRY_ACCESS_KEY_ID"
│
├── key the job sends ........ account 111111111111 (development)
│
└── key in the console ....... account 222222222222 (image registry)
registry the job pushes to ... account 222222222222
(account numbers fabricated)
The variable named after the registry account held a key from the development account. That worked for as long as the development key was active and allowed to push across accounts. The console showed a perfectly healthy key, just not the one the job was using.
Our second conclusion was "someone pasted the wrong key into the variable". It was wrong too.
Twist Three: It Was Announced
The development account's IAM is managed in Terraform. Its history had a commit from a week earlier that deactivated the static CI keys of legacy users. It was part of a planned move from long-lived keys to short-lived OIDC credentials for CI.
The commit set one flag to Inactive, and a comment next to it listed the SDK image push among the key's remaining users.
So nobody broke anything by accident. The infra team switched the key off on purpose, and the change itself said the SDK release still depended on it.
Why did nobody notice for a week? The publish job only runs on release tags. No tag, no job, no failure. The breakage sat there until release day.
We scanned the whole group for the variable name. Another project had hit the same error two days earlier, quietly switched to OIDC, and moved on. Four more projects still used the variable and hadn't run an affected job since. They would have broken on their next release.
The Fix, in Two Halves
Unblock today, properly. The tempting move was to click "Activate" in the console. But the key was managed by Terraform, so the next apply would switch it off again. A manual click also quietly undoes someone else's security work. Instead, a one-line merge request set the flag back to Active in the development environment, with a comment marking it temporary. It went through the infra pipeline and got a plan review like any other change. Once it was applied, we retried the job, the image was published, and the upgrade MRs opened.
Fix it for good. The registry account already had an IAM role that trusts GitLab's OIDC tokens. The shared CI template's build jobs already used it. The publish job got the same recipe:
release-publish:
id_tokens:
CI_OIDC_TOKEN:
aud: https://gitlab.example.com
variables:
AWS_ROLE_ARN: <registry account's CI push role>
AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/oidc-token
before_script:
# static keys outrank web identity in the AWS credential chain
- unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
- printf '%s' "$CI_OIDC_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"
- aws ecr get-login-password ... # unchanged
No secret to rotate, nothing for a cleanup to switch off, and the cloud audit log can see which project and ref made each push.
The Part Where Green Didn't Mean Anything
Two things about release-only jobs made this trickier than it looked.
The fix couldn't rescue the release it was written for. Retrying a job in a tag pipeline runs the CI config from the tagged commit. An OIDC change merged to main doesn't reach a tag that already exists. That's why the stopgap had to live outside the repository.
The merge request's green pipeline proved nothing about the change. The job only runs on tags, so the MR pipeline never ran it. Green there meant valid YAML and nothing else broken. So we looked for evidence somewhere else: another project's tag pipeline, using the same role through the shared template. Its log showed assumed-role/<the CI push role>. The trust policy accepted tag pipelines from that group, and the login path was the same. We compared the two setups line by line, and the only differences were an extra unset and a different AWS CLI build.
One more trap. Once the OIDC change is merged, the static key is no longer a fallback, because the job unsets it. The next release is the real test. We accepted that because the failure would be loud (it happens before anything is pushed) and easy to undo (revert, then tag again).
Epilogue: Not Every Red Is Yours
The next morning, the main branch went red right after the OIDC change merged. The timing pointed straight at the change.
It wasn't the change. The unit tests crashed with 'FixtureManager' object has no attribute '_arg2fixturedefs'. The test framework had shipped a new minor version overnight that removed a private attribute a popular plugin still relied on. The project's lock file wasn't committed, and the framework was pinned to "*", so CI picked up the new release on whatever commit ran next. The fix belonged to another team, and the diagnosis took one diff of the installed versions between the last green run and the first red one.
Hints for the Reader
- "Active in the console" is not "what the job sends". The console shows the real key is healthy. It says nothing about which key your CI variable holds. Reveal the variable and compare IDs before you rotate anything.
- Decode the access key ID. The account is in the key ID. One line of Python tells you whether a variable named after one account holds a key from another.
- Read the error precisely.
UnrecognizedClientException/InvalidClientTokenIdmeans AWS doesn't recognise the key ID or the token.SignatureDoesNotMatchmeans the ID is fine and the secret is wrong. - A cleanup that names a live consumer is a blocker, not a footnote. If the change itself says "still used by X", migrate X first or warn its owner with a date. Consumers that only run on tags or schedules fail late, far from the change.
- Reverse IaC-managed state through IaC. A console click on a Terraform-managed key gets reverted by the next apply, and it undoes someone else's work without telling them.
- A tag-only job isn't tested by your MR pipeline. Green there is about syntax. Find evidence in another project's tag pipeline, or cut a throwaway pre-release tag.
- Retrying a tag job won't pick up your fix. It runs the tagged commit's config. For an already-cut release, fix the environment or cut a new tag.
- Unpinned dependencies make the timing lie. When red appears right after your merge, diff the installed versions before you diff your code.