The Usual Suspects Had Alibis
A short detective story about a release that died with “Access denied” right after two big CI changes. Both were blamed. Neither did it.
A frontend team cut a release. The image built, the tag got pushed, and then the last job failed. That job is the one that bumps the image version in the GitOps repo so ArgoCD can roll it out:
Cloning into 'manifests'... remote: HTTP Basic: Access denied. If a password was provided ... If a token was provided, it was either incorrect, expired, or improperly scoped. fatal: Authentication failed for 'https://<gitlab>/<gitops-dev-repo>.git/' ERROR: Job failed: command terminated with exit code 128
The engineer who got the ping had a bad feeling, because two of their own changes had just shipped.
Suspect #1: the OIDC migration
Two weeks earlier they had moved the shared CI templates off long-lived AWS keys and onto GitLab OIDC federation. Every build job now got short-lived credentials. That's a big change in a template every team inherits, and now an auth error had shown up in a job built from that template. It looked like a strong suspect.
Suspect #2: the new runner fleet
The failing job had also run on a brand-new shared runner cluster in a different cloud account. A new runner sits on a new network path with a new service account. Again, a strong suspect.
Interrogation
The case got settled by asking one question: who is actually rejecting the credential?
Look at the error again. remote: HTTP Basic: Access denied comes from GitLab. That means the runner reached GitLab, presented a credential, and GitLab said no. A network or runner problem would show up as a timeout, a DNS failure or a TLS error, not a polite 401 with a help link. So the runner had an alibi.
Then the clone line itself:
git clone https://oauth2:${IMAGE_UPDATE_TOKEN}@${CI_SERVER_HOST}/${MANIFEST_REPO}.git
That's a static GitLab access token in a CI variable. The OIDC work only changed how jobs log in to AWS. It never touched how jobs log in to GitLab. Version control showed the clone line hadn't changed in a very long time. OIDC had an alibi too.
The real culprit
With both suspects cleared, the error message gives three options: incorrect, expired, or improperly scoped. You don't have to guess between them, because GitLab will tell you. The access tokens API for the target repo returns this:
name active revoked expires last_used ci-image-update-token false false <today> <yesterday evening> readonly-sync-token true false <next year> <minutes ago>
The token was not revoked and not misconfigured. It expired today. It worked last night, and the first release after midnight UTC hit the wall. Nobody had changed anything. The calendar did it.
The twist: it had a twin
Tokens rarely get created one at a time. Someone sets up a batch of GitOps repos in one afternoon, and every token gets the same default lifetime. So the next check was the sibling repos:
repo token expires dev ci-image-update-token <today> <- dead prod ci-image-update-token <today> <- dead, nobody had noticed yet qa ci-image-update-token <in 6 months> shared ci-image-update-token <in 3 months>
The prod token had died at the same moment. Nobody had released to prod that day, so it was a silent failure waiting for the next prod release. Two other repos had been rotated at some point, which is the only reason they were still alive.
While the token list was open, one more expired token turned up at group level. It belonged to a security-scanning integration and had been dead for four days without anyone noticing.
Resolution
- Mint new project access tokens for the two dead repos, with the same scopes and a long expiry.
- Update the CI variable. One gotcha here: it's a single group-level variable with seven environment-scoped values, each pointing at a different repo's token. So you update the dev value and the prod value, not “the variable”.
- Retry the job.
Nothing in the templates changed, and nothing needed to.
Hints for the reader
- Read who rejected you. If the “Access denied” comes from the remote, the network and the runner did their jobs. Look at the credential, not the plumbing.
- Don't let recency pick your suspect. Your latest change is the first thing you think of. Check whether it actually touches the code path that failed. An AWS auth change can't break a GitLab clone.
- “Incorrect, expired, or improperly scoped” can be answered by asking the server. Most platforms expose token metadata, like
expires_at,last_used_atandrevoked. Last used yesterday and inactive today means it expired. - One expired token means you should check its batch. Credentials created together die together. Find the siblings before they find you on a release day.
- OIDC removes one kind of secret, not all of them. After you federate your cloud logins, list what static tokens are still left: git push tokens, registry tokens, scanner tokens. Each one has a countdown on it.
- Expiry dates you don't monitor are outages you've already scheduled. A nightly job that checks
expires_atacross your tokens takes about twenty lines of code. Write it before the next midnight.