The Usual Suspects Had Alibis

A short detective story about a release that died with “Access denied” right after two big CI changes. Both were blamed. Neither did it.

A frontend team cut a release. The image built, the tag got pushed, and then the last job failed. That job is the one that bumps the image version in the GitOps repo so ArgoCD can roll it out:

Cloning into 'manifests'...
remote: HTTP Basic: Access denied. If a password was provided ... If a token was
provided, it was either incorrect, expired, or improperly scoped.
fatal: Authentication failed for 'https://<gitlab>/<gitops-dev-repo>.git/'
ERROR: Job failed: command terminated with exit code 128

The engineer who got the ping had a bad feeling, because two of their own changes had just shipped.

Suspect #1: the OIDC migration

Two weeks earlier they had moved the shared CI templates off long-lived AWS keys and onto GitLab OIDC federation. Every build job now got short-lived credentials. That's a big change in a template every team inherits, and now an auth error had shown up in a job built from that template. It looked like a strong suspect.

Suspect #2: the new runner fleet

The failing job had also run on a brand-new shared runner cluster in a different cloud account. A new runner sits on a new network path with a new service account. Again, a strong suspect.

Interrogation

The case got settled by asking one question: who is actually rejecting the credential?

Look at the error again. remote: HTTP Basic: Access denied comes from GitLab. That means the runner reached GitLab, presented a credential, and GitLab said no. A network or runner problem would show up as a timeout, a DNS failure or a TLS error, not a polite 401 with a help link. So the runner had an alibi.

Then the clone line itself:

git clone https://oauth2:${IMAGE_UPDATE_TOKEN}@${CI_SERVER_HOST}/${MANIFEST_REPO}.git

That's a static GitLab access token in a CI variable. The OIDC work only changed how jobs log in to AWS. It never touched how jobs log in to GitLab. Version control showed the clone line hadn't changed in a very long time. OIDC had an alibi too.

The real culprit

With both suspects cleared, the error message gives three options: incorrect, expired, or improperly scoped. You don't have to guess between them, because GitLab will tell you. The access tokens API for the target repo returns this:

name                         active  revoked  expires     last_used
ci-image-update-token        false   false    <today>     <yesterday evening>
readonly-sync-token          true    false    <next year> <minutes ago>

The token was not revoked and not misconfigured. It expired today. It worked last night, and the first release after midnight UTC hit the wall. Nobody had changed anything. The calendar did it.

The twist: it had a twin

Tokens rarely get created one at a time. Someone sets up a batch of GitOps repos in one afternoon, and every token gets the same default lifetime. So the next check was the sibling repos:

repo        token                   expires
dev         ci-image-update-token   <today>      <- dead
prod        ci-image-update-token   <today>      <- dead, nobody had noticed yet
qa          ci-image-update-token   <in 6 months>
shared      ci-image-update-token   <in 3 months>

The prod token had died at the same moment. Nobody had released to prod that day, so it was a silent failure waiting for the next prod release. Two other repos had been rotated at some point, which is the only reason they were still alive.

While the token list was open, one more expired token turned up at group level. It belonged to a security-scanning integration and had been dead for four days without anyone noticing.

Resolution

  1. Mint new project access tokens for the two dead repos, with the same scopes and a long expiry.
  2. Update the CI variable. One gotcha here: it's a single group-level variable with seven environment-scoped values, each pointing at a different repo's token. So you update the dev value and the prod value, not “the variable”.
  3. Retry the job.

Nothing in the templates changed, and nothing needed to.

Hints for the reader