The Token That Died While Nobody Was Deploying
A short detective story from the world of CI/CD.
The symptom
A pull request merges and the deploy pipeline runs. Everything is green: seven Docker images built, unit tests, integration tests, a full browser end-to-end suite. Then the very last job, the one that hands the release to the GitOps repository, goes red after less than a second:
RequestError [HttpError]: Bad credentials status: '401' url: 'https://api.github.com/repos/<org>/<gitops-repo>/dispatches'
The code was fine. It was never going to reach the dev environment anyway.
The shape of the pipeline
The Project splits "build" and "deploy" across two repositories, a common GitOps setup:
application repo GitOps repo ┌──────────────────────────┐ ┌──────────────────────────┐ │ build images │ │ bump image version │ │ run tests │ │ for env=dev|stg|prod │ │ trigger_the_deployment ──┼── POST ───►│ ArgoCD syncs the cluster │ │ repository_dispatch │ (token) │ │ └──────────────────────────┘ └──────────────────────────┘
The last step sends a repository_dispatch event to the GitOps repo with a small JSON payload: which environment, which service, which version. Calling another repository's API needs a credential that reaches beyond the repo the workflow runs in. The built-in GITHUB_TOKEN can't do that, so years ago someone created a personal access token and stored it as a secret.
Wrong turn: the run that didn't exist
The first attempt to read the failed job's logs from the terminal returned this:
failed to get run: HTTP 404: Not Found
The run was right there in the browser. The repository wasn't missing, though. The gh CLI had three GitHub accounts logged in, and the active one belonged to a different organisation. GitHub answers 404 rather than 403 for private repos you can't see, so "no access" looks exactly like "no such thing". Running the one command with a different account's token, without switching anything globally, fixed it:
GH_TOKEN=$(gh auth token -u <right-account>) gh run view --job <id> --log-failed
It's a small thing, but it's the second time this exact disguise has cost time on this machine.
Reading the 401 properly
The status code says more than the message. GitHub returns:
- 401 Bad credentials when the token itself is rejected: expired, revoked, malformed, or its owner is gone;
- 403 or 404 when the token is valid but isn't allowed to do this to that repository.
So this wasn't a permissions puzzle. Nobody had to compare scopes. The token was simply dead.
The twist: the timeline
The run history explained how it died unnoticed:
last successful deploy ──────────── ~11 months, no runs at all ───────────► first failure
● ✗ ✗ ✗
The Project had been quiet for almost a year. The token stopped working sometime in that gap, and nothing tried to use it until the first merge after the break. All tests passed, and the very last step failed.
The second twist was a quick search through the workflow files. The same secret fed the staging workflow (twice, a standard and a hotfix path) and the production workflow, which only run on a tag and a release. Nobody had triggered those since the break either. Dev failing loudly was the lucky version: the next production release would have built everything and then deployed nothing.
The fix: stop owning a token at all
The obvious fix is a new personal access token. The workload owner didn't want that, and rightly so. A PAT always authenticates as one human, and a long-lived one is how this happened in the first place.
The replacement is a GitHub App, the closest GitHub has to a service account:
- Create it at the organisation level, not in a repository. Webhook off, no OAuth, and exactly one permission: Contents: Read and write, which is what
repository_dispatchrequires. - Install it only on the target (GitOps) repository.
- Store its ID and private key as secrets in the source (application) repository, the one running the workflow.
- Before each dispatch, mint a short-lived token, scoped to the one target repo:
- name: Generate the GitOps token
id: gitops_token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
owner: <org>
repositories: <gitops-repo>
- name: Emit repository_dispatch
uses: <dispatch-action>
with:
token: ${{ steps.gitops_token.outputs.token }}
The token lives for an hour and is created fresh on every run. Nothing long-lived is left to expire quietly during the next quiet year. All four dispatch steps got the change in a single pull request.
The last surprise: merging the fix deployed nothing
The PR merged. No pipeline started.
The dev workflow had paths-ignore: [".github/**", ...], which is sensible, because editing CI config shouldn't redeploy the application. But this PR only touched .github/. The fix to the deploy workflow was exactly the kind of change the deploy workflow ignores. Testing it takes a manual workflow_dispatch run or the next real code change. That run is still pending as of writing, so the story ends honestly at "should work", not "works".
Hints for the reader
- Read the status code before the message. 401 means "replace the credential". 403/404 means "fix the access". They need different fixes.
- Long-lived tokens die during quiet periods. If a pipeline step depends on one, the first run after a break is the one that finds out. Prefer credentials minted per run (GitHub App tokens, OIDC) over stored ones.
- When one environment's credential breaks, search for every other place it's used. The environments you deploy least are the ones you'll discover last, usually on release day.
GITHUB_TOKENstops at the repo boundary. Cross-repo hand-offs need a real identity. Make it an App installed only on the target, holding only the permission the call needs.- Your CI's path filters apply to CI fixes too. If deploy workflows ignore
.github/**, a fix to one won't run itself. Keep aworkflow_dispatchtrigger and plan the manual run. - A 404 from the CLI on something you can see in the browser is usually the wrong account, not a missing resource.