The Tokens Every Branch Could Read
A short story about two checkboxes in GitLab, one wrong assumption about who creates tags, and a bot that would have quietly broken every release.
The line in the docs
We weren't looking for a security problem. We were moving nine internal Python libraries at the Company to install their dependencies from an artifact registry instead of GitLab's package index. Plumbing work.
The shared CI template's documentation had a reassuring line about the registry credentials used to publish those libraries:
Required variables … set at group level, Masked + Protected.
The project's ticket said the same thing. Before relying on it, we asked the GitLab API for the variable flags — names and flags only, never values:
PUBLISH_USER protected=false masked=true PUBLISH_TOKEN protected=false masked=true PROMOTE_USER protected=false masked=true PROMOTE_TOKEN protected=false masked=true
Masked, yes. Protected, no. And none of the nine libraries had a protected tag rule.
That gap matters more than it looks. Masked only hides a value in job logs. It doesn't stop a job from using it. Protected is what restricts a variable to protected branches and protected tags. Without it, every pipeline gets the token — including merge-request pipelines on any feature branch.
Roughly seventy people had Developer access to those repositories. Any one of them could open a merge request, add one line to .gitlab-ci.yml, and their pipeline would hold a token with deploy rights to the registry. That's the registry every service at the Company installs its packages from.
Nobody had done that. But nothing stopped it either.
Two checkboxes, and an order
The fix is famously simple: tick "Protect variable". It's also famously easy to get wrong, because the publish job runs on tags, and GitLab only passes protected variables to protected refs.
Protect only the variables ─► tag pipelines get nothing ─► every release fails
Protect only the tags ─► MR pipelines still read it ─► nothing fixed
Protect both ─► MR pipelines: empty
tag pipelines on a protected tag: token present
So it's a pair, and the order matters: protected tags first, then the variables. The other way round, any release in that window fails at the publish step with "token not set".
The plan we wrote said: add a protected tag rule v*, then protect the variables. It looked obviously right. Release tags are v1.2.3, aren't they?
The data disagreed
Before clicking anything, we checked what actually creates tags in these repositories. Three things turned up.
- The release validator strips the prefix. The tag-validation job does
TAG="${CI_COMMIT_TAG#v}". It's happy withv3.0.1and3.0.1. - The template told people to tag without it. On every merge request, the template's own bot comment said: *"To publish, create a git tag
3.0.1on main."* Nov. - People had followed that advice. Among the release tags:
2.2.2,2.8.3.
With a v* rule, the next person who followed the bot's instructions would push 3.0.2. It wouldn't be protected, it would get no token, and the release would fail. The fix would have broken releases for exactly the people reading our own docs.
For these libraries the right pattern was *****, every tag. They only use tags for releases, so nothing else is affected.
The wrong turn
Next question: who loses the ability to create tags? A protected tag rule set to "allowed to create: Maintainers" locks out everyone below that.
We pulled group members with access level 40 and up, saw five names at level 40, and wrote in the ticket: "These five Developers will no longer be able to create tags."
A colleague asked a harmless question: "Maintainers and Owners can still create tags, right?"
Right. And level 40 is Maintainer. Developer is 30. Our filter had skipped the actual Developers entirely and mislabelled the Maintainers. The corrected picture was the opposite of what we'd posted:
- Owner (50): can still create release tags. That's everyone who released in the last six months.
- Maintainer (40): can still create release tags. That's the five we'd wrongly warned.
- Developer (30): can no longer create release tags. About seventy people, plus one bot.
To make sure nobody who actually releases would be locked out, we went tag by tag through the API. A tag doesn't record who pushed it, but its pipeline does: tag → pipeline for that ref → pipeline's user. Every release tag in six months came from an Owner.
We posted the correction on the ticket the same afternoon. Writing "I was wrong, here is the real list" beats leaving a ticket that sends people chasing the wrong five.
The second repository
The same publish credentials turned out to have a second home. Another repository, a protocol-definitions project outside the library group, held its own project-level copy of both token pairs. Also unprotected, also readable from around seventy Developers' merge requests. Protecting the group copy didn't help while this copy was still readable.
The obvious move was to repeat the library fix: *, then protect. The data said otherwise again.
This project releases in a more elaborate way. A person pushes v2.75.1. That tag pipeline then runs a job which, using a bot token with Developer rights:
person ──push──► v2.75.1 ──pipeline──► job:
git tag -f go/v2.75.1
git push -f origin go/v2.75.1 ← bot, Developer
trigger pipeline on main ← publish happens here
A * rule would have made go/v2.75.1 a protected tag. The bot, a Developer, couldn't create it. Even if it could, protected tags can't be force-pushed, so a re-run would fail as well. The job pushes the tag before it triggers the publish pipeline, so the whole release would stop silently halfway.
So here the right pattern was **v***:
- The workflow only runs pipelines for tags matching
^v[0-9]+…, so no other tag ever sees a token. - The
go/tags never run a pipeline at all, so they don't need protecting. - Publishing happens in a pipeline on
main, which was already a protected branch.
Two repositories, same credential, opposite patterns. Both answers came from looking at what creates tags, not from what tags usually look like.
One more find: the second repository's other token pair was referenced nowhere — no file, no include, no recent branch. We didn't protect it; we deleted it. An unused deploy credential is pure risk.
Proving it without printing it
"I ticked the box" isn't proof. We wanted to see a merge-request pipeline fail to read the token, without the token ever appearing in a log. One throwaway job in a draft MR did it:
check-token-visibility:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
script:
- for v in PUBLISH_TOKEN PROMOTE_TOKEN READ_ONLY_TOKEN; do
eval "val=\${$v:-}";
[ -n "$val" ] && echo "$v=set" || echo "$v=empty";
done
The important part is the control: a read-only token that must stay visible, because normal builds need it. If everything had printed empty, the test would have proven nothing except that the job was broken.
PUBLISH_TOKEN=empty PROMOTE_TOKEN=empty READ_ONLY_TOKEN=set
Same result in the second repository, and every normal job in both merge requests stayed green. We closed the draft MRs.
What this doesn't prove is the release path itself. So the ticket stayed open until the next real release of the bot-driven repository shows the go/ tag being pushed and publishing going green. Then it closes.
Hints for the reader
- Masked is not protected. Masking hides a value from logs. Only Protected keeps it away from pipelines on unprotected branches. Check the flags through the API rather than trusting the docs.
- Protecting a release secret takes two settings: protected tags and protected variables. Do the tags first. Protecting only the variables breaks releases; protecting only the tags fixes nothing.
- Choose the tag pattern from data, per repository. Ask what the release validator accepts, what your own tools tell people to type, and whether any CI job creates or force-pushes tags.
v*and*can each be right, and wrong in the repository next door. - Bots with low privileges are where protected tags bite. A CI job that creates or force-pushes tags with a Developer-level token will fail on a protected tag, often halfway through a release.
- Know your access levels: 30 Developer, 40 Maintainer, 50 Owner. "Allowed to create: Maintainers" includes Owners.
- Look for copies of the credential. Protecting one copy of a token doesn't help while another copy is still readable. Delete copies nobody uses.
- Prove the result with a control variable. A set/empty check in a throwaway MR, with one variable that must stay
set, is evidence. A ticked box isn't. - When you get it wrong, correct it where you said it. A follow-up comment that says "I was wrong, here's the real list" costs a minute. Leaving the wrong list in a ticket sends people chasing the wrong five.