The Case of the Runner That Was Blamed Twice

A short detective story about a CI runner accused of two crimes in one morning. It was innocent of the first. For the second, it was framed by a single word added to its name tag.

 ┌──────────────────────────────────────────────────────────────┐
 │                      INVESTIGATION BOARD                     │
 ├──────────────────────────────────────────────────────────────┤
 │                                                              │
 │   CASE 1: "pipelines keep getting canceled"                  │
 │       suspect: the new runner ......... NEVER EVEN TOUCHED   │
 │       culprit: a GitLab setting doing exactly its job        │
 │                                                              │
 │   CASE 2: "image builds fail on the new runner"              │
 │       suspect: the new runner ......... GUILTY, ON PAPER     │
 │       culprit: one tag, added to it that same morning        │
 │                                                              │
 │   evidence: 5 canceled pipelines, 0 jobs that ever started   │
 │             7 projects, 1 identical error, 1 changed label   │
 └──────────────────────────────────────────────────────────────┘

The Setup

The Company was moving its CI to a new home. The old GitLab runner was a classic: one big VM, Docker executor, privileged containers. It worked, and nobody wanted to explain why it was safe.

The new runner lived on a shared Kubernetes cluster. Jobs became pods, nodes came from Karpenter, and one rule was deliberately strict: no privileged pods. CI jobs run code from many teams, so they're treated as untrusted.

The migration followed a careful pattern. The new runner had its own tags and refused untagged work, so it picked up nothing unless a pipeline asked for it by name. Teams opted in one at a time. The old runner kept serving everyone else.

That pattern is what made the next morning so confusing.

Case 1 — The Pipelines That Died Unborn

The first report came in mid-morning:

Look at the deploy repo's pipelines. They're getting canceled. I think it's the new runner. Its pod looks fine though.

The pipeline list was striking. Three canceled in a row, then one green. Earlier in the morning: two canceled, then one green.

The first useful step was to put two columns next to each other: when each pipeline was canceled and when the next one was created.

 canceled at   next pipeline created at
 ───────────   ────────────────────────
  07:35:47      07:35:46
  07:36:05      07:36:04   ← that one succeeded
  09:20:56      09:20:47
  09:21:23      09:21:17
  09:22:01      09:22:00   ← that one succeeded

Every cancellation landed within seconds of the next pipeline appearing. Then the canceled jobs themselves: no start time, no runner assigned. Not one had reached any runner, old or new.

Something upstream of every runner was doing this. It was GitLab's "auto-cancel redundant pipelines", switched on for the project. When a new pipeline starts on a branch, older ones that haven't started yet get canceled.

But these pipelines had started. Hadn't they? Their first stage was a gate job with when: delayed and start_in: 5 minutes, meant to give ArgoCD time to roll out before downstream checks fired. A delayed job sits in a scheduled state, and GitLab counts scheduled as not started. So every pipeline spent its first five minutes cancelable. Any two merges less than five minutes apart killed the older pipeline.

A developer had merged a handful of MRs back to back. Completely normal behaviour for a deploy repo.

Verdict on the runner: innocent. It never saw these jobs.

The wrong turn

The first closing summary said something reassuring: "Nothing was lost. The newest pipeline runs against the newest main, which contains all the earlier changes."

Then the next question came: "So should we turn auto-cancel off?" To answer it honestly, someone had to read what the surviving pipeline actually did. That's where the reassuring summary fell apart.

A deploy-health job in that pipeline didn't look at the state of main. It diffed this push against the previous push:

git diff "$CI_COMMIT_BEFORE_SHA" "$CI_COMMIT_SHA"

Each merge is its own push. So the one surviving pipeline checked the health of the last merge only. The other three deploys in that burst never got a health report, and nobody was told.

"The newest pipeline covers it" is only true if every job looks at the branch, not the push. Auto-cancel went off. The earlier summary got an explicit correction rather than a quiet edit.

Case 2 — The Error That Wasn't Supposed to Be Possible

Less than an hour later, a second report arrived with a log pasted in:

error: failed to solve: failed to read dockerfile:
  failed to mount /tmp/buildkit-mount…: [{Type:bind …}]:
  … err: operation not permitted

This time the new runner had run the job. The pod name in the log proved it.

The error itself was familiar. Weeks earlier, during the first trial of the new runner, the team had hit exactly this wall. The shared build template uses BuildKit's buildctl-daemonless.sh, which starts a rootful BuildKit daemon inside the job container. Rootful BuildKit needs to bind-mount its snapshots and write to cgroups. On Kubernetes that means privileged: true, and nothing narrower will do. That's why the new runner was never supposed to receive these jobs.

So why was it receiving them?

The runner's own job history told the story in one query. Until about 09:15 it had only run the platform team's infrastructure jobs. After 09:15 it was running image builds for frontend apps, backend services, everything.

 project               job               runner   time    result
 ───────────────────   ───────────────   ──────   ─────   ───────
 service-A             build_container   OLD      08:51   success
 service-A             build_container   OLD      08:58   success
 service-A             build_container   NEW      09:15   failed
 service-A             build_container   NEW      09:25   failed
 frontend-B            build_container   NEW      09:37   failed
 service-C             build_container   NEW      09:40   failed
 …seven projects, one identical error

Then the runner's tag list. Next to its two opt-in tags sat a third: the shared tag the old runner serves, the one every build job in the organisation carries. Someone had added it that morning. The new runner went from "picks up nothing unless asked" to "competes for everything" in one edit.

Lightweight jobs like linters and manifest updates passed on it, which made the outage look partial and random. Only image builds failed, and only when the new runner happened to win the race for the job.

Verdict on the runner: it did fail the builds. But it was set up to. Removing the tag sent all builds back to the old runner within minutes.

The Real Fix — and the Twist in the Upgrade

"Remove the tag" stops the bleeding. It doesn't let the migration continue. The obvious next question was the one everyone asks: can't we just fix it in the runner config?

Technically, yes: privileged = true. But that would hand node-root to every CI job on the runner, including the infrastructure jobs that carry cloud credentials. That's the exact design the migration existed to avoid. A cleverer runner-side hack (overriding the build script to point at a shared daemon) would push every build in the company through one unauthenticated socket. No.

The real fix was in the template. The shared build component, owned by another team, had moved on: a newer major version and a published rootless BuildKit image. Rootless BuildKit runs inside a user namespace, needs no privileges, and needs one flag on Kubernetes (--oci-worker-no-process-sandbox). The template simply pinned the old version from months earlier.

So: bump the version, switch the image, done?

Not quite. Reading the diff between versions turned up the twist. The new major version had moved its entire body, registry login and build, from before_script into script. The Company's wrapper templates did this:

before_script:
  - !reference [.shared-build-job, before_script]   # ← no longer exists in v2

A bare version bump would have pointed every wrapper at a key that no longer existed. The wrappers were rewritten to keep only their own steps. The upgraded component then merges its registry credentials into the same config file. The file got checked to be sure it merges (it uses jq) rather than overwrites.

Proving It Without Breaking Anyone

The change would land on roughly seventy projects at once, so it was proven before anyone reviewed it.

The method: a throwaway branch in one of the projects that had failed. The diff was tiny: point the include at the fix branch, and run the same build twice with a per-job tags: override. One copy went to the new runner, one to the old.

 build_container          → new runner (unprivileged)   ✅ 2m33s
 build_container_on_old   → old runner (privileged)     ❌ ...

The second wrong turn came from this test, not from the fix. The old-runner build had compiled everything, then failed at the very last step:

The image tag '…-<commit>' already exists … (immutable)

Both jobs built the same commit, so they produced the same tag, and the registry refuses to overwrite tags. The first job to finish won. A one-line commit to get a fresh tag, a rerun, and both runners came back green, with the same image, cache, credentials and secrets as before.

Then the question that actually matters: is it safe, and will pipelines produce what they did before? That meant more than "it built once":

The Last Catch

The rollout plan ended with a reasonable-sounding step: "merge, then watch the first builds on the new runner."

The engineer reviewing it spotted the hole immediately:

If the runner isn't tagged, how am I supposed to see that?

Correct. With the shared tag removed, no build would ever reach the new runner after the merge. "Watch the new runner" would watch nothing. The plan became three stages, each answering a different question:

 stage 1  merge            → new template, OLD runner only   (is it safe for everyone?)
 stage 2  per-job canary   → one MR build routed to NEW      (does it work there?)
 stage 3  re-add the tag   → builds split across both        (does it hold under real traffic?)
          rollback at any stage: revert, or remove the tag

There's a known gap, written down rather than hidden: a release-tag pipeline uses the CI file from the tagged commit, so stage 2 can't canary the release path. That one gets watched in stage 3.

Hints for the Reader