The Case of the Allowlist With No List

A short detective story about CI runners, a firewall, and a health check that said "all good" while holding a 403.

┌──────────────────────────────────────────────────────────────┐
│                     INVESTIGATION BOARD                      │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│ MYSTERY:   every end-to-end test login fails on new runners  │
│                                                              │
│ SUSPECT #1:  the new runners can't "see" the QA environment  │
│ SUSPECT #2:  QA itself is down                               │
│ KEY CLUE:    "Health check passed (403)"                     │
│ TWIST:       the "allowlist" rule has no list                │
│                                                              │
└──────────────────────────────────────────────────────────────┘

The Complaint

The Company runs a nightly browser test suite against its QA environment: real Chrome, Firefox and WebKit instances, driven by Playwright, logging in as a dozen test users and clicking through the products.

One morning the pipeline went red, the day after it had been moved onto a new home. Until then it ran on a couple of on-premise CI machines; now it ran on a shared, autoscaling CI runner cluster in AWS. Faster, bigger, no more queueing behind other teams.

The first question from the platform team was a reasonable guess: "Is it the shared runners? Do they just not know about QA?"

Chapter 1 — Ruling Out the Obvious

"Not knowing about QA" usually means DNS, or a private network the runner can't route to. The job log ruled both out within a minute:

=== Pinning DNS for hosts from env/.env.qa ===
  pinned <qa-app-host>   -> <public CDN edge IP>
  pinned <qa-login-host> -> <public CDN edge IP>

Every QA hostname resolved, to public edge IPs of the CDN/WAF in front of QA. Nothing private, nothing missing. The runner could find QA just fine.

So was QA down? From an engineer's laptop, the same URL came back 200. QA was up.

Two suspects down, and the runner could reach a healthy QA. Something in between was saying no.

Chapter 2 — The Health Check That Lied

The test suite starts with a health check, and the log line was right there:

[global-setup] Health check passed (403) — https://<qa-app-host>/

Passed, with a 403. The check treated "the server answered" as "the server is healthy", and a 403 is technically an answer.

Then came the logins, all of them:

✗ user1 — login failed: Login entry point not found
  (neither "Log in" button nor email field appeared).
...
Error: [global-setup] All logins failed. Aborting tests.

The browser wasn't missing a button on the app. It was looking at the WAF's block page, which has no "Log in" button because it isn't the app at all.

A couple of lines earlier, the job had helpfully printed one more clue:

=== Egress IP: <the cluster's NAT gateway address> ===

The old on-prem machines reached the internet from office addresses. The new cluster reaches it from its NAT gateway. To a firewall, those are two completely different clients.

Chapter 3 — "But Why Does the Runner Need QA at All?"

It's a fair question, and it's worth answering, because it explains why this matters.

Nothing gets deployed to QA from this pipeline. It's an end-to-end test suite: the runner pod launches headless browsers, and those browsers are the users. Every page load, every login, every click goes out from the runner, through the cluster's NAT gateway, to QA's edge. If the edge doesn't like the source address, the tests aren't testing the product. They're testing the firewall.

Chapter 4 — It Was Already in the Commit History

Before touching any firewall, the team read the pipeline's own history. The CI config already had a comment on exactly this:

Jobs move to the cloud runners once QA's firewall allows the cluster's NAT IP. Until then they stay local.

A few days earlier the same move had been tried and reverted for the same reason. This time, in a fresh sandbox copy of the pipeline, the jobs went onto the shared runners, but the firewall change they depended on never happened.

So this wasn't a regression. It was a known prerequisite that got skipped.

Chapter 5 — The Twist: The Allowlist With No List

The WAF had a short list of custom rules, and one name stood out: the QA allowlist rule. That looked like the place to add an address.

The platform engineer opened it, expecting a list of IPs, and found this:

(ends_with(http.host, "qa.<company-domain>"))     → Block

No IP condition at all. "It's not that rule, I guess."

It was. It just didn't work the way its name suggested:

  Request to QA
       │
       ▼
┌──────────────────────────────┐   source IP in the
│ Rule A: trusted-IP bypass    │── allowlist? ──► SKIP all remaining
│ action: Skip                 │                  custom rules ──► QA ✓
└──────────────────────────────┘
       │ not in the list
       ▼
┌──────────────────────────────┐
│ Rule B: "QA allowlist"       │
│ host ends with qa.…          │──► BLOCK (403, WAF block page) ✗
└──────────────────────────────┘

The "allowlist" rule was a blanket block. The real allowlist lived one rule earlier, in a trusted-IP bypass rule (action Skip) that let known addresses jump over everything after it. The office IPs were there, which is why the old runners had worked. The new cluster's NAT address wasn't.

Chapter 6 — Choosing Where the Fix Goes

There were two ways in:

  1. Add the NAT address to the Skip rule. One line, same as every other allowed IP. But "skip all remaining rules" means all of them: the geo block, the rules protecting internal endpoints, the rules keeping parts of production off the open internet. And the address belongs to a shared cluster, so every workload running there would get the same pass.
  2. Exclude the address in the QA block rule only.
(ends_with(http.host, "qa.<company-domain>")
   and not ip.src in {<cluster NAT address>})

This opens QA, and nothing else changes.

They picked option 2. The fix should be exactly as big as the breakage, and the breakage was "the test runner can't reach QA", not "the test runner should bypass production protections".

One more worry came up: "Isn't that IP temporary?" In a cluster where nodes come and go every few minutes, it's a fair question. The Terraform answered it: a single NAT gateway with an Elastic IP, which stays the same until someone deletes it. The logs agreed. Jobs on different nodes, hours apart, all left through the same address.

Chapter 7 — Case Closed (For the Weekend)

The rule was changed and the failed jobs retried:

[global-setup] Health check passed (200)
✓ user1 — login complete
✓ user2 — login complete
...  9/9 logins, in all three browsers

From 403 to 200, and every login went through. Two test scenarios still failed afterwards, on a slow page load. That was a different story, handed to the test owners. The door to QA was open.

Then, being sensible, the engineer reverted the change for the weekend. A firewall rule that lets a shared cluster into an environment deserves a second pair of eyes from whoever owns the firewall. Nobody would run the suite before Monday anyway. The fix was proven, documented in the ticket with the before/after expressions, and parked until the review.

Summary — Hints for Fellow Investigators

  1. Moving CI jobs to new runners moves their IP address. Anything your jobs reach that's IP-gated (a WAF, a database firewall, a partner API, even a registry's rate limit) needs revisiting. Make the job print its egress IP; it costs one line and saves an afternoon.
  2. A health check that accepts 403 isn't checking health. "The server responded" and "the server let us in" are different things. Fail the check on 401/403, and you'd find this in seconds instead of after minutes of login retries.
  3. "Login button not found" can mean "wrong page entirely". When a UI test can't find something that's obviously there, look at what the browser actually loaded. A block page, a captcha or a maintenance banner all look like "the button disappeared".
  4. Read the rule, not its name. The "allowlist" rule was a block rule; the actual allowlist was a bypass rule upstream of it. Firewall rules run in order, and the order is part of the logic.
  5. Scope the fix to the breakage. A "skip everything" allowlist is convenient, but for a shared cluster it opens far more than the one door you needed.
  6. Shared NAT addresses are shared. Allowing a cluster's egress IP allows every workload in that cluster. Decide whether that's acceptable, and say so in the rule's description.
  7. Check the history before you debug. The answer was sitting in a CI comment and a reverted commit from earlier that week.

   BEFORE                                   AFTER
  ┌──────────────────────────────┐        ┌──────────────────────────────┐
  │ runner ─► NAT ─► WAF ─✗ 403  │        │ runner ─► NAT ─► WAF ─► QA ✓ │
  │ "Health check passed (403)"  │  ════► │ "Health check passed (200)"  │
  │ 0/9 logins                   │        │ 9/9 logins                   │
  └──────────────────────────────┘        └──────────────────────────────┘