The Case of the Load Balancer That Never Said Its Name

A short detective story about a patch release, a missing TLS handshake field, and a status code most people have never seen.

┌──────────────────────────────────────────────────────────────┐
│                     INVESTIGATION BOARD                      │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│ MYSTERY:   every request through the load balancer → 421     │
│                                                              │
│ SUSPECT #1:  the routine chart bump merged that morning      │
│ SUSPECT #2:  the load balancer itself                        │
│ KEY CLUE:    the render diff showed only an image tag change │
│ TWIST:       the load balancer never says which host it      │
│              wants, and the proxy started asking             │
│                                                              │
└──────────────────────────────────────────────────────────────┘

The Complaint

Users started reporting a strange error: **421 Misdirected Request**. Not a 500 or a 502, and not a timeout. A 421 is rare enough that most engineers have to look it up.

It wasn't one environment, either. Dev, QA and production all returned it, on every page, for every user.

Chapter 1 — The Usual Suspect

The Company runs its platform on Kubernetes behind an AWS Application Load Balancer. The ALB terminates the user's TLS and forwards each request to Traefik, the in-cluster ingress proxy, which routes it to the right service. Everything is GitOps: merge a change, and ArgoCD applies it.

That morning an automated drift checker had opened its weekly tickets: these Helm charts are behind upstream, please bump them. One MR per environment, a dozen charts each. Traefik was in the list, going from v3.7.17 to v3.7.18.

A patch release. The platform team had even rendered the old and new chart against the real values file before merging. The diff was one line: the image tag.

The three MRs were merged eleven minutes apart: dev, then QA, then prod. By the time anyone noticed dev, prod was already down.

First move, before any investigation: revert Traefik in all three environments. The 421s stopped. The suspect was confirmed, but the motive wasn't.

Chapter 2 — What Does 421 Even Mean?

A 421 Misdirected Request says, roughly: "this request came in on a connection that wasn't meant for this host." Browsers reuse HTTP/2 connections across hostnames that share a certificate. A proxy returns 421 when the hostname negotiated in the TLS handshake doesn't match the hostname in the request.

The hostname negotiated in the handshake is SNI, Server Name Indication: the client names the host it wants during the handshake, before any HTTP is sent.

That's where the load balancer came in.

 user ──TLS (SNI: app.example.com)──▶ ALB ──new TLS (no SNI)──▶ Traefik :443
                                       │
                          terminates, re-encrypts,
                          never says which host it wants

The ALB opens its own fresh TLS connection to the backend, and it sends no SNI on it. For years that didn't matter. So what had changed in a patch release?

Chapter 3 — Reading the Source

The render had shown nothing, because there was nothing to show. The change wasn't in the chart. It was in how Traefik builds its configuration at runtime, from the Ingress objects in the cluster.

The application's own migration guide (not the chart's release notes) had a v3.7.18 entry. The Go source made it concrete:

  1. Traefik runs in "act like ingress-nginx" mode, reading the existing nginx-style Ingress objects unchanged.
  2. As of v3.7.18, that provider creates a separate TLS option for every Ingress host, unconditionally.
  3. A small middleware compares the TLS option of the connection with the TLS option of the router that matched the request. If the names differ, it returns 421. If the request arrived without TLS, the check is skipped.
  4. The ALB's connection has no SNI, so Traefik gives it the default TLS option. The router now has a host-specific option. default ≠ app-host-option, so the result is 421, on every request.

Nothing was broken in Traefik or in the ALB. Each side was doing exactly what it was designed to do, and the two designs had just stopped fitting together.

Chapter 4 — Two Ways Out

Option A, the obvious one: have the ALB talk plain HTTP to Traefik. No TLS means the check is skipped. It was simple, and it was the first plan written down.

It had a catch. The hop from the load balancer to the proxy would no longer be encrypted. "It's inside the VPC" is an answer, but not one the team wanted to give without discussing it first.

Option B, the one that shipped: keep HTTPS from the ALB, but point it at a different Traefik entry point, one where TLS is switched on at the entry point level. Routers on such an entry point inherit its TLS settings, and they inherit them without a per-host option name. That resolves to default, which is exactly what the SNI-less connection has. The names match, the check passes, and the traffic stays encrypted.

 BEFORE                                    AFTER
 ALB ─HTTPS→ :443 "websecure"              ALB ─HTTPS→ :80 "web" (entry-point TLS)
   conn option:   default                    conn option:   default
   router option: per-host   → 421 ✗         router option: inherited default → 200 ✓

Chapter 5 — The Small Traps on the Way

The chart wouldn't take the setting. The obvious values key for "TLS on this port" was rejected by the chart's JSON schema. The flag went in as a raw command-line argument instead, outside the chart's schema.

A redirect loop waiting to happen. Some Ingresses force an HTTP → HTTPS redirect. Traefik decides whether a request "is already HTTPS" by looking at the connection itself, or at the X-Forwarded-Proto header if the sender is trusted. So the entry point first needed to trust forwarded headers from the VPC's address range. With TLS on the entry point this matters less, but it also keeps the real client IP in the logs.

Order of operations is the whole game. ArgoCD syncs the proxy and the ALB Ingress objects as separate applications. If the ALB starts speaking HTTPS to a port that doesn't speak TLS yet, every request fails, and the ALB's health check (which uses a separate port) stays green the whole time. So the fix went in as four separate MRs, merged one at a time with a check in between:

  1. Trust forwarded headers on the target entry point.
  2. Turn on TLS on that entry point. Verify the rollout.
  3. Point the ALB at the new port.
  4. Re-apply the Traefik patch release.

A red herring in the console. After step 3, the AWS console showed a scary "HTTP:80 Not reachable" warning on the load balancer. It referred to the public-facing listener on port 80, which the security groups deliberately block. It had nothing to do with the backend port 80 that had just been wired up. Same number, different side of the load balancer.

Changing the port also replaced the target group. The AWS Load Balancer Controller derives the target group name from a hash that includes the port, so changing the port created a new target group. The load balancer itself kept its name and DNS, and nothing upstream had to change.

Chapter 6 — Proving It

The proxy's access logs weren't in the observability platform, so the proof came straight from the pods. One more small trap: kubectl logs deploy/... reads one pod, not all of them. Counted across every replica:

entryPointName = "web"        3555 requests
entryPointName = "websecure"     0 requests   ← the control

Zero on the old entry point is what made the 3555 meaningful.

Then curl from the outside. The main app returned 200 over HTTP/2 and HTTP/1.1, and an unknown path returned 404. One API returned 401, which looked alarming for a second. The response headers settled it: server: uvicorn, www-authenticate: Bearer, and a JSON body saying authentication was required. That was the application answering a request with no token. The whole chain worked, all the way to the app.

Dev ran the new release with zero 421s. QA and production would follow the same four steps after a soak period, not eleven minutes later.

Hints for the Reader