The Case of the Ingress That Wasn't There
A short detective story from the world of GitOps and ingress controllers.
┌──────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├──────────────────────────────────────────────────────────┤ │ │ │ MYSTERY: ArgoCD's own ingress has no ADDRESS │ │ │ │ SUSPECT #1: a self-signed TLS cert with no IP SAN │ │ SUSPECT #2: an ingress-controller migration in progress │ │ KEY CLUE: a "vanished" IngressClass object │ │ │ └──────────────────────────────────────────────────────────┘
The Complaint
The Company runs a small fleet of remote edge devices, each one a self-contained little Kubernetes cluster managed entirely through GitOps. After a fresh reinstall on one of them, the platform team noticed something odd: ArgoCD's own web UI, normally reachable through an ingress, simply had no address. Not slow. Not erroring in an obvious way. Just... not there.
To make things messier, the Project was mid-migration — swapping out one ingress controller for another (Traefik replacing an nginx-based one) — so there were two plausible suspects tangled together from the start: a broken migration, or something the reinstall itself had disturbed.
Two separate mysteries turned out to be hiding in that one symptom. This is the story of both.
Chapter 1 — The Cert With No Address
First stop: check how ArgoCD's ingress was actually configured. It looked standard — terminate TLS at ArgoCD's own server, let the ingress controller talk to it over HTTPS.
Except the logs told a different story:
x509: certificate is valid for argocd-server-<pod-id>, not <service-cluster-ip>
ArgoCD's argocd-server ships a self-signed certificate issued for its own pod hostname — no IP Subject Alternative Names at all. The ingress controller, naturally, was connecting to it over its Service's ClusterIP. An IP address trying to satisfy a certificate that only claims to be valid for a hostname. Classic TLS handshake failure, just dressed up in a way that took a moment to place.
The fix turned out to already be documented upstream, just not applied here: don't ask the ingress to verify a backend cert that was never meant to be verified that way. Run argocd-server in insecure mode and terminate TLS at the ingress instead:
configs:
params:
server.insecure: true
...then point the ingress at ArgoCD's plain-HTTP service port, not the HTTPS one.
The twist: applying half of that fix — just telling the ingress to stop verifying the backend's certificate — didn't fully work. A different error showed up right after: a plain 502, no TLS complaint in sight. Turned out the new ingress controller decides whether to speak HTTP or HTTPS to a backend based on the port number the Ingress object points at — port 443 forces HTTPS regardless of any annotation saying otherwise. The annotation asking for a plain-HTTP backend had been set correctly. The Ingress was still pointed at port 443. The annotation was simply being ignored, silently, because the port made the decision first.
Two settings had to agree, not one:
annotation says: backend-protocol: HTTP Ingress points at: port 443 <-- this wins, always ----------------------------------------------------------- actual behavior: HTTPS to backend, annotation ignored
Fixed by pointing the Ingress at the plain-HTTP-mapped port instead. Cert error gone, 502 gone.
Chapter 2 — The IngressClass That Vanished
Case apparently closed — except a second, unrelated ingress started misbehaving a few days later, during the same controller migration. And this one had no error message pointing anywhere useful.
The migration plan was simple in principle: stand up the new controller, confirm it works, then remove the old one's GitOps-managed application. That last step is where things went sideways.
┌──────────────────────────────────────┐
│ 1. Migrating ingress controllers: │
│ delete the old controller's │
│ GitOps-managed application │
└──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ 2. Automatic prune + self-heal delete│
│ every resource that application owned│
└──────────────────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ 3. ...including a shared │
│ IngressClass object │
└──────────────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ 4. New controller's compatibility mode│
│ NEEDS that IngressClass to exist │
│ — but never creates it itself │
└───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ 5. An unrelated Ingress silently │
│ breaks (nothing points at the cause) │
└──────────────────────────────────────┘
GitOps automated pruning-and-self-healing doesn't just remove an application's obvious, visible resources — it removes everything that application owned, including cluster-scoped objects that other, unrelated things quietly depend on existing. In this case: an IngressClass object, created as a side effect of installing the old controller, that the new controller's nginx-compatibility mode needed to already exist — because that compatibility mode was built to be a drop-in replacement, not a fresh install, and drop-in replacements don't create the class, they assume it's there.
Deleting the old controller deleted the shared object neither controller "owned" on paper, and the new controller had no idea it was supposed to recreate it. The break showed up on a completely unrelated ingress (ArgoCD's own, again — small cluster, everything's a neighbor), with nothing in any log connecting the dots back to a controller migration that had, on the surface, gone fine.
Fix options, in order of how much they cost to set up:
- Recreate the shared object as its own standalone manifest, owned by neither controller's application — so removing or reinstalling either one never touches it again.
- Or, if the replacement controller's chart supports it, let that chart own and recreate the object as part of its own lifecycle — ties it to whichever controller currently needs it. This is the one used here.
Chapter 3 — One More Loose Thread
While closing out the migration, one more small, sneaky setting turned up: a leftover, hardcoded "status address" left over from a different environment's config, silently overriding the controller's own auto-detected address on every ingress it managed. Two independent settings existed for the same purpose — one hardcoded, one automatic — and the hardcoded one always won when both were present, with no warning that the automatic one was being ignored. Every Ingress status in the cluster was quietly reporting the wrong address until that leftover was removed.
Small, but the kind of thing that could've cost another whole afternoon on its own.
Summary — Hints for Fellow Investigators
If you run ArgoCD (or anything else) behind an ingress, especially mid-migration between ingress controllers, a few takeaways:
- Don't verify a cert that was never meant to be verified that way. A self-signed backend cert with no IP SAN and a Service connected to by ClusterIP is a mismatch by design, not a bug to route around — terminate TLS at the ingress instead of fighting the backend's cert.
- One annotation isn't the whole story. Ingress controllers often make backend-protocol decisions from more than one signal (port number and annotation, in this case). If a setting doesn't seem to take effect, check what else might be deciding the same thing with higher priority.
- Automated pruning removes everything an application owned — not just what you meant to remove. Before deleting a GitOps-managed application, ask what cluster-scoped resources it created as a side effect, and whether anything else quietly depends on them still existing.
- A compatibility/drop-in mode is narrower than it looks. "Drop-in replacement" usually means "assumes the old setup's supporting objects are still there," not "recreates them for you."
- When two settings can achieve the same outcome, one of them silently wins. Look for legacy, hardcoded overrides before trusting that the "proper" automatic setting is actually in effect.