The Case of the Road That Was Never Built
A short detective story about Terraform, Kubernetes, and error messages that point everywhere except the crime scene.
┌────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├────────────────────────────────────────────────────────────┤ │ │ │ VICTIM: "Instances failed to join the cluster" │ │ | │ │ .------------------+------------------. │ │ | | | │ │ ┌────▼────┐ ┌─────▼─────┐ ┌─────▼─────┐ │ │ │ SUSPECT │ │ SUSPECT │ │ SUSPECT │ │ │ │security │ │ node IAM │ │ subnet │ │ │ │ groups │ │ policies │ │ tags │ │ │ └────┬────┘ └─────┬─────┘ └─────┬─────┘ │ │ | | | │ │ '--------- ALL INNOCENT ---------------' │ │ | │ │ ┌─────▼──────┐ │ │ │ REAL CAUSE │ │ │ │ one CLI │ │ │ │ flag │ │ │ └────────────┘ │ └────────────────────────────────────────────────────────────┘
The Brief
The Company needed a shared CI cluster: one Kubernetes cluster, in its own AWS account, running build agents for every team. Greenfield. The Terraform had been written months earlier and never applied — nobody had pressed the button.
The job was to press the button.
It took five attempts. Every single failure produced an error message that pointed somewhere other than the actual problem. This is the story of the two best ones.
Chapter 1 — The Chicken and the Egg
The first terraform plan failed instantly:
Error: reading EKS Cluster (...): couldn't find resource with data.aws_eks_cluster.<cluster>
Reasonable, on reflection. The configuration installs a GitOps controller into the cluster using Helm — and Terraform's Helm provider has to be configured with the cluster's address and credentials before it can plan anything. Those come from a data source that reads the cluster.
Which doesn't exist yet. Because this same configuration is what creates it.
Terraform resolves provider configuration before planning resources that use that provider, so "it'll exist by the time we get there" isn't an argument it accepts. This is a well-known wrinkle, and it has a well-known workaround: build the cluster first, in a separate targeted run, then apply everything else normally.
terraform apply -target=module.eks_managed_nodes
One command, run once, and the chicken-and-egg is broken. Straightforward.
That command is the villain of this story. It just took fifteen minutes to find out.
Chapter 2 — Fifteen Minutes to Nowhere
The targeted apply ran. The VPC appeared. The cluster control plane came up. The worker node group started building.
Then it sat there. And sat there. And after roughly fifteen minutes:
Error: waiting for EKS Node Group ... unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: NodeCreationFailure: Instances failed to join the kubernetes cluster
Anyone who has run Kubernetes on AWS knows this error, and knows it's a liar. "Instances failed to join" is the symptom of about nine different root causes: wrong security groups, missing node IAM policies, absent subnet tags, a broken auth configuration, a bad machine image, DNS switched off on the VPC.
So the investigation started down the usual list. The VPC module was read line by line. DNS hostnames: enabled. DNS support: enabled. Internet gateway: defined. NAT gateway: defined, sitting in a public subnet with an elastic IP. Private route table with a default route pointing at it, associated with every private subnet. Node IAM role with all three required policies attached. Subnet tags correct.
Every suspect had an alibi. The configuration was, as written, completely fine.
Which raised a much better question than "what's misconfigured?"
What actually got built?
Chapter 3 — The Road That Was Never Built
The apply log lists every resource it creates. Filtering it for just the networking module produced a suspiciously short list:
module.vpc.aws_vpc.vpc module.vpc.aws_subnet.private_subnet
That's it. That's the whole list.
No NAT gateway. No internet gateway. No route tables. No public subnets. The configuration defined all of them. Terraform simply hadn't built any of them.
The private subnets, having no route table of their own, quietly fell back to the VPC's default — which routes traffic within the VPC and nowhere else. The machines booted perfectly. They just couldn't reach the Kubernetes API endpoint to register themselves, or the container registry to pull the software that does the registering.
They weren't misconfigured. They were walled in.
WHAT THE CONFIG DESCRIBED WHAT ACTUALLY EXISTED
┌──────────────────────┐ ┌──────────────────────┐
│ private subnet │ │ private subnet │
│ ┌──────────┐ │ │ ┌──────────┐ │
│ │ node │ │ │ │ node │ │
│ └────┬─────┘ │ │ └────┬─────┘ │
│ │ │ │ │ │
│ ┌────▼─────┐ │ │ X no route │
│ │ route │ │ │ │
│ │ table │ │ │ (nothing else was │
│ └────┬─────┘ │ │ ever created) │
│ │ │ │ │
│ ┌────▼─────┐ │ │ │
│ │ NAT │ │ │ │
│ └────┬─────┘ │ │ │
└────────┼─────────────┘ └──────────────────────┘
│
the internet
So why did Terraform skip them?
Because of the flag from Chapter 1. -target does not mean "this module and the infrastructure around it." It means: start at the target, walk the dependency graph upward, build only what's strictly required to produce the target's inputs.
The cluster module needed subnet IDs. Producing a subnet ID requires a VPC and a subnet. It does not require a NAT gateway — a route table association depends on the subnet, not the other way around. There is no dependency edge from "subnet exists" to "subnet can reach the internet."
That edge exists only at runtime, and Terraform's graph has never heard of runtime.
So Terraform did precisely what it was asked. It built a cluster inside a VPC with no exit, reported success on everything it created, and left the consequences to surface fifteen minutes later as a node registration timeout.
The fix is one flag longer:
terraform apply -target=module.vpc -target=module.eks_managed_nodes
Verdict: the node group went from failing after fifteen minutes to reaching ACTIVE in under three. Nodes join fast when there's a road.
Chapter 4 — The Apply That Wore the Wrong Badge
Cluster up, nodes healthy. One thing left: install the GitOps controller. The pipeline ran, and:
Error: installation failed
* namespaces is forbidden: User ".../<the read-only role>/..."
cannot create resource "namespaces" at the cluster scope
This looked like a straightforward permissions gap — except the permissions had been set up carefully, and deliberately. The pipeline used two separate identities: a read-only role for planning on merge requests, so any branch could safely preview changes, and a write-capable role for applying, locked to protected branches only. The write role had full cluster-admin. It was correct.
And the apply job was using the write role on the AWS side. That was verifiable.
So how was it talking to Kubernetes as the read-only one?
Here's the twist. The pipeline follows the standard safe pattern: plan into a saved file, review it, then apply exactly that reviewed file. Nothing surprising gets applied, because the apply doesn't re-plan.
But a saved plan doesn't just store intended changes. It stores every value Terraform read while planning — including data source results. And the cluster credential comes from a data source, which produces a Kubernetes token by signing a request with whatever AWS credentials were active at the moment it was read.
At plan time. Under the read-only role.
That token gets written into the plan file. The apply job opens the file and replays it. It authenticates to AWS as the write role, then turns around and talks to Kubernetes using a token minted by, and identifying, the plan role.
PLAN JOB APPLY JOB
(read-only identity) (write identity)
│ │
│ reads cluster credential │
│ ── signs token as READ-ONLY ──┐ │
│ ▼ │
│ ┌──────────────────┐ │
└───────────────────►│ saved plan file │──────┘
│ (token inside) │ │
└──────────────────┘ │
▼
talks to Kubernetes as
the READ-ONLY role
✗
The access rules were never wrong. The apply simply never used the identity everyone assumed it was using.
The fix: make the plan that feeds a real apply run under the write identity, while merge-request plans keep the read-only one. Same split, correct direction.
There's a sting in the tail. Those cluster tokens expire in about fifteen minutes. The apply step sits behind a manual approval button — so if someone plans, goes to lunch, and comes back to click "apply," it fails on an expired credential with nothing whatsoever having changed. The durable fix is to stop baking the credential in at all: configure the provider to fetch a fresh token at the moment it's needed.
Chapter 5 — Case Closed
The controller installed in forty-five seconds. Cluster done.
Two wrong turns are worth recording, because both were confident and both were wrong:
"Whoever creates the cluster is automatically its administrator." Widely believed, and false for the module in use here — the setting that grants the creator admin access defaults to off. Had that gone unchecked, everything would have built successfully and then failed on the very last step, requiring another full cluster rebuild to fix. Caught by reading the module's defaults instead of trusting folklore. Pure luck of timing.
A read-only role that can't do its job. The plan role was initially given the obvious "view everything" cluster permission. But Kubernetes' built-in view role deliberately excludes secrets — and Helm stores its release state in secrets. A read-only role using the obvious policy silently cannot inspect Helm releases. There's a second, less obvious read-only policy that includes them; that's the one you need.
Summary — Hints for Fellow Investigators
- When an error names a component, check whether that component was ever built. "X failed to start" and "X was never given what it needs" produce identical symptoms. Read the apply log for what was created, not just what errored.
-targetis a scalpel, and it cuts more than you think. It builds only what's needed to produce the target's inputs — not the surrounding infrastructure that thing depends on at runtime. Dependencies your infrastructure has while running are invisible to a dependency graph built from configuration references. Target whole modules that must exist, and treat every targeted apply as suspect.- A saved plan is a snapshot of everything that was read, including credentials. If a provider is configured from a data source, that data source's value — token and all — travels inside the plan file. Split plan and apply across different identities and you may discover the apply is wearing the planner's badge.
- Short-lived tokens plus manual approval gates are a bad pair. Anything that mints a credential during planning is on a clock. A human-in-the-loop button can easily outlast it.
- Verify the defaults of modules you didn't write. "Obviously the creator gets admin" cost nothing here only because it was checked before the expensive step, not after.
- Fifteen-minute feedback loops punish guessing. When each attempt costs a quarter of an hour, the cheapest move is to stop hypothesizing and go read what actually exists.
ATTEMPT 1..4 ATTEMPT 5 ┌───────────────────────────┐ ┌───────────────────────────┐ │ x no credentials │ │ ✓ VPC + a way out │ │ x ghosts in the state │ ═════► │ ✓ nodes joined in <3 min │ │ x cluster walled in │ │ ✓ controller installed │ │ x wrong badge on apply │ │ boringly, in 45 seconds │ └───────────────────────────┘ └───────────────────────────┘