409: The Case of the Resource That Already Existed
A short detective story from the world of Infrastructure as Code.
┌────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├────────────────────────────────────────────────────────────┤ │ │ │ TERRAFORM PLAN VENDOR'S API SAYS │ │ "I will create this" ✗ "409 Conflict — it │ │ (state has no record already exists" │ │ of it at all) │ │ │ │ ┌───────────────────────────┐ │ │ │ SUSPECT: a resource │ │ │ │ that exists in the real │ │ │ │ world, but not in state │ │ │ └───────────────────────────┘ │ │ │ │ MOTIVE: someone built it by hand, weeks ago, │ │ before the pipeline ever got here │ │ │ └────────────────────────────────────────────────────────────┘
The Complaint
It started as a routine rollout. A feature that had already shipped safely to one environment was finally being promoted to the next one down the line — a small, well-tested piece of Infrastructure-as-Code, gated behind a feature flag, flipped from false to true. Nothing about it looked risky.
The deployment pipeline ran. Ten resources went up cleanly — new service accounts, new access grants, a couple of small configuration tweaks. Then, on the eleventh, it stopped cold:
Error: 409 Conflict: The requested stream conflicts with an existing stream with module.<redacted>.event_stream.hook, on event_stream.tf line 30
Terraform's plan had said, in black and white, that this resource would be created — it had no record of it anywhere in its state file. And yet the vendor on the other end was refusing, insisting something with the same shape already existed.
Something didn't add up.
Chapter 1 — "Haven't We Seen This Before?"
The engineer investigating it had a nagging feeling. This exact error — the same wording, the same kind of resource — felt familiar. Surely this had already been solved once, somewhere, recently?
A search through the team's internal notes turned up plenty of adjacent context — how this feature had been rolled out gradually, environment by environment, gated behind a flag; how a routine plan in these repositories tends to be full of unrelated, already-merged changes waiting on a separate promotion step. But nothing about this specific error. Nothing about a fix.
It was tempting to conclude the obvious thing: this is new, nobody's hit it before, let's solve it from scratch. That conclusion got written down as a working note — and then immediately challenged. "I'm almost certain I fixed this already," came the reply. "I just don't remember how."
That single sentence changed the whole shape of the investigation. If the internal notes didn't have the answer, maybe the code itself did.
Chapter 2 — Reading the File's Own Memory
Instead of treating the notes as the final word, the investigation turned to the one place that never forgets: the version history of the file the error pointed at.
$ git log --follow -- event_stream.tf refactor: rename the resource label fix: adopt pre-existing resource instead of re-creating it feat: switch to the new event-stream resource type ...
There it was. Weeks earlier, in an earlier environment in the same rollout, this exact 409 had already happened — and already been fixed. The commit message told the whole story in four sentences:
terraform apply was failing with a 409 — the resource already exists in this environment's account, created out-of-band while the rollout was blocked on a dependency. Terraform had no record of it, so it kept trying to create a duplicate instead of adopting it.A duplicate. That was the missing piece. Somewhere, a resource matching this exact shape had been created by hand — not by Terraform — while a different team, eager to start their own integration testing, didn't want to wait for the infrastructure rollout to catch up.
But this raised an obvious question: if it had already been fixed once, why was the exact same error happening again, in a different environment, today?
Chapter 3 — Same Bug, Different Address
The answer turned out to be almost embarrassingly simple once it was said out loud: every environment in this system is a fully separate account, with its own separate state file. Fixing a problem in Environment 1 teaches Environment 1's state file about a resource that exists in Environment 1's account. It has zero effect on Environment 2 — which is a completely different account, with its own completely empty state, and, it turned out, its own completely real, hand-created duplicate resource sitting there waiting.
Environment 1 (fixed weeks ago) Environment 2 (today)
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ Terraform state: empty │ │ Terraform state: empty │
│ Vendor account: HAS the │ │ Vendor account: HAS the │
│ resource (hand-created, │ │ resource (hand-created, │
│ out-of-band) │ │ out-of-band, independently) │
└─────────────┬───────────────┘ └─────────────┬───────────────┘
│ terraform import │ terraform import
▼ ▼
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ Terraform state: now knows │ │ Terraform state: now knows │
│ about the resource │ │ about the resource │
└─────────────────────────────┘ └─────────────────────────────┘
Two independent teams, two independent environments, the same very human shortcut taken twice: "we don't want to wait for the pipeline, we'll just create it ourselves for now." Nobody did anything wrong, exactly — they just didn't tell Terraform.
Before committing to that theory, one more avenue got tried: reaching the vendor's admin API directly, using credentials already available in the account, to confirm the duplicate resource's exact identity. That path hit a dead end — the credentials on hand were scoped for the application itself to use the service, not to administer it, and the request came back with a firm permissions error. It was a reasonable thing to try, and it didn't work, and that was fine: the vendor's own web dashboard gave up the resource's real identifier in about ten seconds, no special access required.
Chapter 4 — The Fix, and the Cleanup Nobody Should Forget
The fix itself, once the root cause was clear, was almost anticlimactic: terraform import. It's the tool built for exactly this situation — teaching a state file about a resource that already, legitimately exists in the real world, without touching the resource itself.
The process, mirroring the earlier environment's fix almost exactly:
- Confirm the live resource's configuration actually matches what the Terraform code expects (it did — the earlier fix had already broadened the configuration to match, so no code changes were needed this time, just the import).
- Add a small, clearly-labeled, temporary pipeline job whose only purpose is to run
terraform importagainst the specific resource ID, and mark it as something that has to be triggered by hand — never automatically. - Run it once. Watch it succeed.
- Immediately open a follow-up change to delete that job again. A one-off import job left lying around in a pipeline is a loaded gun — harmless the day after it's used, and a very confusing surprise if someone accidentally re-triggers it months later against a resource that's now correctly tracked.
Both steps landed as two small, separate, reviewable changes — one to adopt the resource, one to clean up after itself — exactly matching the shape of the fix from the earlier environment, weeks before.
Summary — Hints for Fellow Investigators
If you manage cloud or SaaS resources with Terraform (or any declarative IaC tool) across multiple environments, a few takeaways worth keeping in your back pocket:
- "We've never seen this before" is a claim about your notes, not about reality. Before accepting that an error is uncharted territory, check the actual version history of the file involved. A one-off fix committed straight into the code, without ever being written up anywhere else, is easy to miss — and
git log/git blameon the exact file is often faster and more authoritative than any wiki search. - A 409-style "already exists" conflict on an empty state file almost always means a resource was created out-of-band — by a teammate, a different team, or an earlier manual test — and nobody told your infrastructure-as-code about it.
terraform importis the fix: teach the state file about reality, don't fight reality to match a state file that's wrong. - Fixing this once does not fix it everywhere. If your environments are genuinely separate accounts/tenants with separate state, the same "someone built it by hand first" shortcut can happen independently in each one. Expect to repeat the fix, and don't be surprised when you do.
- Temporary pipeline jobs need a matching cleanup commit, immediately. If a fix requires a one-off manual step, ship its removal as a near-simultaneous follow-up — don't let "run this once" quietly become "this could get re-run by accident, someday."