The green deploys that shipped nothing
A short detective story from the world of cloud deployments.
The ask
It started as a one-liner: "Deploy the QA branch to our dev environment on AWS ECS. What do I need to change?"
The honest answer looked like "nothing". The Project — a real-time web app built on Elixir and Phoenix, running on ECS on EC2 — had a manually triggered GitHub Actions workflow that builds whatever branch you point it at, pushes the image to ECR, and rolls it out. The deploy role trusted any branch. Pick the branch, press the button.
By the end of the day that button had gone green three times, and dev had received exactly one of those deploys.
Clue 1: a service that doesn't exist
The first run built and pushed the image fine, then failed on the very last step:
service/<cluster>/<service> is MISSING
The service was there in the console. It just had a different name. Months earlier, an infrastructure change had added a name override to the ECS service, giving it a new suffix. The Terraform map key stayed the same, so the IaC looked unchanged. Only one feature branch's workflow picked up the new name. Every other branch still pointed at the old one.
The detail that made it sneaky: the step just before, "fetch the current task definition", passed. The old task-definition family was never deleted. The pipeline pulled a stale, months-old definition without complaint and only fell over when it tried to update a service that no longer existed under that name.
Two-line fix, merged, rerun. Green.
Clue 2: green, green, and still old
Over the next hour two more commits went out the same way. Both runs green. Then came a routine question: "Is dev actually up to date with the latest pipeline?"
ECS answered differently:
pipeline's latest run → registered revision 7 (newest commit) service is running → revision 5 (two commits behind)
The service events told the story the pipeline didn't:
(deployment …) deployment failed: tasks failed to start. rolling back to deployment … … has reached a steady state.
The ECS deployment circuit breaker had done its job and rolled back both deploys. The GitHub Action was configured to wait for service stability, and a rolled-back service is perfectly stable — on the old revision. The step passed. Nobody was told.
CI job ECS service
────── ───────────
register rev 7 ──────────────► start rev 7 tasks
wait for "stable" … tasks die ✗ tasks die ✗
circuit breaker trips
roll back to rev 5
"service is stable" ◄──────────── steady state (rev 5)
✅ success
Clue 3: the healthy service that couldn't reach its database
Fine, so dev was on the older revision. At least that one worked?
It didn't. Its log stream held thousands of lines like this:
FATAL 28000 (invalid_authorization_specification) no pg_hba.conf entry for host "…", user "…", database "…", no encryption
The managed PostgreSQL instance enforced SSL. The newer commit — the one being rolled back — was precisely the commit that turned SSL on. The version left running couldn't open a single database connection.
And ECS reported it HEALTHY throughout. Both health endpoints came from an off-the-shelf health-check library with no checks plugged in. They answered 200 as long as the HTTP server was up. The load balancer kept sending users to an app that failed every query, and the safety net that would have rolled back a broken revision never fired, because nothing ever called it broken.
So the situation was inverted: the good revision kept getting rolled back, and the broken one kept passing health checks.
Clue 4: exit code 137
Why did the new revision fail to start? The stopped tasks said so plainly:
Exit code: 137 reason: OutOfMemoryError: Container killed due to memory usage
First instinct: the new code leaks memory. Wrong. Tasks from the old revision were being killed too, whenever they restarted during the failed rollouts. This wasn't a code regression. It was a sizing problem that had been sitting there unnoticed.
The task definition looked like this:
task memory = 900 ← reserved on the host container memory = 450 ← hard ceiling; OOM-killed above this
One container in the task. In ECS, the container-level value is a hard limit, whatever the task reserves. Half the allocation was reserved on the host and could never be used.
What pushed the new revision over 450 MiB? Once its database connection worked, the first log lines came from an hourly background job pulling news feeds: several copies at once, fetching and parsing pages, and backing off on a wall of 429 Too Many Requests. The task died seconds later. The old revision never got that far, because without a database the job couldn't even be scheduled. Its tasks hit the same ceiling some other way. The likeliest suspect is the endless connection-retry storm, but nobody proved it, and with that ceiling it hardly mattered.
The fix, plus a side quest
The fix was two numbers in Terraform: task 1024, container 1024. The hosts had room — each instance offered about 3.8 GB to tasks, and even a rolling deploy at 200% put no more than two tasks on any one host.
Running terragrunt plan then threw a red herring:
Error: Required plugins are not installed there is no package for hashicorp/aws 6.52.0 cached in .terraform/providers ERROR There is no variable named "dependency". ERROR resolving dependency "<network-discovery>" outputs: Failed to execute "tofu output -json"
It looks like a lockfile problem. It isn't. Terragrunt's per-unit caches hold symlinks into one shared provider store, and something had pruned old versions from that store. The obvious move — terragrunt init in the dependency the error named — failed the same way and now blamed that unit's dependency. Each fix just moved the error one level down the chain.
A one-line sweep for dangling symlinks showed nearly every unit in the repository was broken. The caches only hold downloaded copies (real state lives in S3), so the fix was to delete them all and re-run:
find . -type d -name .terragrunt-cache -prune -exec rm -rf {} +
After that the plan was clean. One more nice property showed up: the pipeline copies the latest task-definition revision and only swaps the image. So terraform apply registers a revision with the new memory, the next CI run inherits it, and nobody has to fight Terraform for control of the running service.
Verified where it counts
The fourth run went green too. This time nobody took the run's word for it:
- service on the newly registered revision, one deployment,
COMPLETED, no rollback - both new tasks: zero errors in their logs, no OOM kills
- every database error in the last 20 minutes traced back to the two old tasks, now gone
- migrations confirmed by asking the live node directly through ECS Exec: all applied, none pending
One last correction. Midway through, the migration logs showed timestamps from two months earlier, and the first reading was "dev hasn't been migrated since then". Wrong again. Those were simply the oldest migrations that existed only on the QA branch. They had never had a chance to run, because no revision that could reach the database had ever stayed up long enough.
Hints for the reader
- A green deploy step is a claim, not a fact. "Wait for stability" passes after a rollback. After every deploy, check that the service runs the revision you just registered, and fail the job if it doesn't.
- Readiness should test what your app needs. A probe that only proves the web server is up will keep a database-less app "healthy" and stop your rollback safety net from ever firing. Put a cheap database round-trip in readiness, not in liveness.
- In ECS, container memory is a ceiling, not a share. With one container, set it equal to the task's memory. A limit of exactly half usually means a copied default nobody revisited.
- Size for the boot spike, not the steady state. Background jobs that start right after boot set the real peak.
- When you rename a resource, delete the old one — or something will happily keep reading it.
- When an error names the wrong component, follow the chain. Fixing whatever the message names can just push the error one link further along.
- Fixes unmask each other. The SSL fix was correct, but the first thing it let the app do was use more memory than the container allowed. Two separate problems combined into a deploy that failed, rolled back, and went green.