← Back to Blog

The Not-So-Read-Only Plan

Our CI pipeline logged into the cloud with a pair of long-lived access keys that had sat in a shared secrets store for years. They were the kind of credential that works forever, which is also precisely what made them the thing we wanted gone. The goal was clean and fashionable: exchange the permanent keys for short-lived federated identity — the CI system presents an OIDC token, the cloud validates it against a trust policy, and a role is assumed for the job. No keys to rotate, no keys to leak, no "who left this in the group variables" archaeology.

It took three separate pipeline failures to learn that the plan that plans the infrastructure is not actually read-only — and one more to learn that the "success" we were celebrating had never really happened.

The hook: swap the keys, keep the pipelines green

The setup ran three cloud environments (a "dev-ish", a "test-ish", a "prod-ish"), each its own account, each with its own state. The design was textbook: two roles per environment.

┌──────────────────────────────┐        ┌──────────────────────────────┐
│  WRITE role                  │        │  READ-ONLY "plan" role       │
│  trust: only main + release  │        │  trust: any feature branch   │
│  branch                      │        │                              │
│  perms: everything           │        │  perms: the "read-only"      │
│                              │        │  AWS managed policy          │
│  used by: real applies       │        │  used by: MR previews        │
└──────────────────────────────┘        └──────────────────────────────┘

The idea: a merge request on any developer's branch should be able to run a harmless preview plan against the infra, but only protected refs should ever touch the real apply. So the plan role was given the managed "read-only" policy, and the apply role got full access. Clean separation. Least privilege, done properly.

Then we pushed a branch and watched the pipeline.

Twist one: "read-only" can't read

The first MR plan failed on the very first resource it tried to refresh:

Error: AccessDenied ... not authorized to perform:
  secretsmanager:GetSecretValue  on <a secret>

Not a typo, not a trust-policy problem. The role was assumed — that worked. It's that the "read-only" policy explicitly forbids reading secret values. "Read-only" there means "you may list, describe, and look at metadata" — but reading the actual secret value is a separate action that the managed "everything except writes" policy deliberately excludes. As with standing on a chalk outline, "all reads" turned out not to mean "all reads".

Here's the part that made us chase the wrong thing for a while: our sibling project — the same team, the same federation pattern — had done this exact migration weeks earlier and its plan role needed none of this. Its read-only role worked on the managed policy alone.

The difference wasn't the role. It was the state. The sibling project's state held no secret-value resources, so its plan never touched a value and never tripped the boundary. Our state held dozens of them (database master credentials, service tokens, per-service secrets). A plan refreshes the live value of every one of those, and refreshing a value requires the one permission "read-only" refuses to grant.

Lesson one: "read-only" is a claim about the policy, not about what your state actually needs. A plan that refreshes secret values is not read-only, no matter what the role is called.

Twist two: the single encrypted secret

We granted the plan role a scoped "read these secret values" statement. Dev and test environments flipped to green. Production stayed red — on one secret.

One secret. Production passed everything except a single value. Why was this one different?

Because it was encrypted with a customer-managed key. The cloud's managed "read-only" policy also refuses Decrypt, and this one secret was locked behind a key the team owned. Reading it required two permissions, not one: read the value and decrypt it. We'd granted the first and forgotten the second.

The fix was a Decrypt grant that looked alarming on paper — Resource: "*" — but was safe in practice, because it was pinned to the single cloud service that legitimately calls decrypt, via a ViaService condition, and still scoped to the secrets we'd already whitelisted. Wide-looking, narrow in effect.

Lesson two: one stubborn secret is the tell that something else is being decrypted. A value that won't come back is usually a key problem, not a secret problem.

Twist three: the cluster says "who are you?"

Next the same plan, now able to read secrets... failed on the Kubernetes resources with a flat Unauthorized. Not a permissions error — an identity error. The role had IAM access in order: it could list clusters, read everything AWS. But the Kubernetes API simply didn't recognize it.

Cloud IAM and cluster auth are separate axes. Getting valid cloud credentials gets you to the cluster's front door; it does not open the door. You have to explicitly register the principal with the cluster (an "access entry") and say what it may do once inside. Our plan role had cloud access and zero cluster presence, so every kubectl-flavored call it made during the plan came back Unauthorized.

We gave it a read-only cluster role — one that is actually read-only but still includes Secrets, because the configuration tools read release state from Secrets while planning, and the stock "view" role deliberately hides those.

Lesson three: correct cloud credentials and correct cluster access are two separate grants. "Authentication worked" and "authorization to the thing I'm touching" are different sentences.

Twist four (the sneaky one): the "success" that never happened

Somewhere in here we had a run that passed. Fully green. "The fix works!" Except it didn't — when we re-ran it, the same job failed.

The disappearing success was a stale replay. When you click "re-run" on a finished CI job, the system doesn't run your latest code — it replays the exact commit that job was created from, which in our case was an old, pre-fix revision. The green run had actually been a different job (from an older pipeline, running under a leftover credential path), and we'd been comparing an old success against a new failure and calling it a regression.

old job (old commit, old creds) -> green   <- the "success" we trusted
new job (fixed commit, OIDC)    -> red     <- the real signal

Pushing a real commit gave us a clean pipeline at the current revision and the fuzzy picture resolved.

Lesson four: a re-run is not a fresh run. If a passing job can't be reproduced by the code you'd ship, it wasn't your code passing.

Resolution

We landed the change: two scoped roles per environment (write on protected refs, read-only-with-secret-access on feature branches), explicit cluster access entries so both roles are recognized, and the pre-existing federation provider picked up by import rather than recreated. All three environments' plans pass under the short-lived identity, and the long-lived keys — the original reason for the whole exercise — are scheduled for deletion.

Hints for the reader