The read-only user who needed a shell

A short detective story about AWS IAM Identity Center, EKS and one small request.

The request

The message was simple. A backend engineer at the Company, a regulated fintech platform, had read-only access to production through an SSO provider and AWS IAM Identity Center. They wanted to get into the backend pods on the production Kubernetes cluster and run a few Django shell commands.

kubectl get pods worked. kubectl exec did not.

That part was no mystery. The read-only permission set was mapped into EKS with AmazonEKSViewPolicy, which gives get, list and watch, and nothing else. No exec, no port-forward, no Secrets. By design.

So the obvious fix was one line: give the read-only role exec as well.

Twist one: exec is not "read"

Before writing that line, it was worth asking what a shell inside a backend pod actually gives you.

env prints the database password, third-party API keys and the framework's secret key, all injected as environment variables.
The pod runs with its own AWS role (IRSA). From inside the pod you are that role. Here, that role could sign on the platform's behalf with a KMS key.
The Django shell itself writes to the production database.

"Read-only, plus exec" is really "production write access, plus the signing key". And adding it to the shared read-only permission set would have given it to everyone holding that set, not just the one engineer who asked.

So the plan changed. Build a separate permission set, assign it to one person, and keep it as narrow as possible:

 SSO provider
     │
     ▼
 IAM Identity Center ── permission set "ProdShellOperator" (assigned to one person)
     │                     └─ customer-managed policy: eks:DescribeCluster only
     ▼
 member account ─────── role AWSReservedSSO_ProdShellOperator_<hash>
     │
     ▼
 EKS access entry ───── view policy (cluster-wide, read only)
     │                  + Kubernetes group "shell-operators"
     ▼
 Role + RoleBinding ─── pods/exec, in the application namespace only

On the AWS side the role only needs eks:DescribeCluster, which is enough for aws eks update-kubeconfig. Everything else happens inside Kubernetes. A namespaced Role cannot reach outside its namespace, so "exec only in the application namespace" is built into the structure, not left to a convention. One infrastructure namespace already had an admission policy that blocks exec even for cluster admins.

The policy was created, the permission set was created, and the assignment button was clicked.

Twist two: "Failed to create assignment"

Failed to create assignment for user <user> to AWS account <account>
with permission set <permission-set>

That's all the console said. It gave no reason and no hint.

The reason does exist, just not on that screen. Identity Center keeps a status record for every assignment attempt:

aws sso-admin list-account-assignment-creation-status --instance-arn <instance> --filter Status=FAILED
aws sso-admin describe-account-assignment-creation-status --instance-arn <instance> \
  --account-assignment-creation-request-id <request-id>

Both failed attempts reported the same FailureReason:

Received a 404 status error: Not supported policy
arn:aws:iam::<target-account-id>:policy/<policy-name>.

Look at the account in that ARN. It is the production account, the target of the assignment. It is not the account where Identity Center lives.

Twist three: the policy in the wrong account

The fix seemed clear: the policy is missing, so create it. And it was created, in the management account, because that's where the engineer was logged in to edit Identity Center.

Then came the natural follow-up question: maybe I should put the ARN in the permission set instead of the name?

You can't. A permission set can only attach a customer-managed policy by name and path. Identity Center takes that name into each account the permission set is assigned to, looks it up there, and attaches what it finds to the role it creates. A policy in the management account is invisible to that lookup. The assignment would have failed with the same 404, however many copies existed in the wrong place.

The real fix was a small Terraform unit in the production account, creating a policy with exactly that name. One detail mattered here: the module had to create the exact name, with no random suffix. (The Terraform role module in use adds suffixes by default. Its policy module doesn't. Worth checking before you rely on a name.)

After the apply, the assignment succeeded on the first try.

The chicken and the egg

One step in that chain was still blocked: the EKS access entry needs the role ARN, and the role only exists after a successful assignment. Its name ends in a hash that differs per account, so you can't predict it in advance either. That fixes the order:

policy in the target account (apply)
permission set and assignment
read the new role ARN from the target account's IAM
EKS access entry (apply), then the RBAC Role via GitOps

Re-provisioning the permission set later keeps the same role name, so step 4 never has to be redone.

Epilogue: one profile, not two

The engineer now had two roles, read-only for AWS and the operator role for the cluster. That's one profile too many. The fix was to attach AWS's managed ReadOnlyAccess policy to the operator permission set as well.

That turned up the last small trap. Searching "ReadOnlyAccess" in the permission set editor returns hundreds of service policies (AmazonS3ReadOnlyAccess, AmazonEC2ReadOnlyAccess, …) and the plain one disappears among them. It looked like it didn't exist. Setting the Type filter to Job function shrinks the list to about a dozen entries, and there it is.

When the same setup was repeated for the staging environment, it got its own permission set and policy name. A name-based reference means one name across every account, and a policy called "Prod…" living in staging would mislead everyone who read it later.

Hints for the reader

exec into an application pod is not a read permission. It hands over the pod's secrets and its cloud identity. Grant it per person, never to a shared read-only role.
Scope exec structurally. Use an access-entry group plus a namespaced Role. Grant pods/exec both get and create, because modern kubectl execs over a websocket.
Generic console errors usually have a real reason one API call away. For Identity Center, check the assignment creation status.
Customer-managed policy references are names, resolved in the target account. The policy must exist there, under exactly that name, before you assign. Creating it in the management account does nothing.
Some things only exist after the step before them. Reserved SSO roles appear only after a successful assignment, so plan the apply order around that.
If a managed policy seems to be missing, filter by type. It may just be buried.

This is a lightly fictionalized account based on a real infrastructure session. Names, systems, and identifying details have been generalized.