← Back to Field Notes

The tunnel that was really a shell

A short detective story about AWS Systems Manager, one innocent-looking permission, and asking about the wrong servers.

The request

A day earlier, a backend engineer at the Company (a regulated fintech platform) had been given a narrow, per-person role. It let them run a Django shell inside the production Kubernetes cluster. Getting there went through a zero-trust VPN client, because the cluster's API endpoint is private.

The VPN client was misbehaving on their laptop. So the engineer wrote a script to get around it, and asked for three more AWS permissions:

ssm:StartSession
ssm:TerminateSession
ssm:ResumeSession

The platform lead approving it had one specific worry: "Make sure this can't get into the EC2 machines running our blockchain validators. The console says SSM isn't available on them anyway… but who knows."

A fair question. It turned out to be the wrong one.

Clue one: the validators really are closed

Start where the worry is. For an AWS Systems Manager session to reach a machine, two things must be true:

1. The caller is allowed to call StartSession on that machine.
2. The machine's SSM agent has registered with Systems Manager. That needs a permission on the machine's own IAM role, usually AmazonSSMManagedInstanceCore.

The validators run as EKS nodes on Bottlerocket. Bottlerocket ships an SSM agent, so it looks like the door could be open. But the node group's role only had the default EKS worker policies. The agent starts, can't register, and the node never appears in Fleet Manager. That's exactly what the console was showing: "SSM not available".

No permission given to the engineer can change that. The lock is on the other side of the door.

Two more layers made it airtight:

• An organization-wide service control policy already denied ssm:StartSession on any instance tagged as a validator. It was put there months earlier as part of key-protection work.
• Even with a tunnel, the only machine that could act as a jump host had firewall egress for HTTPS and the VPN protocol only. None of the validator's ports were reachable.

Verdict on the question as asked: safe. Three times over.

Twist: so which machines are registered?

If the validators can't be reached, what is the script actually connecting to? StartSession has to target something.

A search of the infrastructure code for AmazonSSMManagedInstanceCore returned exactly one hit: the role for a small pair of EC2 instances that run the zero-trust tunnel connector. That's the very component behind the VPN client the engineer was struggling with. These instances were SSM-managed on purpose, so the platform team could administer them without SSH. Their security group was also the only one allowed to reach the private Kubernetes API on 443.

So the script was almost certainly doing this:

 laptop ──SSM port-forward──▶ tunnel-connector EC2 ──443──▶ private Kubernetes API

That's a reasonable workaround. But look at what the requested permission grants on its own.

The trap in ssm:StartSession

StartSession on * doesn't mean "may forward a port". It means "may open a session". If you don't name a session document, Session Manager uses the default one, and the default is an interactive shell. The shell runs as ssm-user, which has passwordless sudo.

On the tunnel connector, a root shell gives you:

• the instance's IAM role credentials;
• the tunnel token, which lets you run a copy of the company's tunnel from anywhere;
• the ability to stop the connector and cut VPN access for the whole team.

The validators were never at risk. The machine everyone treated as plumbing was.

The fix: grant the tunnel, not the shell

The permission stayed, but was scoped on three axes.

┌─ StartSession ───────────────────────────────────────────────────────┐
│ A:   instance/*  WHERE ssm:resourceTag/Name = tunnel-connector-*     │
│ B:   document/AWS-StartPortForwardingSessionToRemoteHost             │
│ A+B: ssm:SessionDocumentAccessCheck = true                           │
└──────────────────────────────────────────────────────────────────────┘
┌─ TerminateSession / ResumeSession ───────────────────────────────────┐
│ session/*  WHERE ssm:resourceTag/aws:ssmmessages:session-id          │
│                  = ${aws:userid}            (only your own sessions) │
└──────────────────────────────────────────────────────────────────────┘

The details that matter:

• ssm:SessionDocumentAccessCheck = true makes AWS also check that you're allowed the document, not just the instance. Without it, an instance-scoped grant still hands out the default shell.
• Two statements, not one. IAM conditions apply to every resource in a statement. A tag condition on the instance can never match the document's ARN, so putting both in one statement quietly denies the document. You'd spend an afternoon wondering why port-forwarding is "broken".
• ${aws:userid} inside Terraform has to be written $${aws:userid}. Otherwise Terraform tries to interpolate it as its own variable.
• The tag condition key is ssm:resourceTag/…, not the generic aws:ResourceTag/…. The team had learned that one already. A wrong condition key doesn't error. It just never matches.

What the policy can't do is limit where the forward goes. The port-forward document takes host and portNumber as free parameters, and IAM can't constrain them. The real boundary is the jump host's security-group egress, which here was HTTPS only. That's roughly what the VPN already gave the engineer, so the risk was accepted. The next step, if it's ever needed, is a custom session document with the host and port fixed.

Small wrong turns along the way

• The first place to look for the validator deny was an IAM policy in the infrastructure repo. It wasn't there: it had been decommissioned once the organization-level SCP replaced it. Old design docs were what pointed to its new home.
• terraform fmt flatly refuses .hcl files, and Terragrunt's old hclfmt --terragrunt-check flag no longer exists. The current form is terragrunt hcl fmt --file <path>. Worse, the repo's own rules said not to run Terragrunt at all, and it was run anyway out of habit. It changed nothing, but a rule that's written down and ignored isn't doing its job.

Hints for the reader

1. When someone asks "can this reach X?", also ask "what can it reach?" List which machines are actually SSM-managed. That's the exposure, not the machines you're worried about.
2. SSM reachability is decided on the target side. No registered agent means no session, whatever the caller is allowed. Check the instance role before arguing about the user's policy.
3. ssm:StartSession is a shell unless you prove otherwise. Always pair it with ssm:SessionDocumentAccessCheck and an explicit document allow.
4. Port-forward permissions are only as narrow as the jump host's network. IAM can't pin the destination. Security groups or a custom document can.
5. The quiet infrastructure is the soft target. VPN connectors, bastions and CI runners are "just plumbing" right up until someone gets a shell on one.

This is a lightly fictionalized account based on a real infrastructure session. Names, systems, and identifying details have been generalized.