The Bucket That Never Should Have Been Opened

We were asked to make an S3 bucket public so robots could download a bundle of identity-root certificates. The request came with the policy already drafted — a classic public-read bucket policy. It seemed like a ten-minute job. It was not.

The hook: "make it public"

The workload owner wrote: the bundle needs to be reachable by robots, and in production the usual edge domain isn't delivering it. Their proposed fix was one line away: attach a bucket policy granting s3:GetObject to Principal: "*" so anyone could fetch the file directly from S3.

There was a good reason this felt right: the bundle is public by nature — it's a set of public root CA certificates, nothing secret. The property that matters is integrity, not secrecy. So "public" sounded harmless.

But "public bucket" and "public content" are different things, and the fix would have shifted the wrong lever.

The first twist: the edge, not the bucket

The thing the request actually wanted was a reachable public URL. That's the edge's job — a CDN in front of the bucket. And with an edge, you get public reachability without making the bucket public at all.

The correct mechanism is Origin Access Control (OAC): the CDN authenticates to S3 with secrets of its own, scoped to a single distribution, and the bucket stays fully locked down — all four Block-Public-Access settings still on.

So instead of a Principal: "*" policy, the bucket policy said the opposite:

Principal = { Service = "cloudfront.amazonaws.com" }   # not "*"
Action    = "s3:GetObject"
Resource  = <the bundle object only>
Condition = { StringEquals = { "AWS:SourceArn" = <this one distribution> } }

Public to the internet, invisible to everything except one authenticated edge. The bucket was never "opened".

The twist that cost a redeploy: a default flag

First deployment shipped. Then the real test — fetching the bundle through the new endpoint:

HTTP/2 403
x-cache: Error from cloudfront
server: AmazonS3

A 403 with x-cache: Error from cloudfront — the error came from S3, not CloudFront. The edge was reaching the origin and being denied. We'd set up OAC and the policy... hadn't taken effect.

The rabbit hole was the Terraform module. We used the community CloudFront module, and in v4 of that module, create_origin_access_control defaults to false. Nobody flips a flag to "no OAC" on purpose, but the default did exactly that: the distribution was created with an unauthenticated S3 origin — no OAC attached at all. Every fetch hit a private bucket with no credentials and got turned away.

The tell was right there in the API output:

OriginAccessControlId: ""    # empty — the origin was naked

One line fixed it — opt in with create_origin_access_control = true — and the response flipped to 200.

The region trap: certificates only live in one place

Next we wanted a friendly hostname instead of the raw CloudFront domain. That needs an ACM certificate, which the owner already had — a wildcard covering every subdomain.

Here's the part that trips everyone: CloudFront only accepts certificates from ACM in us-east-1. Our wildcard certificate existed in a different region (where it serves the load balancers). To CloudFront, it might as well not exist. A regional certificate in the "wrong" region is invisible to the distribution.

We didn't fight it — we issued a dedicated certificate for the specific hostname in us-east-1 and left the regional wildcard alone. Two certs, different regions, different jobs, no conflict.

The false alarm: "the DNS doesn't resolve"

After all that, the final check: resolve the hostname.

curl: (6) Could not resolve host: ...

Another dead end? No — the local machine was behind VPN, and its DNS was serving a stale "this doesn't exist" cache from before the record was created. Public resolvers answered instantly, and the endpoint served 200 to anyone not sitting behind that one stale cache. A real-looking failure with a mundane cause:

public DNS (any)  -> CloudFront IPs    -> 200  ✓
local VPN resolver -> "No answer"      -> curl error  ✗   (just caching)

Resolution

The bundle ships over HTTPS from a proper edge. The bucket is private — no Principal:"*" anywhere. The distribution uses the dedicated us-east-1 certificate at TLSv1.2_2021. Both the default CloudFront domain and the friendly hostname return 200 with the full bundle payload.

Hints for the reader

"Public content" and "public bucket" are different. If you need a public URL, put an edge (OAC) in front of a private bucket — never open the bucket to *.
When a 403 comes back as x-cache: Error from cloudfront and server: AmazonS3, the denial is at the origin leg, not the viewer. Check whether the origin is actually authenticated (the OAC is attached), not just whether you meant to attach one.
Read the defaults of the module you reuse. Community CloudFront module v4 silently ships an unauthenticated origin unless you opt into OAC. A "no-op default" you didn't notice is a footgun.
CloudFront is region-snobby about certificates. Your wildcard in another region is worthless to it; the cert must live in us-east-1.
A local resolve failure is not an outage. Check a public resolver before you blame DNS. Older "no answer" results get cached by routers and VPNs and lie to you longer than you expect.

This is a lightly fictionalized account based on a real infrastructure session. Names, systems, and identifying details have been generalized.