The Scanner Said We Were Safe. The CPU Said “For Now.”

A short detective story from the world of JVM security: a scanner's all-clear, a JDK hiding inside someone else's image, and a production fleet that was safe only by accident.

The hook

A new Java CVE dropped with a catchy name: Java4Shell. It was a heap overflow in the JVM's Base64 decoder, pre-auth, with remote code execution demonstrated against several well-known Java products.

Our security scanner already had an answer: not affected.

The question that came in was short: “The scanner says we're safe, but who knows... can you check?”

That “who knows” turned out to be the right instinct.

What the bug actually needs

Reading the write-ups, Java4Shell isn't “every JVM is on fire”. Three things have to line up:

  1. A JVM in the affected range: OpenJDK 17.0.6+, or 18 and later (21, 25, 27...).
  2. Attacker-controlled Base64 reaching java.util.Base64 after the JIT has compiled the decoder.
  3. The right CPU. The bug lives in a hand-written AVX-512 assembly routine (an “intrinsic”) that HotSpot only uses on x86-64 CPUs with AVX-512 VBMI. That means Intel Ice Lake and newer, and AMD Zen 4 and newer. ARM is fine, and so are older x86 chips.

So the triage plan was: find every JVM, find its version, find the CPU under it.

Clue 1: there's only one JVM, and it isn't ours

The Company's platform is mostly Python, Node, Elixir and Go. No pom.xml, no Gradle builds of our own. A quick scan suggested no Java at all.

Except one workload: the blockchain nodes. The network runs on an open-source Ethereum-compatible client, and that client is written in Java. We don't build it. We pull the vendor's Docker image, and the vendor's image ships its own bundled JDK.

That's exactly the kind of thing a dependency scanner doesn't look at. It reads our manifests, and the JDK isn't in any of them. It sits inside someone else's image.

Clue 2: reading the JDK version without Docker

No Docker daemon on the laptop, so there was no docker run … java -version. It turns out you don't need one. A container image is just JSON plus tarballs behind an HTTP API:

token    ← registry auth endpoint (anonymous pull)
index    → pick the linux/amd64 manifest
config   → "history": which build step made which layer
layers   → stream ONLY the layer from "COPY /opt/java/openjdk"
           tar -xzO opt/java/openjdk/release

The answer:

IMPLEMENTOR="Eclipse Adoptium"
JAVA_RUNTIME_VERSION="25.0.3+9-LTS"

Java 25. Squarely in the affected range, in every environment, including the older image external partners run.

So the scanner's “not affected” didn't mean the JDK was safe. It meant the scanner never looked inside the image.

Clue 3: the CPU decides

Last condition: what's under the JVM?

   environment     node family            AVX-512 VBMI?   exposed?
   -----------     --------------------   -------------   --------
   dev             Intel Ice Lake (m6i)   yes             YES
   staging         AMD Zen 3 (m6a)        no              no
   production      AMD Zen 3 (m6a)        no              no

Production was safe. But not because anyone had decided it should be. Someone had once picked AMD instances to save money, and Zen 3 simply doesn't have AVX-512.

The twist: production was safe by accident. One routine “let's move to the newer instance generation” change, to m7a (Zen 4) or m7i, and the vulnerable path quietly switches on, with no code change and no scanner alert.

The fix: protect everywhere, and protect narrowly

There was no patched JDK yet. Oracle had shipped fixes, but the open-source build our vendor uses hadn't. So the fix had to be a JVM flag, and there were three candidates:

We picked the narrowest one and applied it to all environments, not just the exposed one. The point was to stop depending on which instance family someone picks next quarter.

One gotcha: some node types override the JVM options instead of extending them. A flag added only to the shared setting would have silently skipped those nodes, so it went into every override too.

Wrong turn: the check that couldn't fail

After rollout, the obvious verification was to ask the running JVM:

kubectl exec <pod> -- jcmd 1 VM.flags -all | grep DisableIntrinsic

Denied. The namespace has an admission policy that blocks exec, attach and port-forward, because those pods hold validator private keys on disk.

And the quick loop meant to check every pod at once printed... a list of pod names with blanks after them:

for p in $(pods); do
  kubectl exec "$p" -- jcmd … 2>/dev/null | grep … | awk '{print $NF}' || echo MISSING
done

It never printed MISSING, because a pipeline's exit status is the last command's, and awk happily succeeds on empty input. A check that can't fail isn't a check.

The tempting next move was “let's just remove the policy for a moment.” We didn't. Turning off a key-protection control to read a JVM flag is backwards. There was a better way.

Proving it without touching the protected pods

The proof came in two parts, using only reads.

Part 1: what each pod was started with. The pod spec is fixed at start time, and reading it is just an API GET, which admission doesn't block:

validator-0   started 13:08   JAVA_OPTS=... -XX:DisableIntrinsic=_base64_decodeBlock
validator-1   started 13:08   JAVA_OPTS=... -XX:DisableIntrinsic=_base64_decodeBlock
...all 9 pods, all after the sync

Part 2: whether the flag actually works on this JDK and CPU. We started a throwaway pod in a different namespace, on the same image and the same instance type. It ran a tiny Java program that decodes a Base64 buffer 200,000 times (enough to get it JIT-compiled), twice, with -XX:+PrintIntrinsics:

avx512vbmi                                   ← the CPU has the feature
=== WITHOUT flag
  @ 63  java.util.Base64$Decoder::decodeBlock (186 bytes)  (intrinsic)
=== WITH flag
  @ 63  java.util.Base64$Decoder::decodeBlock (186 bytes)  inline (hot)
UseAVX = 3

(intrinsic) means the vulnerable assembly routine is in use. inline (hot) means the method is compiled from plain Java instead. Same image, same CPU, one flag of difference. Proof without a single exec into a key-holding pod.

Plot twist: upstream ships a fix the same day

A few hours later the blockchain client's maintainers published a patch release. Its whole changelog was one line: Java4Shell mitigation.

The obvious first assumption was “new JDK”. It wasn't. Reading the bundled JDK from the image layers again showed the same Java 25.0.3. The release only added -XX:-UseBASE64Intrinsics to the launcher's default JVM options.

Could we drop our own flag, then? Only if our options didn't override the vendor's defaults. So we read the launcher script straight out of the image:

eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $CLIENT_OPTS -classpath ... Main

Our JAVA_OPTS is appended after the defaults, not substituted for them, and nothing we set turns the intrinsic back on. The vendor's flag is a superset of ours. So the upgrade and the cleanup shipped together, as one PR and one restart round.

Hints for the reader