The Scanner Said We Were Safe. The CPU Said “For Now.”
A short detective story from the world of JVM security: a scanner's all-clear, a JDK hiding inside someone else's image, and a production fleet that was safe only by accident.
The hook
A new Java CVE dropped with a catchy name: Java4Shell. It was a heap overflow in the JVM's Base64 decoder, pre-auth, with remote code execution demonstrated against several well-known Java products.
Our security scanner already had an answer: not affected.
The question that came in was short: “The scanner says we're safe, but who knows... can you check?”
That “who knows” turned out to be the right instinct.
What the bug actually needs
Reading the write-ups, Java4Shell isn't “every JVM is on fire”. Three things have to line up:
- A JVM in the affected range: OpenJDK 17.0.6+, or 18 and later (21, 25, 27...).
- Attacker-controlled Base64 reaching
java.util.Base64after the JIT has compiled the decoder. - The right CPU. The bug lives in a hand-written AVX-512 assembly routine (an “intrinsic”) that HotSpot only uses on x86-64 CPUs with AVX-512 VBMI. That means Intel Ice Lake and newer, and AMD Zen 4 and newer. ARM is fine, and so are older x86 chips.
So the triage plan was: find every JVM, find its version, find the CPU under it.
Clue 1: there's only one JVM, and it isn't ours
The Company's platform is mostly Python, Node, Elixir and Go. No pom.xml, no Gradle builds of our own. A quick scan suggested no Java at all.
Except one workload: the blockchain nodes. The network runs on an open-source Ethereum-compatible client, and that client is written in Java. We don't build it. We pull the vendor's Docker image, and the vendor's image ships its own bundled JDK.
That's exactly the kind of thing a dependency scanner doesn't look at. It reads our manifests, and the JDK isn't in any of them. It sits inside someone else's image.
Clue 2: reading the JDK version without Docker
No Docker daemon on the laptop, so there was no docker run … java -version. It turns out you don't need one. A container image is just JSON plus tarballs behind an HTTP API:
token ← registry auth endpoint (anonymous pull)
index → pick the linux/amd64 manifest
config → "history": which build step made which layer
layers → stream ONLY the layer from "COPY /opt/java/openjdk"
tar -xzO opt/java/openjdk/release
The answer:
IMPLEMENTOR="Eclipse Adoptium" JAVA_RUNTIME_VERSION="25.0.3+9-LTS"
Java 25. Squarely in the affected range, in every environment, including the older image external partners run.
So the scanner's “not affected” didn't mean the JDK was safe. It meant the scanner never looked inside the image.
Clue 3: the CPU decides
Last condition: what's under the JVM?
environment node family AVX-512 VBMI? exposed? ----------- -------------------- ------------- -------- dev Intel Ice Lake (m6i) yes YES staging AMD Zen 3 (m6a) no no production AMD Zen 3 (m6a) no no
Production was safe. But not because anyone had decided it should be. Someone had once picked AMD instances to save money, and Zen 3 simply doesn't have AVX-512.
The twist: production was safe by accident. One routine “let's move to the newer instance generation” change, to m7a (Zen 4) or m7i, and the vulnerable path quietly switches on, with no code change and no scanner alert.
The fix: protect everywhere, and protect narrowly
There was no patched JDK yet. Oracle had shipped fixes, but the open-source build our vendor uses hadn't. So the fix had to be a JVM flag, and there were three candidates:
-XX:DisableIntrinsic=_base64_decodeBlockdisables only the vulnerable decode routine. Cost: negligible.-XX:-UseBASE64Intrinsicsdisables the Base64 encode and decode intrinsics. Cost: small.-XX:UseAVX=2disables all AVX-512. Cost: real, and fatal on ARM JVMs.
We picked the narrowest one and applied it to all environments, not just the exposed one. The point was to stop depending on which instance family someone picks next quarter.
One gotcha: some node types override the JVM options instead of extending them. A flag added only to the shared setting would have silently skipped those nodes, so it went into every override too.
Wrong turn: the check that couldn't fail
After rollout, the obvious verification was to ask the running JVM:
kubectl exec <pod> -- jcmd 1 VM.flags -all | grep DisableIntrinsic
Denied. The namespace has an admission policy that blocks exec, attach and port-forward, because those pods hold validator private keys on disk.
And the quick loop meant to check every pod at once printed... a list of pod names with blanks after them:
for p in $(pods); do
kubectl exec "$p" -- jcmd … 2>/dev/null | grep … | awk '{print $NF}' || echo MISSING
done
It never printed MISSING, because a pipeline's exit status is the last command's, and awk happily succeeds on empty input. A check that can't fail isn't a check.
The tempting next move was “let's just remove the policy for a moment.” We didn't. Turning off a key-protection control to read a JVM flag is backwards. There was a better way.
Proving it without touching the protected pods
The proof came in two parts, using only reads.
Part 1: what each pod was started with. The pod spec is fixed at start time, and reading it is just an API GET, which admission doesn't block:
validator-0 started 13:08 JAVA_OPTS=... -XX:DisableIntrinsic=_base64_decodeBlock validator-1 started 13:08 JAVA_OPTS=... -XX:DisableIntrinsic=_base64_decodeBlock ...all 9 pods, all after the sync
Part 2: whether the flag actually works on this JDK and CPU. We started a throwaway pod in a different namespace, on the same image and the same instance type. It ran a tiny Java program that decodes a Base64 buffer 200,000 times (enough to get it JIT-compiled), twice, with -XX:+PrintIntrinsics:
avx512vbmi ← the CPU has the feature === WITHOUT flag @ 63 java.util.Base64$Decoder::decodeBlock (186 bytes) (intrinsic) === WITH flag @ 63 java.util.Base64$Decoder::decodeBlock (186 bytes) inline (hot) UseAVX = 3
(intrinsic) means the vulnerable assembly routine is in use. inline (hot) means the method is compiled from plain Java instead. Same image, same CPU, one flag of difference. Proof without a single exec into a key-holding pod.
Plot twist: upstream ships a fix the same day
A few hours later the blockchain client's maintainers published a patch release. Its whole changelog was one line: Java4Shell mitigation.
The obvious first assumption was “new JDK”. It wasn't. Reading the bundled JDK from the image layers again showed the same Java 25.0.3. The release only added -XX:-UseBASE64Intrinsics to the launcher's default JVM options.
Could we drop our own flag, then? Only if our options didn't override the vendor's defaults. So we read the launcher script straight out of the image:
eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $CLIENT_OPTS -classpath ... Main
Our JAVA_OPTS is appended after the defaults, not substituted for them, and nothing we set turns the intrinsic back on. The vendor's flag is a superset of ours. So the upgrade and the cleanup shipped together, as one PR and one restart round.
Hints for the reader
- A day-one “not affected” from a scanner is a statement about what it scanned. JDKs, Python interpreters and OpenSSL builds bundled inside third-party images are invisible to manifest-based scanning. Go read the image yourself.
- You don't need Docker to inspect an image. The registry API plus
targets you any single file, such as the JDKreleasefile or the launcher script, from the exact layer that wrote it. - “Safe because of the hardware” is a time bomb. If the CPU decides your exposure, apply the mitigation everywhere, or the next instance-type change silently undoes it.
- Prefer the narrowest mitigation, and check where config gets overridden rather than merged.
- Verify the effect, not just the config. “The flag is in the spec” and “the flag disables the vulnerable path on this JDK and CPU” are two different facts. A throwaway pod with diagnostic output proves the second.
- Don't lower a security control to verify a security fix. API reads and an out-of-namespace test pod were enough.
- Give every check a way to fail.
… | awk || echo MISSINGcan't print MISSING. Test the captured value instead. - Read the patch before trusting the patch notes. “Security fix release” here meant new default flags, not a new JDK. It was still useful, but only once we confirmed how it combines with our own settings.