The Address That Was Half a Key
A short detective story from the world of blockchain infrastructure.
A new validator operator imported their node's private key into a wallet and got a different address from the one our setup script had told them to register. They were right, and the script had been wrong on every node we'd ever built with it.
The complaint
The Project runs a private blockchain on Hyperledger Besu. Some validator nodes are hosted by outside operators on their own cloud accounts. We hand them a Terraform stack and a one-command setup script. The script generates the node key, starts Besu, and writes a small info file with three values the operator sends back to us: the validator address, the public key, and the enode URL (the node's public key plus its IP and port, which peers use to connect).
The operator's message had two parts:
I imported the private key into MetaMask, but the account address doesn't match the validator address in the info file.
and:
cat on the key file shows the key, but with something in parentheses at the end.
A key that gives the wrong address sounds like a user error. Maybe they pasted the wrong thing, or the wallet expected a 0x prefix. That was the first instinct. Then we looked at the screenshot they'd attached.
The giveaway
This is the info file's layout, with fabricated values:
Validator Address : 0x7c1e9ad04b52f8a1c6e3d9b07a44e1f2c85d3a90
Public Key : 7c1e9ad04b52f8a1c6e3d9b07a44e1f2c85d3a90 b3f61e…(88 more hex)
└──────────── identical ────────────────┘
The "address" was the first 40 characters of the public key.
That can't happen by accident. An Ethereum address is the last 20 bytes of keccak256(public key). It's a hash, and it has no reason to look anything like the key it came from. When a value that should be a hash equals the start of its input, the math was never done. Something copied a string.
Where the string came from
The script got the address like this:
docker run --rm -v "$KEY_FILE:/opt/besu/key:ro" "$BESU_IMAGE" \
public-key export-address 2>/dev/null \
| grep -oE '0x[0-9a-fA-F]{40}' | head -1
It reads as: run the export, throw away the noise, grab the first thing shaped like an address.
It has two hidden assumptions, and both were false:
- "
2>/dev/nullthrows away the noise." Besu in Docker writes its log lines to stdout, so redirecting stderr hides nothing. - "The first address-shaped string is the address." Before printing the address, Besu logs the public key it loaded, as
0xfollowed by 128 hex characters.grep -owith an unanchored 40-hex pattern happily matches the first 40 characters of that longer string, andhead -1takes it because the log line comes first.
stdout from the container ───────────────────────────────────────────────────────────── … INFO … public key 0x7c1e9ad04b52…(128 hex) ← grep matches HERE 0x5e0f…(the real address) ← never reached
There was no error and no empty value. The script produced a well-formed address with no private key behind it. It would have taken on-chain votes, and a stake sent to it would have been locked for good.
The public key in the file went through the same kind of pipeline, but its 128-character pattern matched the right key, so the public key and enode were correct. Only the address was wrong.
Twist one: should we roll back?
The operator's enode had already gone into the network's peer configuration the day before. The first question on our side was whether to revert that change.
No. The enode is built from the public key, and the public key was fine. Reverting would have disconnected a correctly configured node to fix a bug that had never touched it. We only needed to correct the address, and we could compute it locally from the public key:
cast keccak 0x<public key> # take the last 40 hex characters
Two independent tools agreed on the result. The operator could now check it against their wallet.
Twist two: it had worked before
The script wasn't new. It lived in a template, and every validator repo had been copied from that template. The bug was in all of them: the template (twice, once for cloud and once for bare metal) and three client repos.
One of those clients had already been through the whole onboarding, including staking, and reported that it all went fine. How could it, with a wrong address in their info file?
Staking gives the answer. The stake is sent from the operator's wallet, and the wallet derives the real address from the key. The stake only succeeds if the on-chain proposal named that same address. So that operator must have given us their wallet address rather than the one in the file. The bug had been sitting in production, and a person had quietly routed around it.
The fix
Three changes, from most to least important:
- Stop scraping stdout. Besu can write the value to a file with
public-key export-address --to=<file>. We mount a scratch directory into the container and read the file. Log lines never go near it. - Match whole values only.
grep -xE '0x[0-9a-fA-F]{40}'rejects a substring of a longer hex string. The-owas the bug. - Check the relationship between values. If the address equals the first 40 characters of the public key, the script now stops with "do not share this address". An address can't legitimately look like that, so this only ever fires on a parsing bug.
Testing it without touching anything real
Our laptops had no Docker, so the one part that couldn't be checked locally was whether this Besu version actually supports --to. We built a throwaway copy of the stack in a development account and made it a node of one: a custom genesis, an empty peer list and no bootnodes. It started, sat at zero peers and block zero, and ran the script.
Two small detours on the way:
- The VPC scare. The new VPC had the same address range as an existing one in that account. Someone suggested dropping the test box into the existing VPC "to be safe". It's the other way round. A stack with fresh state can't modify a VPC it doesn't own, and identical address ranges only clash if the networks are connected. Putting a throwaway node with an open peer-to-peer port inside a shared network is the riskier choice.
- Infrastructure-as-code friction. The first plan failed with no matching EC2 Subnet found. Before the network exists, Terragrunt substitutes placeholder outputs, and a data-source lookup asked AWS about a subnet ID that was literally all zeros. Later, an interrupted destroy left its state lock behind, and the next run failed with a
412 PreconditionFailed, labelled as an apply even though it was a destroy, and signed by our own laptop.
The result: the address in the info file matched keccak256(public key) exactly, and it didn't start with the public key's first characters.
The loose end
The "something in parentheses" was explained to the operator as their shell prompt running into a key file that has no trailing newline. We added ; echo to the command in the info file. Later, while writing it up, we found an earlier note saying Besu's key file does end with a newline. The script's import path writes the key without one, which might reconcile the two. Nobody checked which path this operator used, and nobody saw the parentheses. We've recorded that explanation as plausible, not proven.
Hints for the reader
- Containerised CLIs often log to stdout.
2>/dev/nullis not a filter. If a tool can write its output to a file, use that. grep -oreturns substrings. Pattern-matching a fixed-length value out of mixed output will happily cut it out of a longer value. Use-x, or better, don't scrape.- Encode the invariant. When you have a value and the thing it's derived from, check that they relate the way they must. It costs one line and would have caught this on the first run.
- "It worked for the last client" can mean a person worked around it. Ask how it worked, not only whether it did.
- A bug in a template gets copied into every repo made from it. When you fix one copy, list all the others.
- Say what you haven't verified. A plausible explanation you gave someone isn't a fact until you've checked it.