The Flatline That Gave It Away
Two NVIDIA DGX Sparks. One big model spread across both. The obvious question: could I have one of those machines back? The first experiment said no, emphatically — and it was measuring the wrong thing.
The setup
Two DGX Sparks, wired directly to each other, running DeepSeek V4 Flash split across both GPUs. Warm, properly benchmarked: 86.6 tok/s peak, 66.4 mean across a spread of prompts.
It works well. It also eats the entire cluster. Tensor-parallel inference means both machines serve one model, so there is no second box left for anything else. That's a real cost when you only own two.
So: what if I ran a smaller model on one node instead, and got a whole machine back?
Experiment one: the obvious answer
I put NVIDIA's Nemotron-3 Super — 120B parameters total, 12B active per token, 4-bit weights — on a single node. Same benchmark harness, same prompts, same warm-up discipline.
15.1 tok/s.
Against 66.4. About 4.4× slower. The conclusion writes itself: freeing a machine costs you most of your speed, so don't. Case closed.
I nearly stopped there.
The problem with that number
Two things changed between the two measurements. The model changed, and the node count changed. A single ratio spanning both tells you the total, and nothing about the split.
That matters because the two answers lead opposite ways. If the gap is mostly hardware, the second machine is doing heavy lifting and must stay. If it's mostly the model, the second machine is nearly irrelevant and I should be shopping for a better single-node model instead.
A 4.4× ratio is compatible with either. It's also exactly the kind of number people quote as though it settled something.
The fix is cheap: run the same model at both settings, and let the axis you can control cancel out.
Experiment two: the same model, both nodes
24.9 tok/s.
Now the arithmetic works:
Nemotron, 1 node → 2 nodes : 15.1 → 24.9 = 1.65× hardware
Nemotron vs DeepSeek, both 2 nodes: 24.9 → 66.4 = 2.70× model
─────
1.65 × 2.70 = 4.4× ← the gap I started with
The second machine is worth 1.65×. Sub-linear, which is expected — you double the memory bandwidth but pay for the two halves to talk.
Everything else, the larger factor, was the model.
I had been about to credit the hardware for something the software was doing.
The flatline
The clue had been sitting in the raw output the whole time. Here are the two models across five very different prompts — counting to 300, a multiplication table, a JSON array, a binary search tree, and a short story:
Nemotron 15.1 15.2 15.2 15.1 15.1 ← flat DeepSeek 86.6 81.8 77.0 67.5 29.6 ← 2.9× spread
One of these is a flatline. The other varies by nearly 3× depending on what you ask for.
That's not noise. It's a fingerprint.
DeepSeek is using speculative decoding: a small draft model guesses several tokens ahead, the big model verifies them in one pass, and every guess that survives is free throughput. Crucially, the guessing works brilliantly on predictable text — a multiplication table, bulk JSON — and poorly on prose, where the next word genuinely could be anything.
So a speculating server's speed depends on the content. A non-speculating one charges the same for every token, however obvious that token was.
Flat means no speculation. And it means the number is honest: 15.1 wasn't a bad sample, it was the whole distribution.
Which reframes the entire comparison. I hadn't measured "big model vs smaller model" or "two machines vs one". I'd mostly measured one model has speculative decoding and the other doesn't — and no amount of extra hardware closes a gap like that.
A coda: the thinking tax
I wired the smaller model up to a coding assistant to get a feel for it. First prompt: "Reply with exactly: NEMOTRON OK."
It answered correctly. It also spent 63 tokens doing so — 56 of them deliberating over the exact formatting of a two-word reply.
With reasoning disabled, the same prompt cost 7 tokens.
Nine times the work for a two-word reply. And because those thinking tokens are generated at the same speed as everything else, they come straight out of the latency budget. At ~25 tok/s that's the difference between two and a half seconds and a third of a second.
Benchmarks hide this almost perfectly. A 1,500-token generation amortises a fixed reasoning preamble into nothing, so the model benchmarks fine and still feels sluggish on the short turns that dominate real interactive use.
Then a sharper edge. I capped a request at 30 tokens. The response came back well-formed, no error, finish_reason: length — and content: null. The reasoning had consumed the entire budget before reaching an answer. Any tool that issues short-budget requests — generating a title, summarising, classifying — gets empty strings instead of failures.
Hints for the reader
A ratio across two changed variables is not a result. If two systems differ in several ways, vary one at a time until the axis you control cancels out. One extra experiment turned an unusable 4.4× into two numbers I could act on — and inverted which one mattered.
Look for the fingerprint, not just the average. The per-prompt spread said more than the mean did. Flat throughput across wildly different content is a strong signal about mechanism, not just performance. Averages hide exactly the structure you need.
Comparing across a feature boundary measures the feature. If one side has speculative decoding and the other doesn't, you are benchmarking speculative decoding. That's a fine thing to measure, but it should be stated, because "model A is 4.4× faster than model B" invites a conclusion the data doesn't support.
Throughput and latency aren't the same claim. A model can post respectable tokens-per-second and still feel slow, if a chunk of those tokens is spent before the answer starts.
The two machines stayed on the big model. Not because the second machine earns its keep — it contributes a modest 1.65× — but because nothing I had available on one node could make up the rest.
That's a different conclusion from the one I started with, and it points somewhere different too: not at buying more hardware, but at finding a single-node model that speculates.