The Second Cable I Didn't Buy
Two NVIDIA DGX Sparks, one high-speed cable between them, and a large language model split across both. The obvious upgrade was a second cable. Measuring first saved the money — and forced me to retract something I'd been telling people.
The setup
Two NVIDIA DGX Spark machines, wired directly to each other over a single 200 Gb/s QSFP link — no switch, just a cable between the two boxes' ConnectX-7 adapters. A large model runs split across both, half the weights on each box, so every token generated requires the two halves to talk.
That last sentence is why a second cable looks obviously correct. Tensor-parallel inference is chatty by construction. If the machines are constantly exchanging data, the wire between them is the bottleneck, and more wire is more speed.
It's a clean story. I wanted a number before spending on it.
Measurement one: zero
I sampled the network interface counters during an active generation run — bytes in, bytes out, over a fixed interval.
0.00 Gb/s.
Not a small number. Zero. The model was demonstrably running, tokens were streaming, and by the operating system's own accounting nothing was crossing the wire.
The mistake is a good one, because it hides behind a completely reasonable assumption: that traffic leaving a machine appears in that machine's network statistics.
This traffic is RDMA — one machine writing directly into the other's memory, deliberately bypassing the kernel networking stack for latency. The counters I was reading live in that stack. They never see it. They aren't wrong; they're blind, and being blind reads exactly like being zero.
The real numbers live in the fabric adapter's own counters, one directory over. With one trap of their own: they're denominated in 4-byte words, not bytes. Read them naively and you understate throughput by exactly 4× — plausible enough to publish without blinking.
Measurement two: too good
With the right counters, I measured how fast the link could push data end to end, then measured how fast the model could process a long input prompt.
The prompt-processing number came back at 113,000 tokens per second.
Anyone with a feel for this hardware knows that's not a real number. It was a cache hit. I'd reused the same prompt across runs, and the server had cached the processed prefix — so I was timing a lookup, not the computation. Unique content per run, and the honest number appeared.
Worth dwelling on which of these two errors was more dangerous. The zero was obviously wrong and got fixed in minutes. The 113,000 was flattering, and flattering numbers get published. If a result makes your setup look great, that's the one to attack hardest.
The actual answer
With honest instrumentation, the question answered itself:
one cable delivers ~185 Gb/s ──────────────────────────────────────────────────── during token generation 3.17 Gb/s 1.7% ▏ during heavy prompt intake 24.40 Gb/s 13.2% ██▍
Peak demand, during the most bandwidth-hungry phase, is 13% of one cable.
A second cable would take utilisation from 13% to 7%. It buys nothing. Token generation here is latency-bound — the two halves exchange small messages and wait on each other — and no amount of extra width makes a round trip shorter.
That's not a universal verdict. A second cable would matter for splitting across more machines, for a pipeline arrangement that ships large intermediate results between stages, or for training, where gradient synchronisation genuinely is bandwidth-hungry. For this workload, on a pair of DGX Sparks, it's money for a graph that stays flat.
The retraction
Here's the part I liked least.
Earlier, configuring the link, I'd found something real: on the DGX Spark each QSFP port presents as two logical devices across two PCIe domains, and driving only one caps out at roughly 109 Gb/s — about half the port's rating. Drive both and you reach ~185. I'd written that up as a 1.7× gain, from a single line of configuration.
Every number in that sentence is correct. The framing is wrong.
At 13% peak utilisation, going from 109 to 185 Gb/s is not a speed increase. It's headroom. The deployment is not fast because of it. It would be fast without it. Worth configuring — it's one line, and it's insurance against a future workload that actually is bandwidth-bound — but it is not why anything performs well.
For a while both claims sat on the same page: "1.7× gain" a few paragraphs above "13% utilisation." A reader would have had to reconcile them, and they don't reconcile. A capacity improvement is only a performance improvement if the workload was bound by that capacity. Mine wasn't, and I'd been quoting the microbenchmark as though it were a result.
Hints for the reader
Measure demand before you buy capacity. The upgrade was intuitive, cheap enough to justify, and pointless. One afternoon of measurement beat a plausible architectural argument — and the argument was mine, so I was well-placed to believe it.
A zero can mean "not measured" rather than "not happening." Any time a counter reads exactly zero while the system is visibly busy, suspect the instrument before the system. Bypass paths — RDMA, kernel bypass networking, direct device access — exist specifically to skip the layer your monitoring lives in.
Check the units on anything you read out of a raw counter. A 4× error from word-versus-byte accounting produces a wrong number that looks entirely credible.
Attack the flattering result hardest. An impossible-looking measurement gets fixed. A merely excellent one gets published. Caching, warm state, and reused inputs all bias in the direction you were hoping for.
Headroom is not speed. A capacity number only becomes a performance number when something was actually constrained by it. If you cite a "1.7× improvement" on a link running at 13%, you're quoting a benchmark, not a result — and if the utilisation figure is on the same page, you've published a contradiction.