The Logout That Killed the Cluster
My coding assistant stopped answering. That was the whole bug report.
The model behind it runs on a pair of small NVIDIA DGX Spark machines under my desk — one large open-weights model split across both GPUs, served by vLLM, reached over an SSH tunnel. When it works you forget it exists. When it stops, you get silence.
So I went looking, expecting a dead network tunnel. Everything I checked said the system was fine.
Act 1: the supervisor lied
The tunnel was healthy — up 22 hours, both forwards established. The cluster manager was cheerful:
node_0 Up 46 hours node_1 Up 46 hours
Two containers, both alive, both running the right image. And yet nothing answered on the port.
The thing about "Up 46 hours" is that it's a true statement about a container, and a worthless one about a service. A container is a box. The process inside it had been dead for fifteen of those hours, and the box didn't care.
What actually knew the truth was a humble cron job. To stop the engine going stale, a script pokes it every 15 minutes and logs the throughput. That log is the highest-resolution outage record on the machine:
16:30 WARM 700 tokens in 8.4s = 83.6 tok/s
16:45 WARM 700 tokens in 10.1s = 69.2 tok/s <-- below baseline
17:00 SKIP server not responding
... every 15 minutes, all night
Dead since 16:45. And look at the last successful line — 17% slower than baseline. It didn't just die; it was already struggling, minutes before.
The system journal explained why, and it was not subtle:
low memory! at or below SIGTERM limits: mem 2.00% sending SIGTERM to process "VLLM::Worker": badness 980 escalating to SIGKILL after 1.1 seconds
An out-of-memory daemon had killed it. Not by accident — by configuration. The vendor's OS image ships with a userspace OOM killer whose preference list reads, roughly:
--prefer (vllm|VLLM|sglang|llama-server|trtllm|python3|...) --avoid (systemd|sshd|dockerd|containerd|NetworkManager)
Read that as a policy: the inference server is the designated victim; the infrastructure is protected. Perfectly sensible for a workstation — you keep your shell, you lose the big job. Completely wrong for a box whose only purpose is serving that model. Nobody chose it. It came in the box.
That's fault one. Restart it and move on, I thought.
Act 2: the restart that wouldn't
The relaunch hung. The head node logged its startup normally and then stopped, mid-sentence, forever:
world_size=2 rank=0 distributed_init_method=tcp://<head>:25000 backend=nccl
That's the rendezvous — where rank 0 waits for rank 1 to show up. Rank 1 never did. Twenty minutes later the head gave up with an error that says nothing at all:
RuntimeError: Engine core initialization failed. Failed core proc(s): {}
An empty set. From the head's point of view nothing had failed — something had merely never arrived.
On the worker node, no model process existed. Its container was up, running a placeholder sleep command, waiting for a job that had died on arrival. And the worker's own logs? Empty. The cluster tool only aggregates the head's output, so the interesting log was on the one machine that wasn't talking.
The real traceback was buried inside the worker's container:
File "multiprocessing/synchronize.py", in __setstate__ self._semlock = _multiprocessing.SemLock._rebuild(*state) FileNotFoundError: [Errno 2] No such file or directory
The parent process had created a named semaphore. The child went to reopen it. It was gone.
The wrong turn
Every instinct says shared memory. Containers plus Python multiprocessing plus a missing semaphore equals "your shared memory is too small" — it's the top answer to that traceback everywhere on the internet.
So I checked. Shared memory: 61 GB, completely empty.
Dead end. Except it wasn't — it was the clue, I just read it backwards. The question isn't why shared memory was too small. It's why it was empty.
The twist
I compared the two machines:
shared memory contents head 3 files, oldest from four days ago <-- survives worker (nothing) dir timestamp = the exact minute of the crash
Same image. Same command. Same user. Opposite outcome.
The difference was who was logged in.
Most Linux distributions enable a systemd setting called RemoveIPC. When a user's last login session ends, it destroys every semaphore and shared-memory segment that user owns. It's tidy-up for interactive logins. It is catastrophic for a long-running job started over SSH.
The head node always had someone connected — my tunnel, my shell — so its memory objects survived for days. The worker only ever received one-shot commands. And the cluster tool starts the remote worker over exactly such a connection: SSH in, launch, disconnect. The moment it disconnects, that's the user's last session ending, and systemd dutifully deletes the semaphores the worker just created.
cluster tool ──ssh──> worker: start model process
cluster tool ──exit─> worker: (last session ends)
│
systemd-logind: "user logged out, cleaning up their IPC"
│
semaphore deleted
│
worker process dies on next spawn ────> head waits forever
Then the part that made me laugh out loud: my own debugging was doing the same thing. Every ssh worker "check something" I ran while a launch was in flight was another session whose exit could wipe the memory. The investigation and the fault were the same action.
The fix is one command per node:
sudo loginctl enable-linger <user>
"Lingering" keeps a user's systemd slice alive permanently, so "last session ended" never happens. The next relaunch came up first try. The keep-warm log flipped from SKIP to 81 tok/s.
Fifteen hours down, fixed by a setting about logging out.
Act 3: "so will it crash again tomorrow?"
Fair question, and the honest answer was: nothing about the cause had been fixed. The OOM policy was untouched. So I went looking for what had actually eaten the memory — the thing the OOM killer had spared while it shot the model.
It was a second model runtime on the same box, serving a smaller model to an in-house call-analytics service. Its configuration held two decisions that only look harmless separately:
- The app calls it to summarise recorded calls, on demand.
- It was configured to keep models loaded forever — the loaded model reported an expiry date in the year 2318.
The timeline snapped together to the second:
16:46 a summarisation task starts; the runtime begins loading a 14.65 GB model 16:51 OOM killer shoots the inference server 16:51 the second model finishes loading (42 seconds) 16:52 the summarisation request fails with a 500, then blows its own 6-minute timeout
Both workloads lost. The task that caused the outage didn't even get its summary.
And there was a quieter finding underneath. Because the first server already held the GPU, the second model got 7 of its 51 layers onto the GPU and ran the rest on CPU — slow enough that a single request couldn't finish inside its caller's timeout. Memory contention had been degrading both jobs for a while before it killed one. That slowness gets blamed on the slow job, never on the contention.
Two more things I got wrong
One. I said unloading the idle model would return 14.65 GB of RAM. It returned about 2 GB of RAM — and 6 GB of swap. After 16 hours idle on a tight machine, the kernel had already paged most of it out. If you size a fix from a process's nominal footprint, you'll be wrong in exactly this direction.
It still helped, but through a door I hadn't looked at. That OOM daemon fires on a conjunction — available memory below a threshold and free swap below a threshold. Free swap had been sitting under its limit continuously, meaning the swap half was permanently satisfied and memory was the only thing standing between the model and a bullet. Freeing swap re-armed a guard I didn't know had been disabled. Read the thresholds as the boolean they are.
Two. The obvious remaining lever was the second model's context window. It had been loaded with a 32,768-token window purely because that's the model's maximum and nobody had set anything — and the key-value cache for that window was 6.4 GB, nearly as large as the 6.7 GB model itself.
Shrink it, obviously. But shrink it to what? Too small and long inputs get silently truncated — the summary comes back looking perfectly normal, having never seen half the call. So I estimated: speech rate, words per minute, tokens per word, adjusted for language. Concluded that 8k was risky for long calls.
Then I checked the database, and found the application had been recording the answer all along — the token count of every request it had ever sent:
prompt_tokens: min 331 | median 693 | p95 1,796 | max 3,217
Maximum ever sent: 3,217 tokens. Against a 32,768-token window. My estimate was three times too high, and I'd been about to reject a configuration the real data comfortably supports.
Every OpenAI-compatible API response carries a token-usage block. Most applications store it, because that's how they do cost accounting. I'd spent effort modelling a number that was sitting in a table.
Hints for the reader
"Running" is not "working." Container and process supervisors answer a question about liveness. If a service has a warm state, a queue, or a port, ask it.
The most informative log belongs to the component that isn't talking. When a distributed job hangs, go to the silent node first. Aggregators only show you the node that's still healthy enough to log.
A hang is not an error, and it won't be reported like one. Waiting is what a rendezvous does. Expect the eventual message to be empty, late, and on the wrong machine.
Check who set the OOM policy before you tune memory. If a preference list names your workload, the victim was chosen long before the crash — and it isn't the process that caused the pressure, so the post-mortem starts on the wrong corpse.
Beware fixes about "cleaning up after logout" on machines that run unattended work. They assume a human's session bounds a human's resources. Long-running jobs started over SSH break that assumption badly.
When your debugging technique is also the trigger, you can chase a bug forever. If a fault appears only when you're investigating, suspect the investigation.
Before estimating a quantity, check whether the system already records it. Cost accounting, request logs and usage columns are measurements taken under real load with real inputs. Your model of the workload is not.
When an estimate and a measurement disagree, publish both. The size of the miss is what tells you how much to trust that kind of estimate next time.
The deepest lesson is the least technical one. Two independent faults, and both were defaults — a vendor's OOM preference list and a distribution's logout-cleanup setting. Nobody chose either. Nobody reviewed either. They were simply how the machines arrived, and they sat there being reasonable for months until one afternoon they weren't.
The configuration you never made is still configuration.