The 180 Gigabytes That Were Never There
An NVIDIA DGX Spark was running out of disk. The container tooling reported 180 GB of reclaimable space. I reclaimed all of it and got back about one gigabyte.
The obvious move
The box is an NVIDIA DGX Spark that spends its life pulling container images and model weights. Disk was tight. The first stop is the container runtime's own disk report, and it was encouraging: images, build cache, stopped containers, roughly 180 GB reclaimable.
So: prune the dangling images, prune the build cache, prune the stopped containers. The tooling reported success at every step — including one satisfying line about deleting 160 dangling images.
Then I checked the filesystem.
before after freed docker df ~180 GB reclaimable — df 848 G 847 G ~1 G
About a gigabyte. Out of a hundred and eighty.
Why the number was fiction
Two separate things were going on, and both are worth knowing before you trust that report.
The sizes are logical, not physical. Container images are built from stacked layers, and layers are shared. A base layer used by twelve images is counted twelve times in that report. It's summing what the images claim to occupy, not what they occupy on disk. "Reclaimable" is an upper bound assembled from double-counting, and on a machine with many images built from a few common bases, it's wildly optimistic.
Deleting 160 images freed exactly zero bytes. Those were intermediate artifacts from the modern build system, and their actual content doesn't live in the image store — it lives in the build cache, which is a separate pool. Deleting the image records removes bookkeeping entries pointing at data that stays exactly where it was. The count looks like progress. The disk doesn't move. A different command entirely is what drains that pool.
Both mistakes share a shape: I was reading the tool's model of the world instead of the world.
Where the space actually was
Two commands — total disk, then the largest directories — and there it was.
The model weight cache: 506 GB. Not containers at all. Months of downloaded models, most of them dormant, several superseded, one an experiment from a week prior.
Deleting the dormant weights moved the filesystem by exactly the amount deleted, first time, no surprises:
848 G ──────────────────────────────► 574 G
~274 GB of dormant model weights
The whole detour through container pruning was a guess dressed as a diagnosis, and it cost more time than the actual fix.
The near miss
While cleaning up, I nearly deleted something expensive.
An image pulled by digest rather than by tag shows up with its repository set but no tag — which is visually identical to "dangling," the exact category a routine prune targets. It's protected only while a container is actively using it. Stop that service for any reason, run the same cleanup command that had felt harmless all afternoon, and it's gone.
In this case that would have meant a 22.7 GB re-download the next morning, and a service down until it finished.
The fix takes one command: give the image an explicit tag, and it can never be classified as dangling again.
The guard that earned its keep
The other close call was blunter. The deletion list was generated by pattern, and one of the patterns was one wildcard away from matching a different project's data volumes — generated output, not re-downloadable cache. Gone would have meant gone.
What saved it was a rule adopted earlier: every destructive sweep runs behind an explicit name guard listing protected patterns, and the guard aborts the whole pass rather than skipping the match. Not a warning, not a filter — a stop.
Related instinct, same afternoon: a volume with zero container links is not automatically garbage. Sibling artifacts on the host — build images belonging to a named project — were enough to identify orphaned-looking volumes as live project data. "Nothing references it" and "nothing needs it" are different statements.
Hints for the reader
Verify with the filesystem, not with the tool that owns the files. Take a real disk reading before and after any cleanup. If the tool claims 180 GB and the filesystem moves 1 GB, the tool wasn't lying to you — it was answering a different question than the one you asked.
Beware accounting that double-counts shared storage. Anything layered, deduplicated, or copy-on-write — container images, snapshots, deduplicated backups, thin-provisioned volumes — will report logical sizes that sum to far more than the physical footprint. The reclaimable figure is an upper bound built from overlap.
A high deletion count is not evidence of freed space. Deleting hundreds of records that point at data stored in a different pool frees nothing at all, while producing a very convincing progress log.
Find the consumer before choosing the cleanup. Two commands would have pointed straight at a 506 GB directory that had nothing to do with containers. I skipped that step because I already had a hypothesis, and the hypothesis had a convenient tool attached to it.
Guard destructive sweeps by name, and make the guard abort. Any pattern-generated deletion list is one wildcard away from including something irreplaceable. A guard that skips the match keeps going; a guard that stops the run makes you look.
Anything you'd hate to re-download deserves an explicit name. Artifacts pulled by digest sit in the same bucket routine cleanup targets, protected only by being in use at that moment.