The Case of the Package That Wasn't There
A short detective story from the world of Python packaging and artifact proxies.
┌────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├────────────────────────────────────────────────────────────┤ │ │ │ [ public pkg: requests ] [ internal pkg: lib-x ] │ │ HTTP 200 ✓ HTTP 404 ✗ │ │ \ / │ │ \ / │ │ \ / │ │ ┌────────────────────┐ │ │ │ SAME REPO URL │ │ │ │ "pp-pypi-public" │ │ │ └────────────────────┘ │ │ │ │ Verdict: it mirrors PUBLIC PyPI only. │ │ The internal package was never there. │ └────────────────────────────────────────────────────────────┘
The Complaint
It started as a routine follow-up: "there's an old branch that adds support for pulling Python packages through an artifact-curation proxy — can you pick it back up, get it current, and check whether it actually works?"
Simple enough, on paper. The branch was months old. The main branch had moved on without it. And nobody had actually finished proving the idea worked end to end.
Chapter 1 — A Branch With Secrets
The first step was mundane: fetch, compare, rebase. The old branch was one commit ahead of main and several commits behind it — a clean, boring rebase, or so it looked.
Then came the push. Git said no.
! [rejected] feat-branch -> feat-branch (stale info)
That single word — stale — was the first real clue. A fetch of the feature branch itself (not just main) revealed the twist: a colleague had quietly pushed two more commits to the exact same branch, sometime after the local clone had last synced. One of those commits regenerated the dependency lock file. The other made a very specific, very deliberate change: it pointed every internal package's install source at the new artifact-curation proxy, and deleted the old fallback source entirely.
Git's safety check — refusing to push when the remote had moved — had just prevented a rebase from silently deleting a teammate's work. First mystery solved before the real investigation even began: always fetch the branch you're about to force-push, not just the trunk you're rebasing onto.
Chapter 2 — Merging Two Timelines
With the real, current branch tip in hand, the rebase ran again. This time it hit a real conflict: main had bumped the same internal packages to newer version numbers, while the colleague's commit had reassigned where those same packages should be downloaded from. Both changes were valid; they just touched the same lines.
The resolution was straightforward in isolation — keep the newer versions, keep the new source — but it produced something that had never actually been tested: a configuration nobody had run end-to-end yet.
MAIN SAYS: BRANCH SAYS:
lib-x = 2.2.2 (gitlab source) lib-x = 2.1.0 (new-proxy source)
\ /
\ /
▼ ▼
lib-x = 2.2.2 (new-proxy source) <- merged, untested
Chapter 3 — The Package That Wasn't There
Time to actually run it. Locking the dependencies fresh against the merged configuration failed immediately — not with a version conflict, but with something more fundamental. Every internal package the project depended on came back the same way:
Authorization error accessing .../pp-pypi-public/simple/lib-python-internal-service-client/
An authorization error is ambiguous by nature — it could mean "you're not allowed to see this," or it could mean something else entirely wearing an auth error's clothes. Getting real credentials configured turned the ambiguous error into a precise one: the same request came back 404, not 403. Not "forbidden." Not found.
A quick, deliberate test settled it. Query the same proxy for a package everybody knows is public:
curl .../pp-pypi-public/simple/requests/ -> 200 OK curl .../pp-pypi-public/simple/lib-python-internal-x/ -> 404 Not Found
The proxy was doing exactly what it was built to do — mirror the public PyPI index, with the curation and scanning benefits that implies. It had simply never been asked to host, or mirror, anything internal. The colleague's commit had pointed a whole set of internal packages at a storefront that had never carried them, then torn down the only shelf that did.
It's an easy trap: the tool looks like a single unified package index from the outside — one URL, one simple/ API, same shape as every other Python package repository. Nothing about the interface hints that it's scoped to only half the packages you might ask it for.
Chapter 4 — Splitting It Back Apart
The fix undid exactly the part that was wrong, and kept the part that was right:
- Internal packages went back to being sourced from the original internal registry.
- The new artifact-curation proxy stayed in place as the primary source, for every public, third-party dependency — which is where its actual benefit (scanning, caching, curation) lives.
- The original internal registry was restored as a secondary source, exactly where it had been before the well-intentioned but incomplete change removed it.
One more gap remained: the checked-in lock file was now stale relative to the corrected configuration, and regenerating it locally needed credentials that weren't available in the moment. Rather than block on that, the fix was made self-healing at the point it actually matters — the container build step was changed from a plain install to relock-then-install, so a lock file that drifts out of sync with its own manifest doesn't quietly break the pipeline that builds the production image.
BEFORE AFTER install-only relock, then install ┌─────────────────┐ ┌─────────────────┐ │ trusts the │ │ regenerates the │ │ committed lock │ ───── fixed ────► │ lock file first,│ │ file blindly │ │ every build │ └─────────────────┘ └─────────────────┘
The corrected pipeline was pushed and watched live. The build succeeded end to end: the packaging tool itself downloaded through the new proxy, the project's own public dependencies resolved through it too, the internal packages came back from their original home exactly as before, and the final image landed in the registry clean.
Summary — Hints for Fellow Investigators
- An artifact proxy that looks universal usually isn't. A single URL and a familiar API (
simple/<package>/) tell you nothing about scope. Before repointing anything at a new index — public or private — test it with something you already know should and shouldn't be there. - 404 and 403 are different bugs wearing similar disguises. "Authorization error" can mean "you lack permission" or it can mean "there was never anything here to permit." Get real credentials into the loop early, so the error message you're debugging is the real one.
- Fetch the branch, not just the trunk. Rebasing onto an updated
mainis routine. Forgetting to also refresh the feature branch you're about to force-push onto is how you nearly overwrite a teammate's already-pushed work. Let a rejected push make you stop and look, rather than reaching for--force. - Make the boring failure mode self-healing. A dependency lock file silently drifting out of sync with its own manifest is a slow, recurring failure. Regenerating it as a build step, rather than trusting whatever was last committed, turns a class of future outages into a non-event.
- A tool that only does half the job you assumed is still worth keeping — for the half it's good at. The right fix here wasn't to abandon the new proxy; it was to scope it correctly and let it do the one thing it was actually built for.