The safe command that deleted the secret

A short detective story from the world of GitOps and Kubernetes secrets.

The last step

The migration was almost over. For two days the Company's clusters had been moving every secret off Sealed Secrets and onto AWS Secrets Manager, read by the External Secrets Operator. One service at a time, the same careful loop each time:

create the secret in AWS,
copy the live value in,
sync it into a parallel Kubernetes Secret with an -sm suffix,
prove the copy is byte-identical (by hash, never printing a value),
only then switch the workload over.

Eleven secrets had gone through that loop without a hiccup. One was left: ArgoCD's own argocd-secret, which holds the admin password, the SSO client, the key that signs every session and API token, and the TLS certificate. It was still owned by a SealedSecret, and the SealedSecret had to go before the Sealed Secrets controller could be removed.

The plan for this one step was a single, well-documented command, run once per cluster:

kubectl -n argocd delete sealedsecret argocd-secret --cascade=orphan

--cascade=orphan is the textbook way to delete an owner and keep what it owns. Kubernetes strips the ownerReference from the child, then removes the parent. The Secret stays, untouched, just unowned. It's the "safe" flag.

The platform engineer ran it on both clusters, five seconds apart, and reported back: "detached on both, check argocd-secret."

The check that came back empty

The first verification script printed nothing useful:

error: context "get" does not exist

That one was self-inflicted. A small shell helper read $1 inside a function, so it picked up the function's own first argument instead of the cluster name. Fixed, rerun, and this time the output was worse:

Error from server (NotFound): secrets "argocd-secret" not found

On both clusters. The secret the safe command was supposed to keep was gone.

ArgoCD's server log agreed:

error retrieving argocd-secret: secret "argocd-secret" not found

Applications were still green: the controllers run on what they already hold in memory. But every API call was now failing, and the next restart could make ArgoCD generate a fresh signing key. That would invalidate every session and every CI token.

Restore first, understand second

The loop from earlier paid off here. One step before, the External Secrets Operator had written a full parallel copy, argocd-secret-sm, and it had already been verified hash-identical to the original. Restoring was one pipe, the copy's data poured into a new Secret with the original name and labels, without a value ever touching the screen:

kubectl get secret argocd-secret-sm -o json \
  | jq '{apiVersion:"v1", kind:"Secret", type:"Opaque",
         metadata:{name:"argocd-secret", namespace:"argocd",
                   labels:{"app.kubernetes.io/part-of":"argocd"}},
         data:.data}' \
  | kubectl create -f -

Hashes matched the pre-detach backup. ArgoCD logged Loading TLS configuration from secret and the errors stopped. The gap was a bit over two minutes per cluster. Nobody had to log in again, because the signing key had never changed, and no CI job happened to run in that window.

Now for the why.

The obvious suspect

There was a perfect suspect. The very next change in the plan was a pull request that pointed the External Secrets Operator at argocd-secret itself, merging a few keys into it. When the engineer asked about it, it turned out that PR had already been merged, nine seconds after the second detach command.

A change that targets a Secret, merged right when that Secret disappeared, looks guilty. So the timeline got built properly, from three independent clocks:

T+0s    detach command, cluster A         (shell history)
T+1s    controller A: "Updating argocd/argocd-secret"
T+5s    detach command, cluster B
T+5s    controller B: "Updating argocd/argocd-secret"
T+14s   the suspicious PR is merged        (Git host)
T+17s   first "argocd-secret not found"    (ArgoCD server log)
T+148s  Secret restored by hand
T+245s  the PR is finally synced to the clusters (GitOps controller history)

The PR was innocent. It reached the clusters after the restore, found a Secret that existed again, and merged into it exactly as designed. The real clue sat at T+1s and T+5s. In the same second as each delete, the Sealed Secrets controller had written to the Secret.

What actually happened

--cascade=orphan isn't one atomic operation. Kubernetes:

marks the SealedSecret for deletion and attaches an orphan finalizer,
strips the ownerReference from the Secret,
lets the SealedSecret go.

Step 1 is an update to the SealedSecret. The controller watching SealedSecrets (an older release here) doesn't skip objects that are being deleted. It saw an update, did what it always does, unsealed the data, and rewrote the Secret, with its ownerReference put back.

   you                 API server               Sealed Secrets ctrl      garbage collector
    |  delete --cascade=orphan |                          |                      |
    |------------------------->| mark SealedSecret        |                      |
    |                          | (deletion + finalizer)   |                      |
    |                          |---- update event ------->|                      |
    |                          | strip ownerRef on Secret |                      |
    |                          |<-- rewrite Secret -------|                      |
    |                          |    (ownerRef is back)    |                      |
    |                          | remove SealedSecret      |                      |
    |                          |-------------------------------- owner gone? --->|
    |                          |<---------------------------- delete Secret -----|

When the SealedSecret was gone, the garbage collector found a Secret pointing at an owner that no longer existed, and did its job. The controller's own next log line confirms the order: its status update failed with SealedSecret "argocd-secret" not found.

The command was correct and the flag was correct. The controller raced it.

The mistake underneath

The uncomfortable part is that the migration had a habit that would have caught this. Every External Secrets Operator behaviour that mattered was proven on a throwaway object first: does it adopt an existing Secret, does it drop extra keys, does deleting it keep the merged keys? Each one got a test Secret on the non-production cluster, a check, and a cleanup.

The detach step skipped that. It looked like plain Kubernetes, and plain Kubernetes is well-understood. But it wasn't plain Kubernetes: there was a controller in the loop, and controllers react to events you don't think of as events.

The fix for next time is one line, run before touching anything a controller owns:

kubectl -n <ns> scale deploy sealed-secrets-controller --replicas=0

With nothing listening, the orphan deletion behaves exactly as documented. That's how the controller itself was later removed, without incident.

Hints for the reader

"Orphan" protects you from Kubernetes, not from controllers. Any controller that reconciles the owner can rewrite the child mid-deletion. Stop the controller first.
Keep a verified copy before you touch anything critical. The restore took one pipe because a hash-identical copy already existed. Without it, the same two minutes would have been a credential rotation for every user and every CI job.
Build the timeline from independent clocks before you blame the nearest change. Shell history, controller logs, the Git host and the GitOps sync history together turned "the PR did it" into "the PR arrived after the fix".
Apply your testing habit to the "obvious" steps too. The steps that looked risky all got a throwaway-object test. The step that looked obvious didn't, and that was the one that bit.
Green apps don't mean a healthy control plane. The workloads stayed Synced throughout, while every API call to the GitOps server failed.

This is a lightly fictionalized account based on a real infrastructure session. Names, systems, and identifying details have been generalized.