How would you design a deploy pipeline so a killed CI job can never leave a Helm release ambiguously stuck?

Combine generous, graceful-shutdown-aware CI timeouts (so a kill is rare and, when it happens, gives Helm a chance to finish or clean up rather than being hard-terminated instantly) with a post-deploy verification step that explicitly checks release status and automatically clears a stuck pending-* state before the next deploy attempt — turning "someone has to notice and manually fix a stuck release" into "the pipeline itself detects and resolves the ambiguity automatically on the next run."

The goal is shifting stuck-state recovery from a manual, reactive process to something the pipeline itself handles automatically on its very next run, so a kill becomes a routine, self-correcting event instead of an incident.

• A killed CI job must not require manual intervention before the next deploy can proceed.
• The pipeline must distinguish a genuinely stuck release from one that's correctly mid-legitimate-operation.
• The fix must not silently paper over real deployment failures — only resolve the ambiguous state problem, not mask actual errors.

Generous, graceful-aware CI timeouts: set the CI job's timeout comfortably longer than a normal deploy's worst-case duration (based on observed data, not a guess), and prefer a graceful termination signal (SIGTERM with a grace period) over an immediate hard kill where the CI platform supports it — giving a genuinely slow-but-working helm upgrade --atomic room to either finish normally or complete its own automatic rollback, rather than being cut off mid-operation unnecessarily.

Pre-deploy status check as a standard pipeline step: before every helm upgrade, the pipeline runs helm status <release> (or helm history) and checks whether the current revision is in a pending-* state. This turns "is the release stuck" from a manual investigation into a routine, automated check that runs on every single deploy attempt, not just when someone happens to notice a problem.

Automatic resolution of a detected stuck state: if the pre-deploy check finds a pending-* state, the pipeline automatically resolves it before proceeding — either helm rollback to the last known-good revision (if one exists) or, if this is a first install with no prior good revision, programmatically correcting the stuck Secret's status (as covered in the original stuck-release resolution) — all scripted, requiring no human judgment call for the common case, only escalating to a human if the automatic resolution itself fails.

Idempotent, safe-to-retry pipeline design overall: the deploy pipeline as a whole should be safe to simply re-run after any failure, including a kill — which the pre-deploy check plus automatic resolution directly enables, since re-running the pipeline after a kill now starts by clearing any stuck state left behind, rather than immediately failing again on the same ambiguity.

Alerting on the resolution path itself, not just deploy failures: even though the pipeline resolves stuck states automatically, alert when it actually had to (a stuck state was detected and cleared) — this is a real signal that a kill happened and is worth understanding the cause of, even though the pipeline recovered without needing a human in the loop for the immediate fix.

Automatic resolution of stuck states adds real complexity to the deploy pipeline itself (the resolution logic needs to be genuinely correct, since a buggy automatic rollback could cause its own problems) — a team needs to weigh that complexity against the operational cost of manual stuck-release recovery happening periodically instead. Alerting on every auto-resolved stuck state, rather than only on true failures, risks some alert noise if kills happen frequently for unrelated reasons (e.g. genuinely too-tight timeouts) — worth tuning timeouts first, so the alert stays a meaningful, infrequent signal rather than routine noise.

• Generous, graceful-aware CI timeouts reduce how often a kill happens at all, and give in-flight operations a chance to complete or self-correct.
• A pre-deploy status check, run automatically on every deploy attempt, turns stuck-state detection from manual investigation into a routine pipeline step.
• Automatic resolution of a detected stuck state (rollback, or corrected Secret status) makes the pipeline safe to simply re-run after any failure, including a kill.
• Alerting specifically on "the pipeline had to auto-resolve a stuck state" surfaces the underlying kill events worth investigating, even though they no longer require manual recovery.

• Helm CLI: helm status
• Helm CLI: helm rollback