>DevOps Interview KB

A helm rollback succeeds but the application still behaves like the newer version — why might rollback not fully revert the deployed state?

AdvancedTroubleshootingHelm7 min read

Short Answer

helm rollback reverts the Kubernetes resources Helm manages back to their state at the target revision — but it doesn't undo anything that happened outside Helm's own tracked resources, most commonly a database migration that already ran (and isn't reversed just because the application code rolled back), a change to an external system, or state cached/persisted somewhere Helm has no visibility into or control over.

Detailed Explanation

Helm's rollback mechanism operates entirely on the Kubernetes manifests it manages as part of the release — it has no awareness of, and no ability to revert, any side effect the previous release caused outside of that manifest state, which is exactly the gap that produces this "rollback succeeded but behavior didn't actually change" symptom.

Symptoms

  • helm rollback completes without error, and helm history shows the new rollback revision.
  • kubectl get pods shows the application pods running the older image tag, as expected from the rollback.
  • The application's actual observed behavior (the bug, the data, the API responses) still reflects the newer version's behavior, not the older one's.

Possible Causes

  • A database migration ran as part of the newer release (via a pre-upgrade/post-upgrade hook, or a separate step) that changed the schema or data in a way the older application code isn't compatible with or doesn't reverse — the schema/data change persists regardless of which application version is now running.
  • The newer release changed something in an external system (a third-party API configuration, a feature flag in an external service) that Helm has no knowledge of and no ability to touch.
  • The rollback reverted the Deployment's pod template, but a ConfigMap or Secret it depends on was not part of the same Helm release (managed separately, or by a different tool) and still holds the newer configuration.
  • Cached data (in the application itself, in a CDN, in a client-side cache) is still reflecting the newer version's output, even though the actual running code has genuinely reverted.

Investigation Steps

Confirm what actually changed at the Kubernetes resource level first: kubectl get deployment <name> -o yaml — verify the image tag and other spec fields genuinely reflect the older revision's values, confirming the rollback did what it was supposed to at the resource level before looking elsewhere; if even this didn't revert, the investigation shifts to why the rollback itself didn't apply correctly, a different problem.

Check whether the problematic release included a database migration or other external-system change: reviewing the newer release's hooks (helm get hooks <release> for historical hook definitions, or the chart's source) and any accompanying runbook/change notes for a migration step — if one ran, that's very likely the actual cause, since schema/data changes are exactly the kind of side effect Helm's rollback has no mechanism to reverse.

Check whether all the release's actual dependencies are genuinely managed by the same Helm release: helm get manifest <release> shows exactly which resources this specific release actually manages — a ConfigMap or Secret referenced by the Deployment but created/managed outside this Helm release (a separate kubectl apply, a different release, an External Secrets Operator-managed object) wouldn't be reverted by helm rollback at all, since it was never part of what Helm was tracking for this release in the first place.

Rule out caching as the explanation before assuming the rollback itself is incomplete: testing directly against a pod (bypassing any CDN/cache layer) confirms whether the underlying application is genuinely still running old behavior, or whether the rollback actually succeeded and a caching layer is just serving stale responses — these require completely different remediation.

Resolution

If a database migration is the cause, the fix is a separate, deliberate decision (writing and running a compensating down-migration, or determining the data change is actually safe to leave in place and instead fixing forward) — Helm rollback cannot and should not be relied upon to undo this; it needs a genuinely reversible migration strategy designed for that specific concern. If an out-of-release dependency (ConfigMap/Secret managed elsewhere) is the cause, revert that resource through whatever process actually manages it. If it's caching, address the cache layer directly (invalidation, TTL). Confirm the fix by testing the actual application behavior directly, not just checking that helm history shows a rollback revision.

Interview Follow-Up Questions

  • How would you design a migration strategy specifically to make rollback genuinely safe, given Helm rollback can't undo a schema change on its own?
  • How would you audit a Helm release to confirm exactly which Kubernetes resources it actually manages, versus resources it depends on but doesn't own?
  • How would you communicate to a team, before an urgent rollback, which specific risks (like an unreversed migration) the rollback won't actually address?

Key Takeaways

  • helm rollback only reverts the Kubernetes resources tracked within that specific Helm release — it has no ability to reverse anything outside that scope.
  • A database migration that ran during the problematic release is the most common cause of "rollback succeeded but behavior didn't revert" — schema/data changes aren't undone by rolling back application code.
  • Resources referenced by the release but managed outside of it (a separately-applied ConfigMap/Secret) won't be reverted by helm rollback at all.
  • Verify actual application behavior directly (bypassing caches) before concluding the rollback itself was incomplete, since caching can produce a similar-looking symptom with a completely different cause.

References

Last updated August 22, 2026 · Last reviewed August 22, 2026