How would you set up alerting to catch a CrashLoopBackOff-class issue before it reaches production traffic, rather than discovering it via a user-facing outage?

Catch it at two points before it ever affects real traffic: a deploy-time gate (the rollout itself waits for new Pods to actually become Ready before considering the deploy successful, automatically failing/rolling back if they don't) and a fast, dedicated alert on restart-count/crash-loop metrics scoped to catch it within the first minute or two of a rollout, well before a slower general "error rate is elevated" alert would fire.

The two layers below catch this at different points — one during the deploy itself, before traffic meaningfully shifts, the other as a fast independent backstop in case the first layer is bypassed or misconfigured.

• The deploy process itself should refuse to consider a rollout successful if new Pods aren't actually becoming healthy.
• Alerting must be fast enough to catch the problem during the rollout window, not just eventually via general health monitoring.
• The signal must specifically and unambiguously indicate crash-looping, not be conflated with normal, expected restart activity.

Deploy-time readiness gating: a Kubernetes Deployment's own rollout mechanics already provide a first line of defense if configured and used correctly — kubectl rollout status (or the CI/CD pipeline's equivalent wait-for-rollout step) blocks the deploy pipeline from marking the deployment successful until the new Pods are actually Ready, and a deploy pipeline that treats a rollout timeout as a failure (triggering automatic rollback via kubectl rollout undo, or simply failing the pipeline and paging someone) catches a crash-looping new version before traffic is ever meaningfully shifted to it, assuming the rollout strategy (maxUnavailable/maxSurge) doesn't cut over instantly.

Fast, dedicated crash-loop alerting: a Prometheus alert specifically on kube_pod_container_status_restarts_total (or the rate of restarts) scoped to a short window and low threshold — e.g. alerting if a container restarts more than twice within 5 minutes — catches crash-looping quickly, independent of whether the deploy pipeline's own gating catches it first. This is deliberately a fast, narrow, high-confidence signal (crash-looping is essentially always worth immediate attention) rather than a broad "something might be wrong" alert requiring investigation to even confirm relevance.

Scoping the alert to avoid false positives from expected restarts: some restart activity is normal and shouldn't page anyone — a single restart from a transient issue, or expected restarts during a node drain/maintenance event. Scoping the alert specifically to a rate of restarts within a short window (not just "any restart happened") and excluding known-expected restart sources (e.g. suppressing during an active, deliberate rolling update if that's expected to cause brief restart activity) keeps the alert meaningfully signal, not noisy.

Canary or progressive rollout as a further layer: for higher-stakes deployments, combining this with a canary/progressive rollout strategy (small percentage of traffic first, validated before full rollout) adds another layer specifically limiting the blast radius even if crash-looping isn't caught until some traffic has already reached the bad version — a smaller, contained group of affected users/requests rather than the full production fleet.

Aggressive deploy-time gating (failing the pipeline on any rollout timeout) can occasionally block a legitimately slow-but-healthy startup if the timeout is tuned too tight — worth basing the timeout on observed normal startup time with margin, not an arbitrary guess. Fast restart-rate alerting needs careful threshold tuning to avoid paging on genuinely benign restart patterns; too loose a threshold misses real crash-loops, too tight generates alert fatigue.

• Deploy-time readiness gating (failing the pipeline if new Pods don't become Ready, with automatic rollback) is the first line of defense, catching problems before traffic meaningfully shifts.
• A fast, narrowly-scoped restart-rate alert independent of the deploy pipeline catches crash-looping even if deploy-time gating is bypassed or misconfigured.
• Scoping the alert to a rate within a short window, and excluding known-expected restart sources, keeps it a high-confidence signal rather than noise.
• Canary/progressive rollout adds a further blast-radius-limiting layer even if the earlier detection layers don't catch the problem immediately.

• Kubernetes Docs: kubectl rollout status
• Kubernetes Docs: Deployments — Rollout status
• Prometheus: kube-state-metrics