How would you design the metrics and thresholds that gate automatic canary promotion versus automatic rollback?

Pick a small set of metrics that directly reflect user-facing health (error rate and latency percentiles, not just infrastructure metrics like CPU), compare the canary's values against the stable baseline's values over the same time window rather than against a fixed absolute threshold, and require the comparison to hold for a minimum sustained duration and traffic volume before promoting — a single bad data point or a tiny sample size shouldn't trigger a decision either way.

The design challenge is building a decision rule that's sensitive enough to catch a genuinely bad canary but not so sensitive it rolls back on ordinary noise — the requirements and architecture below spell out how each piece of that balance actually gets implemented.

• The decision must be based on user-facing impact, not just infrastructure-level signals that might not reflect real problems.
• The comparison must account for normal variance, not treat every fluctuation as a signal.
• The system must have a clear, automatic fallback (rollback) when the data doesn't support promotion, rather than hanging indefinitely.

Metric selection: error rate (5xx responses, or application-specific failure signals) and latency percentiles (p95/p99, not just average, since averages hide tail degradation) are the standard core set — they directly reflect what users experience. Business-specific metrics (checkout completion rate, a critical workflow's success rate) are worth adding when available, since infrastructure health doesn't always correlate with actual user-facing correctness.

Relative comparison against a live baseline: rather than a fixed absolute threshold ("error rate must be under 1%"), compare the canary's metrics against the stable version's metrics over the same time window, since both are subject to the same external conditions (a traffic spike, a downstream dependency having a bad moment) — a canary that's statistically similar to the baseline, even if both are somewhat elevated, is a different signal than a canary that's meaningfully worse than a baseline that's otherwise normal. This is what tools like Flagger or Argo Rollouts' analysis templates are built around: a canary-vs-baseline comparison, not an isolated absolute check.

Statistical significance and minimum sample size: require a minimum number of requests and a minimum observation duration before evaluating the comparison at all — a canary receiving 1% of traffic needs enough absolute request volume for its error rate to be a meaningful signal rather than noise from a small sample (5 failed requests out of 50 looks alarming; 5 failed requests out of 50,000 might be entirely normal background rate).

Automatic, unambiguous fallback: if the metrics don't clearly support promotion within a bounded evaluation window, the default action should be rollback, not "wait indefinitely" or "promote by default" — an automated system needs a decisive default in the ambiguous case, and rolling back on ambiguity is the safer default than promoting on ambiguity.

Tighter thresholds (requiring the canary to be very close to baseline) catch more real problems but risk more false-positive rollbacks from normal variance, slowing down legitimate deployments. Looser thresholds deploy faster with fewer false rollbacks but risk letting a genuinely bad canary through. Requiring a longer observation window and larger sample size before deciding gives a more statistically confident signal but slows down the overall deployment pipeline — a real cost when deployment velocity matters. There's no threshold setting that's simply "correct"; it's a deliberate trade-off tuned to the specific system's traffic volume and risk tolerance.

• Base promotion/rollback decisions on user-facing metrics (error rate, latency percentiles) compared relatively against a live baseline, not fixed absolute thresholds.
• Require a minimum sample size and observation duration so decisions reflect a real signal, not noise from a small canary traffic share.
• Default to rollback, not indefinite waiting or automatic promotion, when the evaluation window ends without a clear signal.
• Threshold tightness is a genuine trade-off between false-positive rollbacks and letting a real problem through — tune it to the system's actual risk tolerance and traffic volume.

• Flagger: Canary deployments
• Argo Rollouts: Analysis