How would you tune a hedge-request delay so it targets genuine cold-start tail latency without firing on every normal request?

Set the hedge delay based on a high percentile (commonly p95 or p99) of the service's normal, warm-request latency distribution — a delay set there means the hedge almost never fires for a normal request (since by definition only the slowest 1-5% of normal requests exceed that threshold), while it fires promptly once a request is taking meaningfully longer than typical, which is exactly the profile of a cold start.

Anchor the delay to the observed normal-latency distribution, not a guess: the hedge delay's entire purpose is distinguishing "this specific request is taking unusually long" from "this is just normal request latency" — that distinction only makes sense relative to what normal actually looks like for this specific service, measured from real production data (or realistic load testing), not an arbitrary round number picked without reference to actual behavior.

p95 or p99 of warm-request latency is the standard anchor point: setting the hedge delay at the p95 (or p99, for a more conservative/less-frequently-firing choice) of the service's warm request latency means, by construction, only the slowest 5% (or 1%) of otherwise-normal requests would trigger a hedge unnecessarily — a small, controlled, and predictable rate of "unnecessary" hedges, versus a threshold set too low (closer to median latency) that would fire on a much larger fraction of entirely normal requests, defeating the point of targeting the rare tail specifically.

Distinguish "warm" latency specifically, not overall latency including cold starts: since cold starts are exactly what you're trying to hedge against, computing the percentile from overall latency (cold starts included) would inflate the threshold, making the hedge fire less often than intended, including missing some genuine cold starts that fall just under an inflated threshold — the percentile calculation should be scoped to warm-request latency specifically, requiring the ability to distinguish cold from warm requests in your latency data (via a cold-start indicator in logs/metrics, which many Lambda monitoring setups can provide).

Account for the specific traffic pattern's variability: a service with highly consistent, low-variance warm latency can use a tighter threshold (closer to p95) confidently; a service with more naturally variable warm latency (due to varying payload sizes, downstream dependency variability) might need a more conservative p99 threshold to avoid firing on legitimate-but-slower-than-typical normal requests, since a tighter threshold on a more variable distribution would catch more false positives.

Revisit the threshold periodically as the service's latency profile evolves: a threshold tuned against today's warm-latency distribution can become miscalibrated as the service's actual behavior changes (a code change affecting typical latency, a change in typical payload size) — treating this as a value to periodically re-measure and re-tune, not a one-time setting, keeps the hedge targeting genuinely accurate over the service's lifetime.

• Anchor the hedge delay to a high percentile (p95/p99) of the service's own warm-request latency distribution, measured from real data, not an arbitrary guess.
• p95/p99 thresholds mean only a small, predictable fraction of normal requests trigger an unnecessary hedge, by construction.
• Compute the percentile from warm-request latency specifically, excluding cold starts, or the threshold gets inflated and misses genuine cold-start cases.
• Revisit and re-tune the threshold periodically as the service's actual latency profile evolves over time.

• AWS Builders' Library: Timeouts, retries, and backoff with jitter
• AWS: Understanding AWS Lambda cold starts