Even after mitigating cold starts, a small amount of irreducible tail latency remains. How would you design the caller's retry behavior to handle that remaining tail?

Use bounded retries with exponential backoff and jitter as the default, tuned against the service's actual tail latency (not median), combined with a circuit breaker so retry traffic doesn't compound a real degradation into a worse one; for genuinely latency-critical, idempotent calls, add hedged requests (firing a second attempt after a short delay) to directly target the tail rather than just reacting to it after a timeout.

The design has to separate two different failure modes that look similar from the caller's side but need different responses: an isolated slow request (this one call happened to hit a cold start) and genuine service degradation (the service itself is struggling and more retries would make it worse). The architecture below addresses both.

• Retries must not make the tail-latency problem worse by adding retry storms during a real degradation.
• The retry strategy must distinguish "this specific request was just unlucky" from "the service is actually struggling," which need different responses.
• Idempotency of the underlying operation must be accounted for, since blind retries on non-idempotent operations can cause real damage.

Timeout tuned to the tail, not the median: set the caller's own request timeout based on the service's actual tail latency distribution (e.g. p99 plus margin), not the median — a timeout set too close to the median latency triggers unnecessary retries constantly, adding load without helping actual outliers.

Bounded retry with backoff: retry a failed/timed-out request a small, fixed number of times (commonly 1-3) with exponential backoff and jitter — backoff avoids hammering an already-struggling service, and jitter avoids many callers retrying in lockstep and creating a synchronized load spike right when the service is least able to handle one.

Hedged requests for latency-critical paths: for genuinely latency-sensitive calls, a hedging pattern — firing a second request after a short delay if the first hasn't responded yet, and using whichever completes first — directly targets tail latency specifically, since the odds of both the original and the hedge request hitting a slow cold start simultaneously are much lower than either alone. This trades some extra load for meaningfully better p99, and is only appropriate when the operation is safely idempotent and the extra invocation cost is acceptable.

Idempotency awareness before any retry at all: an operation must be safe to execute more than once before blind retries are appropriate — for genuinely non-idempotent operations, either make them idempotent (an idempotency key the downstream system deduplicates on) before adding retries, or don't retry blindly and instead surface the ambiguous outcome for explicit handling.

Circuit breaking as the escalation path: bounded per-request retries handle isolated tail-latency events; a circuit breaker (tracking recent failure/timeout rate and short-circuiting further calls once a threshold is crossed) handles the different case where the service is genuinely degraded, preventing retry traffic from compounding a real outage into a worse one.

Hedged requests directly improve tail latency at the cost of extra invocations and cost, and only work safely for idempotent operations. Aggressive retry counts reduce individual-request failure rates but risk amplifying load during genuine degradation if not paired with backoff, jitter, and a circuit breaker. Conservative retry policies (fewer retries, higher timeouts) are safer under real degradation but leave more of the irreducible tail latency visible to end users. The right balance depends on how latency-sensitive the specific call path actually is and how much the operation's idempotency allows for safe retries.

• Tune caller timeouts to the tail of the latency distribution, not the median, to avoid triggering unnecessary retries.
• Bounded retries with exponential backoff and jitter handle isolated tail-latency events without amplifying load during real degradation.
• Hedged requests directly target tail latency for idempotent, latency-critical calls, at the cost of extra invocations.
• A circuit breaker is the necessary escalation path distinguishing "isolated slow request" from "the service is genuinely struggling," which need fundamentally different responses.

• AWS: Lambda retry behavior
• AWS Builders' Library: Timeouts, retries, and backoff with jitter
• AWS: Lambda function idempotency