Timeouts, Retries, Backoff & Jitter

The self-inflicted DDoS

Level 1's resilience-primitives lesson introduced the four call-policy parts (timeouts, retries, circuit breakers, isolation) as a first pass; this lesson is the deep walkthrough of the first two. The single most common way a distributed system takes itself down is not a hardware failure. It is a small blip amplified by its own retry logic into a self-inflicted DDoS, and this lesson is the defense.

Every call needs a timeout

A call with no timeout inherits the operating system default, which for a TCP connect can be minutes. When a downstream slows down, requests that would have failed fast instead pile up, each holding a thread and a connection. Your thread pool fills, and now a service that was merely slow makes your service completely unavailable. You need two timeouts: a connect timeout (how long to wait to establish the connection, usually tens of ms inside a datacenter) and a request timeout (how long to wait for the response). Set the request timeout from the downstream's real p99, not a guess. If the downstream's p99 is 80 ms, a 250 ms timeout is generous; a 30 second timeout is a liability.

Propagate a deadline, do not reset it

If the user-facing request has a 1 second budget and it has already spent 400 ms, the call to service B must be told "you have 600 ms left," and service B must pass the remaining budget to service C. gRPC does this natively with deadlines; in HTTP you pass a header like X-Deadline or grpc-timeout. Without propagation each hop uses its own fresh timeout, so a 3-hop chain can legally spend 3x the user's budget doing work whose result the user already gave up waiting for.

Retries need backoff, jitter, and a budget

Retrying immediately after a failure is how a blip becomes an outage: the downstream chokes, every caller retries at once, and the synchronized wave of retries keeps it choked. The formula is exponential backoff with jitter:

  delay = random_between(0, min(cap, base * 2^attempt))

The exponential part (base * 2^attempt) spaces retries further apart as failures persist. The cap bounds the worst-case wait. The jitter (randomizing within the window) is the part juniors omit and the part that matters most: without it, a thousand clients that failed at the same instant all retry at the same instant, recreating the thundering herd. AWS's published guidance is full jitter, exactly the form above.

Cap total retries with a retry budget. Even with backoff, blind retries multiply load. Level 1 introduced this guard as the rough "retries stay under ~10% of request volume"; in production you enforce it as a live ratio, for example retries may not exceed 5% of successful requests over a rolling 10-second window, tightened further for expensive downstreams. When the downstream is broadly failing, the budget exhausts and you stop retrying, which is correct: retrying a dead dependency just delays recovery.

Only retry idempotent operations. A GET is safe. A POST that charges a card is not: a timeout does not tell you whether the charge happened, so a naive retry can double-charge. Make writes safe to retry with an idempotency key the server dedupes on.

Interview nuance: the killer is retry amplification. If the gateway retries 3x, and it calls a service that also retries 3x, and that service calls a database client that also retries 3x, one user request can become 27 database calls. Retry at exactly one layer, usually the outermost one that owns the deadline, and let inner layers fail fast.

Recap: connect plus request timeouts on every call, propagate a shrinking deadline down the chain, exponential backoff with full jitter, a retry budget capping retries to a small fraction of traffic, retry only idempotent operations, and retry at one layer to avoid multiplicative amplification.