Design a Reliable Webhook Delivery System

The receivers are outside your control

A webhook delivery system notifies customer-controlled endpoints when events happen (Stripe firing payment.succeeded to your server). The hard part is that the receivers are outside your control: they are slow, flaky, sometimes down for hours, and occasionally malicious. The interview tests your delivery guarantee, retry and backoff strategy, payload signing, idempotency and ordering, dead-letter handling, and per-tenant fairness.

At-least-once plus consumer idempotency

Offer at-least-once. Persist every event first, enqueue a delivery task, and mark it delivered only when the endpoint returns a 2xx. If you crash after sending but before recording success, you redeliver, so duplicates are possible. This is the honest, standard guarantee; exactly-once delivery to an arbitrary external endpoint is not achievable, so you push idempotency to the consumer. Include a stable, unique event id in every payload (and an idempotency header) and document that delivery is at-least-once, so consumers dedupe on the id. Stripe, GitHub, and Shopify all do exactly this.

Interview nuance: the single most important architectural point: never deliver inline and synchronously from the event producer. If your checkout service calls the customer's webhook URL directly in the request path, a slow or hung customer endpoint backs up your producer and can stall the whole pipeline. Always persist the event and hand delivery to a separate, queue-driven delivery service.

Retries, signing, ordering

On a failure (non-2xx, timeout, connection error) retry with exponential backoff plus jitter over a long window: seconds, then minutes, then hours, up to a day or more, with a capped attempt count. Backoff lets a down endpoint recover without a thundering herd, and jitter prevents all retries for a mass event from firing in lockstep. Use a per-attempt timeout (a few seconds) so a hung endpoint does not tie up a worker.

Sign each payload so the consumer can verify it really came from you and was not tampered with. Compute an HMAC-SHA256 over the raw body plus a timestamp using a per-customer secret, and send it in a header. The consumer recomputes and compares. Include the timestamp and reject old ones to prevent replay attacks, and support secret rotation with an overlap window.

Default to no strict global order because it is simpler and lets you deliver in parallel. When a tenant genuinely needs per-resource order, key delivery by resource id and deliver sequentially per key, holding back the next event for a key until the prior one is acknowledged. This costs throughput for that key, so make it opt-in.

Dead-letter and fairness

After the max attempts, move the event to a dead-letter store, alert, and expose a manual replay or redrive API. Fairness is critical because endpoints vary wildly: isolate delivery per tenant with per-tenant queues (or a fair scheduler), per-tenant concurrency limits and rate limits, per-endpoint timeouts, and circuit breakers that stop hammering an endpoint that has been failing, so one slow or dead customer cannot consume all workers and starve everyone else.

event -> persist -> enqueue delivery task (per-tenant)
worker: POST endpoint (HMAC-signed, timeout)
   2xx -> mark delivered ; non-2xx/timeout -> backoff+jitter retry (cap N)
   exhausted -> dead-letter + alert + manual redrive
circuit breaker + per-tenant concurrency -> one bad tenant cannot starve others

Recap: guarantee at-least-once (persist, enqueue, ack on 2xx) with a stable event id so consumers dedupe, deliver from a separate queue-driven service (never inline), retry with exponential backoff plus jitter over a long window, sign payloads with HMAC-SHA256 plus timestamp and rotate secrets, make ordering opt-in per resource key, and protect everyone with dead-letters plus per-tenant isolation and circuit breakers.