Retry, backoff and the token budget

The loop around the call is yours to write

A model API is one HTTP call, and the client library makes it look like a function call. It is not one. It fails several ways, it costs money per attempt, it is slow enough that the user is watching, and none of that is the library's problem. The loop around it is yours, and it is the piece of AI glue code that is most often written wrong. Every fact in this lesson is ordinary distributed-systems hygiene; what makes it worth its own lesson is that the per-attempt cost is money rather than milliseconds, so the mistakes are expensive in a way a normal retry loop's are not.

Two kinds of failure, and only one is worth repeating

A failed call is either retryable (the same request, sent again, might work) or terminal (it will fail identically forever). Rate limiting, an overloaded server, and a network timeout are retryable. A malformed request, a bad key, and a missing model are terminal. The distinction is not decoration: retrying a terminal error costs a full attempt, in tokens and in seconds, and buys nothing.

class ModelError(Exception):
    def __init__(self, kind, tokens):
        super().__init__(kind)
        self.kind = kind          # "overloaded", "timeout", "bad_request", ...
        self.tokens = tokens      # a failed attempt still burned tokens


TERMINAL = ("bad_request", "unauthorized")

try:
    reply = model.send(prompt)
except ModelError as exc:
    print(exc.kind, exc.tokens)        # unauthorized 0
    if exc.kind in TERMINAL:
        raise

An exception can carry data. exc.kind and exc.tokens are ordinary attributes set in __init__, and reading them is how the handler decides anything at all.

Backoff, and why every client waits a different amount

Waiting a fixed two seconds between attempts is worse than it looks.

Every client that failed at the same moment retries at the same moment, so the overloaded service gets the identical spike it just shed, on a two-second cycle, forever. Exponential backoff spreads the attempts out in time (0.5, 1, 2, 4, ...), and a cap stops the growth before a retry outlives the request that wanted it.

The repair is full jitter: do not wait the ceiling, wait a random amount between zero and the ceiling. The ceiling still doubles; the actual waits scatter, so two clients that failed together almost never retry together.

import random

BASE_DELAY = 0.5
MAX_DELAY = 8.0
rng = random.Random(7)          # seeded, so a test can reproduce the sequence

for attempt in range(5):
    ceiling = min(MAX_DELAY, BASE_DELAY * 2 ** attempt)
    print(attempt, round(ceiling, 2), round(rng.uniform(0.0, ceiling), 3))

# 0 0.5 0.162
# 1 1.0 0.151
# 2 2.0 1.302
# 3 4.0 0.29
# 4 8.0 4.287

random.Random(seed) is a generator you own, so seeding it makes a run reproducible without touching the global random state that the rest of the process shares. rng.uniform(a, b) returns a float between a and b.

A budget is not an attempt count

"Retry up to five times" is the bound almost every retry loop ships with, and in most of them it is the only bound in the file.

Nobody is billed for attempts. The two things that actually run out are tokens (money) and wall-clock seconds (the user waiting, or the deadline of whatever called you). Track those directly, and when one is gone, stop and return what you have.

Aborting to a partial result is a design decision, not a failure to handle the error. A caller that gets back {"status": "partial", "text": "", "tokens": 4200} can decide to degrade, queue the work, or tell the user. A caller that gets an exception thrown from inside a loop learns nothing about what was spent.

Reading the reply before you trust it

A call that returns is not a call that succeeded. When the model is choosing a tool, the reply is a structured object, and a model can produce one that parses as JSON but is missing a field your code then indexes into. Checking the shape before acting on it costs four lines:

REQUIRED = {"name": str, "args": dict}


def valid_tool_call(reply):
    return all(
        field in reply and isinstance(reply[field], kind) for field, kind in REQUIRED.items()
    )


print(valid_tool_call({"name": "search", "args": {"q": "cache"}}))   # True
print(valid_tool_call({"name": "search"}))                           # False
print(valid_tool_call({"name": "search", "args": "cache"}))          # False

The interesting part is which bucket that failure goes in. A reply that fails the shape check is retryable, because asking again may well produce a well-formed one, and it is the one retryable failure where sending the identical request is the wrong move: you would usually add the validation error to the prompt. A request the API itself rejected is terminal. Same loop, opposite verdicts, and telling them apart is the whole job of the handler.

An idempotency key is what makes a retry safe

Everything above assumes a retry is free to send. It is free only when the call has no side effect, and a call that charges a card, sends an email, or writes a row has one.

A timeout tells you the response was lost, not that the request was, so a side-effecting call that is retried on one can do the thing twice.

An idempotency key is a string the client sends with the request, identifying the operation. The server stores the result under that key, so a second request carrying the same key returns the first result instead of doing the work again. Two rules make it work, and both are easy to break:

• It is derived from what the operation is, never from the attempt. Anything that changes between attempts (an attempt counter, a timestamp, a fresh random id) turns every retry into a new operation.
• It is stable across processes. json.dumps with sort_keys=True gives one string for one set of fields, whatever order the dict happened to be built in.

import json

KEY_FIELDS = ("account", "cents", "reason")


def idempotency_key(payload):
    return "refund:" + json.dumps(
        {field: payload[field] for field in KEY_FIELDS}, sort_keys=True
    )


first = {"account": "a1", "cents": 500, "reason": "duplicate charge", "attempt": 1}
again = {"reason": "duplicate charge", "cents": 500, "account": "a1", "attempt": 4}
print(idempotency_key(first) == idempotency_key(again))   # True
print(idempotency_key(first))
# refund:{"account": "a1", "cents": 500, "reason": "duplicate charge"}

The server side of that bargain is not always available to you, and it is not the whole story anyway. A job that restarts from the top will re-send work it already finished, and the gateway will happily answer from its own store, but only if it still has the key. Keeping your own ledger of settled keys means the restart never sends the request at all, which is cheaper and does not depend on how long somebody else's cache lives.

Pitfalls

• Retrying on a bare except Exception retries the terminal failures too. Name the kinds you mean.
• A cap with no jitter is still synchronized. The randomness is the point, not the doubling.
• random.random() uses the process-wide generator, so seeding it for a test changes behavior for every other caller in the process. Own an instance with random.Random(seed).
• A budget checked only at the top of the loop is not a budget. Check it after the attempt that spent the tokens.
• Storing a key after the response arrives leaves a window where the work is done and unrecorded. That window is why the server keeps its own store: yours is an optimization, not the guarantee.

Interview nuance: the follow-up on this is almost always "and what if the retry storm is the thing taking the service down?" The answer is a circuit breaker, which sits above the retry loop rather than inside it: after enough consecutive failures it stops sending anything for a cooling-off period, so a dead dependency gets no traffic at all instead of every client's full retry budget. Retries protect one request from bad luck. A breaker protects the dependency from every request at once. Being able to say that they solve different problems, and that a retry loop without one turns a partial outage into a total one, is the answer being probed for.

Sources: Timeouts, retries and backoff with jitter · random.uniform · Idempotent requests