A model call is a network call

What the call actually is

Strip away the interesting part and a model call is an HTTP request to somebody else's server. Everything you already know about third-party dependencies applies, and the parts people forget are exactly the parts they forget for payment gateways and geocoding APIs:

• It is slow, in seconds rather than milliseconds, and the variance is large.
• It costs money per call, roughly in proportion to the text going in and coming out.
• It fails, with timeouts, rate limits, overload responses, and transport errors.
• It returns text, which is not the same thing as data. That is the next lesson.

The exact client library differs between vendors and changes every few months, so the shape below is deliberately generic. What does not change is the wrapping.

answer = client.complete(prompt)     # one network call: slow, metered, failure-prone

A line like that in the middle of a request handler, with nothing around it, is the same defect as an unwrapped call to a payment provider. It happens to be written by the person most excited about the feature, which is why it so often ships.

Four wrappings, always

A timeout. Pick one and pass it. Every client library takes one, and the default is usually far longer than you want. A timeout is how you convert an unbounded wait into a failure you can handle.

A bounded retry. Transient failures are common enough that one retry is worth it and unbounded retries are how you take down a recovering provider. Three attempts is a reasonable default, with a growing pause between them so a hundred clients do not all return at the same instant.

def call_with_retry(client, prompt, attempts=3):
    for attempt in range(attempts):
        try:
            return client.complete(prompt)
        except TransientError:
            if attempt == attempts - 1:
                raise
            sleep(2 ** attempt)      # 1s, then 2s: give the provider room

A budget. Calls cost money and a loop over user-supplied input is a loop over your invoice. Count the calls and stop at a cap you chose, rather than at the one your finance team discovers.

A defined answer for failure. When the call does not come back, what does your function return? "It raises" is a legitimate answer. So is a cached previous result, a simpler non-model fallback, or a sentinel the caller can render. What is not legitimate is not having decided, because then the answer is whatever exception happens to escape.

Retrying is only safe when repeating is safe

A retry sends the same request again, which is fine when the call only reads. It is not fine when the call causes something.

If the model call is one step in a flow that also charges a card, sends an email, or writes a row, ask what happens when step three fails after step two succeeded. The answer is usually that the retry must not repeat step two, which means the write needs an idempotency key or the retry has to be scoped to just the model call. This is the same reasoning you would apply to any external write, and generated code almost never includes it, because nothing in the prompt mentioned it.

Testing it without the network

You cannot write a reliable test against a live provider: it is slow, it costs money, and it will not fail on demand. So you do what you would do for any other dependency and hand your code a stand-in that fails exactly when you tell it to.

class FlakyModel:
    """A stand-in client. Each entry in script is what the next call does."""
    def __init__(self, script):
        self.script = list(script)
        self.calls = 0

    def complete(self, prompt):
        self.calls += 1
        step = self.script.pop(0)
        if step == "error":
            raise RuntimeError("upstream error")
        return step

Now "the second attempt succeeds" is a test, and so is "every attempt fails", and both run in a millisecond. The counter matters as much as the responses: an implementation that retries forever and one that stops at three both pass a script of two errors, and only the call count separates them.

Interview nuance: "how would you test this?" applied to a model call is a question that separates people quickly. The weak answer describes checking the output text. The strong answer is that the provider gets replaced by a stand-in, and the tests are about attempt counts, timeouts, and what the function returns when the model never answers. That is ordinary dependency discipline, and the fact that the dependency is a model changes none of it.