Online Model Serving & Rollout

Shipping a model is not shipping a stateless service

Shipping a model is not shipping a stateless service, and treating it like one is how teams cause outages that lose money silently. A model deploy changes behavior in ways a green health check cannot catch: the new artifact may load fine and return 200s while quietly making worse predictions. So the serving and rollout layer is designed around two ideas: never trust a new model on real traffic without measuring its decisions, and never let the model service being down take the product down with it.

The rollout ladder

Rollout strategies form a ladder of increasing exposure with a measurement gate at each rung. Shadow (or dark launch) runs the new model on live traffic and logs its predictions but serves the old model's output, so you compare decisions on identical inputs with zero user risk. Canary sends a small traffic slice (1 to 5 percent) to the new model and watches business and operational metrics. A/B splits traffic to attribute a metric change causally. Interleaving, used in ranking, mixes results from two models in one list to compare them with far fewer samples. The non-negotiable piece is automatic rollback: a controller watches an online metric (CTR, revenue, error rate, latency) and reverts to the previous artifact on regression, which requires keeping that previous artifact hot for an instant switch, not a redeploy.

Separate weights from serving code

The registry holds versioned, reproducible artifacts (weights plus the feature schema plus preprocessing) addressed by id; the serving binary loads an artifact by id. This lets you roll a model forward or back by pointing at a different id without shipping code, and it makes rollback a config change measured in seconds.

Inference modes and the latency budget

Real-time (online) inference scores per request and is what most interactive products need. Batch inference precomputes predictions offline (nightly scoring of every user) and serves them from a cache, which is cheapest when inputs change slowly. Streaming inference scores events as they arrive. Micro-batching, grouping requests that land within a few milliseconds into one model call, trades a tiny latency increase for large throughput gains and is essential on GPUs.

Meeting the latency budget is mostly a feature-fetch problem, not a model-math problem. The model forward pass is often the cheap part; fetching dozens of features per request from an online store is where the milliseconds go. Co-locate or cache online features next to the model service, batch the reads, and keep the hot set in memory. If your budget is 30ms and feature fetch is 20ms of it, optimizing the model buys you little.

Interview nuance: the question that fails most candidates is "what happens when the model service is down." A strong answer is a graceful degradation ladder: serve the last cached prediction, then a simpler fallback model that needs fewer or no features, then a static heuristic or default, and only then error. A fraud system, for example, falls back to strict rules rather than approving everything; the fallback's bias should fail safe for the domain.

Recap: roll models out through shadow to canary to A/B with automatic rollback on an online metric, keep versioned artifacts in a registry so rollback is a hot config switch, pick batch/real-time/streaming inference with micro-batching for throughput, spend your latency budget on feature fetch, and always have a graceful degradation ladder for when the model is unavailable.