End-to-End ML System Blueprint

The interview is about the plumbing, not the model

An ML system design interview is almost never about the model. It is about the plumbing around the model: where features come from, how the thing retrains, how you serve predictions in single-digit milliseconds, and how you notice when it silently rots. Candidates who dive straight into "I would use gradient-boosted trees" fail; the ones who draw two planes and a feedback loop pass.

Frame the metric hierarchy first

There is a business metric (revenue, engagement), an ML objective that is a proxy for it (predicted click-through rate), and a training label that is a proxy for that (did the user click within a 30-minute attribution window). These are never identical, and the gap is where products die. Offline metrics (AUC, log-loss on a holdout) tell you the model learned something; online metrics (actual CTR, revenue per session in an A/B test) tell you it helped. Optimizing offline AUC while online engagement drops is the classic trap.

Two planes plus a loop

OFFLINE (training plane)                 ONLINE (serving plane)
raw logs -> ETL -> feature pipeline      request -> feature fetch (online store)
   -> training data -> train             -> candidate gen -> ranking -> response
   -> eval -> model registry  --push-->  -> model service loads artifact
        ^                                        |
        |                                        v
        +---------- feedback log <--------- impressions + outcomes (clicks)

The offline plane is throughput-oriented and runs on a schedule: batch ETL over the warehouse, feature computation, training, evaluation, and a push to a model registry. The online plane is latency-oriented and runs per request: fetch precomputed features, generate candidates, rank, return. They must share one feature definition or you get training/serving skew. The feedback log is the piece juniors forget: every prediction and its eventual outcome must be written back, because without it you cannot build tomorrow's training set or detect drift.

The latency and cost funnel

You do not run a heavy model on millions of items per request. You cascade: candidate generation cheaply narrows millions to hundreds (embedding retrieval or a simple filter), ranking runs the expensive model on those hundreds, and re-ranking applies business rules and diversity on the top dozen. Each stage is cheaper per item and touches fewer items, so total cost stays bounded.

Interview nuance: rollout is not a stateless deploy. A model is code plus weights plus the feature distribution it expects. Ship it through shadow (score live traffic, serve nothing), then canary or A/B (small traffic slice), with automatic rollback keyed on an online metric regression, and keep the previous artifact hot for instant revert.

Monitoring closes the loop

Watch data drift (input feature distributions shift), concept drift (the label relationship changes, for example fraud tactics evolve), prediction drift (output distribution moves), plus operational alarms on feature nulls and ground-truth label delay. Daily retraining only helps if these signals decide when a retrain or rollback is warranted.

Recap: frame business metric to ML objective to label, split an offline training plane from an online serving plane, cascade candidate generation to ranking to re-ranking to hit latency, and close a feedback log so you can retrain and detect drift.