ML Systems Design
Design the production systems around a model rather than the model itself: the two-plane blueprint that wires data, features, training, serving, and a feedback loop; a feature store that kills training/serving skew; a real-time recommendation funnel that keeps heavy models off the hot path; and a serving/rollout layer that ships model updates safely and degrades gracefully when the model service is down.
End-to-End ML System Blueprint
Frame business metric to ML objective to label, split an offline training plane from an online serving plane, cascade candidate generation to ranking to re-ranking to hit latency, and close a feedback log so you can retrain and detect drift.
Feature Stores & Training/Serving Skew
A feature store uses a dual offline/online store fed by one definition to kill code-divergence skew, enforces point-in-time as-of joins to prevent label leakage, tiers features by freshness SLA, and proves correctness by comparing served vectors to offline vectors.
Real-Time Recommendation Systems
Cascade two-tower plus ANN candidate generation into a multi-task ranker into diversity re-ranking to hit p99 under 100ms, feed recent clicks through Kafka/Flink for real-time reaction, handle cold start with content features and popularity, and evaluate with exploration plus online A/B to escape feedback-loop bias.
Online Model Serving & Rollout
Roll models out through shadow to canary to A/B with automatic rollback on an online metric, keep versioned artifacts in a registry so rollback is a hot config switch, pick batch/real-time/streaming inference with micro-batching for throughput, spend your latency budget on feature fetch, and always have a graceful degradation ladder for when the model is unavailable.
LLM / GenAI Infrastructure
Whiteboard the systems that sit around a large language model in production: a RAG pipeline that grounds answers in private data with citations and access control, a billion-vector ANN search service, an AI gateway that controls cost and reliability across many providers, a GPU inference server tuned for throughput and time-to-first-token, an agent platform that bounds cost and defends against prompt injection, an eval-and-guardrail pipeline that gates every model change, and the decision framework for choosing prompting versus RAG versus fine-tuning.
RAG (Retrieval-Augmented Generation) Architecture
RAG is ingestion (parse, chunk, embed, index with ACL metadata) plus a query path of hybrid retrieval, a mandatory reranker, ACL-filtered context assembly, grounded generation with citations, and the RAG triad for eval.
Vector Databases & ANN Search
ANN trades recall for speed via HNSW (RAM, high recall), IVF-PQ (quantized, memory-cheap), or DiskANN (SSD-scale); tune ef_search / nprobe; handle filtered search as a pre-filter pushed into the index; and plan for rebuilds and re-embedding migrations.
Model Gateway / LLM Router / AI Gateway
An AI gateway is a unified multi-provider API adding failover and routing, exact plus semantic caching, per-tenant quotas and cost metering, retries/timeouts/circuit breakers with streaming passthrough, and input/output safety plus audit logging, all without becoming a SPOF.
LLM Inference Serving (GPU Economics)
LLM serving is capped by KV-cache memory, so use PagedAttention to kill fragmentation and continuous batching to keep the GPU saturated; reason in TTFT (prefill) vs inter-token (decode) vs throughput; and add quantization, parallelism, prefix caching, and speculative decoding to stretch a fixed GPU fleet.
LLM Agents & Orchestration
An agent is a bounded loop; the controller enforces step/token/time/cost limits, tool calls are schema-validated and sandboxed, side-effecting tools are idempotent, memory can be durable and resumable, and the central safety problem is prompt injection via tool output, contained by treating output as untrusted, least-privilege scoping, and human approval gates.
LLM Evaluation & Guardrails
Gate every change with offline golden-set eval (programmatic checks, calibrated LLM-as-judge, regression suite) plus online canary/A-B, enforce input and output guardrails at runtime (PII, injection, schema, moderation, groundedness), and close the loop by feeding production failures back into the eval sets.
Fine-Tuning vs RAG vs Prompting
Prompting for behavior, RAG for fresh/private knowledge, fine-tuning (via LoRA adapters, rarely full) for style/format/latency; they compose; drive continuous improvement with a data flywheel; and never fine-tune for knowledge that changes when RAG keeps it fresh.
Real-Time Analytics & Global Data
Design the two systems that show up whenever data has to be either fast at massive volume or correct across the planet: a streaming analytics pipeline that turns a firehose of billions of events per day into sub-second trending and per-minute counts, and a globally distributed database that serves low-latency local reads worldwide without letting two regions double-spend the same balance.
Streaming / Real-Time Analytics Pipelines
Kafka backbone, Flink windows keyed on event time with watermarks for late data, exactly-once via checkpointing where counts must be right, HyperLogLog and Count-Min Sketch for bounded-memory counting, and a Druid/Pinot/ClickHouse serving layer for sub-second queries.
Globally-Consistent Multi-Region Data
Cross-region synchronous writes cost 100+ ms because of the speed of light, so use consensus plus TrueTime/HLC for correct ordering, geo-partition rows to their home region for local reads and writes, add follower reads and leases, and choose a consistency level per workload instead of paying for global strong consistency everywhere.
IoT, Edge & Time-Series
Design the two halves of a large sensor platform: the ingestion path that pulls telemetry from millions of intermittently-connected devices and splits it into a hot alerting path and a cold analytics path, and the specialized time-series storage substrate underneath it that survives high write rates and controls the cardinality explosion that kills most metrics systems.
IoT / Edge Ingestion Architecture
Filter and buffer at the edge, connect over MQTT with per-device certs, absorb bursts and reconnects with a Kafka buffer and backpressure, fork into a seconds-latency hot path and a durable cold path, and drive control and OTA through a device shadow with canary rollout.
Time-Series Databases & Storage Design
A TSDB exploits append-only, columnar, delta-of-delta + XOR compressed storage partitioned by time, keeps old data cheap with downsampling and hot/warm/cold tiering plus retention, serves time-range + tag-filtered aggregations, and lives or dies by controlling tag cardinality.