Model Gateway / LLM Router / AI Gateway

The control plane between apps and providers

An AI gateway is the control plane between your applications and one or more LLM providers. It is the same idea as an API gateway, specialized for the economics and failure modes of LLM calls: dollars per million tokens, provider outages, prompt-injection, and wildly variable latency. Without it, every app hard-codes a provider key, there is no cost visibility, one team's runaway loop drains the shared quota, and a provider outage takes down every feature at once. The gateway centralizes all of that.

Unified API and routing

The gateway exposes one API (usually OpenAI-compatible so SDKs just work) and translates to each provider's format behind it. That single abstraction is what enables failover (if Anthropic 529s, retry on OpenAI or Bedrock), load balancing across providers and regions, and routing policy: send cheap-and-easy requests to a small fast model and only escalate hard ones to a frontier model. Routing can be static (this app uses model X), rule-based (long context goes to a long-context model), or learned (a classifier picks the cheapest model likely to pass eval).

Caching is the biggest cost lever

request -> exact-match cache?  hit -> return (0 tokens, ~1ms)
        -> semantic cache?     (embed prompt, ANN lookup, similarity > 0.95) hit -> return
        -> miss -> route to provider -> stream response -> write both caches

Exact-match caching keys on the normalized prompt plus params and returns identical repeats for free. Semantic caching embeds the prompt and returns a cached answer when a past prompt is near-identical in meaning, which catches paraphrases. Semantic caching needs a similarity threshold tuned carefully (too loose and you serve a wrong cached answer to a different question) and invalidation when the underlying data or prompt template changes. For RAG and personalized prompts, cache the expensive shared sub-parts, not the whole personalized response.

Cost, reliability, safety

Per-tenant rate limits and token budgets stop one team from consuming the shared spend. The gateway meters tokens per request, attributes cost per team, and enforces quotas. Retries with backoff on 429/529, per-provider timeouts, and circuit breakers stop hammering a degraded provider and shift traffic to a healthy one. Because responses stream, the gateway must pass tokens through as they arrive, not buffer the whole completion. Graceful degradation means falling back to a cheaper model or a cached answer rather than failing. The gateway is the natural chokepoint for input scanning (prompt-injection and PII detection), output moderation, and audit logging of every prompt and response, plus per-request latency, token, cost, cache-hit, and error metrics.

Interview nuance: the gateway must not become a latency tax or a single point of failure. Keep its own processing to a couple of milliseconds, run it multi-instance behind a load balancer, and make cache and routing lookups fast (Redis, in-memory).

Recap: an AI gateway is a unified multi-provider API adding failover and routing, exact plus semantic caching, per-tenant quotas and cost metering, retries/timeouts/circuit breakers with streaming passthrough, and input/output safety plus audit logging, all without becoming a SPOF.