Prefill and Decode Disaggregation

Prefill and decode are two different machines

A request to a transformer runs in two phases that share hardware and share weights and have almost nothing else in common. Prefill reads the whole prompt in one pass and writes the KV cache for it, so every weight the model holds is reused across every prompt token in that pass. Decode then emits one token at a time, and each step reads the entire set of weights again to produce that single token. Same GPU, same weights, opposite bottleneck.

The number that separates them is arithmetic intensity: the floating point operations a kernel performs per byte it pulls out of memory. Every accelerator has a ridge point, the intensity at which its peak compute and its peak memory bandwidth are exactly balanced. Below the ridge the math units wait on memory and the kernel is memory-bandwidth-bound; above it memory keeps up and the kernel is compute-bound. Work the two phases out on the same card and they land on opposite sides of it.

one weight matrix, 2 bytes per element, W bytes total, on a fixed model

prefill of a 512-token prompt
  bytes moved   W            the weights are read once for the whole prompt
  FLOPs         512 x W      each of 512 tokens is multiplied through them
  intensity     ~512 FLOP per byte read

decode, one sequence, one step
  bytes moved   W            the same weights, read again
  FLOPs         1 x W        to produce exactly one token
  intensity     ~1 FLOP per byte read

A100 80GB ridge point
  312 TFLOP/s dense BF16 / 2.04 TB/s HBM = ~153 FLOP per byte

  512 is far above 153  ->  prefill runs into the compute ceiling
    1 is far below 153  ->  decode runs into the bandwidth ceiling

That is not a toy result. DistServe measures the same crossing on a real model: for a 13B model, prefilling a 512-token sequence is already enough to put an A100 near compute-bound, while the decode steps for that same request sit at the far end of the bandwidth side.

Two SLOs, and one pool has one knob

Because the phases are different machines, they answer to different service levels. An SLO is a service level objective, the promise you publish about a number and the share of requests that must hit it, and Level 7's SLI, SLO and SLA lesson is where it came from. Time to first token (TTFT) is prefill's number and scales with prompt length. Time per output token (TPOT), the same quantity the serving lesson calls inter-token latency, is decode's number and is what streaming feels like. A single pool has one scheduler and one batch policy, so tuning it for throughput fills the batch with prefill work and misses TTFT, while tuning it for TTFT admits prompts eagerly and leaves the GPU underfed.

The metric that makes that decidable is goodput: the maximum request rate a fleet sustains while still meeting its SLO attainment target, for example ninety percent of requests meeting both TTFT and TPOT. Raw requests per second counts requests you served badly. Goodput counts only the ones you served inside the contract, which is why a change can raise throughput and lower goodput at the same time, and why the fleet you size on throughput is the wrong fleet.

They interfere, and chunking only spreads the interference

Continuous batching admits a queued request the instant a slot frees, which is exactly right when the arriving work is another decode stream. When the arriving work is a long prefill, the batch step it occupies is a step that would otherwise have advanced every running sequence by one token.

one 32k prefill arrives while 60 sequences are decoding

no chunking
  t=0ms      decode  decode  decode      every stream advances one token per step
  t=90ms     [==== 32k prefill owns the batch step, ~1.2s ====]
  t=1290ms   decode  decode               the 60 streams resume
             new request TTFT: excellent. everyone else's TPOT: 1.2s of nothing

chunked prefill, 512-token chunks
  t=90ms     [chunk][decode][chunk][decode][chunk][decode] ...
             each step carries one slice of prefill alongside the decodes
             no stream pauses longer than one chunk, so TPOT stays smooth
             the prefill now finishes later, so its own TTFT is worse

Chunked prefill, introduced by Sarathi, is the colocated mitigation: split a long prompt into fixed-size pieces and piggyback each piece onto a decode step so no single step is enormous. It works, and it is the right first move. It is a mitigation rather than a fix because both phases still share one pool and one scheduler, so the chunk size is a single knob with the two SLOs tied to opposite ends of it. Raise the chunk size and prefill completes sooner while decode stutters; lower it and decode smooths out while TTFT slides. You can choose a point on that curve. You cannot leave the curve.

Disaggregation: two fleets, two scaling laws

Disaggregation leaves the curve by giving each phase its own machines. A prefill pool runs prompts and produces KV cache; a decode pool receives that cache and streams tokens. The pools scale independently, can sit on different GPU types, and no longer contend for the same batch step, so a burst of long prompts cannot reach an in-flight stream at all.

The published results are worth carrying because they are stated against baselines. DistServe reports serving 7.4x more requests, or holding SLOs 12.6x tighter, at over ninety percent SLO attainment compared with the serving systems it measures against. Splitwise reports 1.4x more throughput at twenty percent lower cost by splitting the phases across machine types.

What it costs: the KV cache has to move

The obvious objection is that the KV cache produced by prefill is exactly what decode needs, so a split forces it across a wire. Put numbers on it before deciding whether that is fatal.

Splitwise cuts the tax further by overlapping the copy layer by layer, sending each layer's KV as soon as that layer has produced it, so most of the transfer hides behind prefill compute that was being paid for anyway. Measured that way the overhead stays under seven percent, with a constant non-overlapped component of roughly 8ms on A100 and 5ms on H100 over InfiniBand. Multi-node disaggregation therefore assumes RDMA. A TCP fallback is not a fallback, and the widget above is the argument: at 3 GB/s the copy costs more than the work it was meant to save.

Ratio planning: the pool shape follows the workload shape

Two pools means a second sizing question: how many of each. The prefill-to-decode replica ratio follows the ratio of prefill work to decode work, and those follow prompt length and output length once you divide by each phase's measured per-node rate. The consequence is that two products running the same model on the same hardware want differently shaped fleets.

Table

Both rows stream for about the same length of time, so decode demand is nearly identical. Prefill demand differs by a factor of 66, and that factor is the whole reason one fleet cannot be copied onto the other product.

Workload	Prompt	Output	Prefill work	Decode occupancy	Implied shape
Interactive chat	300 tokens	500 tokens	300 token-passes	500 steps, ~15s of a slot	decode-heavy, most replicas decode
Document summarization	20,000 tokens	400 tokens	20,000 token-passes	400 steps, ~12s of a slot	prefill-heavy, most replicas prefill

Compute the ratio from measurements, not from the token counts alone: prefill and decode consume a node at different rates, so the numbers above become replica counts only after dividing by a measured prefill rate in tokens per second and a measured decode capacity in concurrent sequences.

The beat that keeps this honest

vLLM's own documentation is blunt about what disaggregated prefilling is for: it does not improve throughput. What it buys is the ability to tune TTFT and inter-token latency separately, and to keep tail inter-token latency under control. If your problem statement is "we want more tokens per dollar", this is the wrong lever and the serving lesson's levers are the right ones.

TaiChi draws the boundary more precisely still. Aggregation wins when TTFT is tight and TPOT is relaxed, because a colocated pool can spend whole batch steps on prefill the moment a prompt lands. Disaggregation wins when TPOT is strict and TTFT is relaxed, because an isolated decode pool is never interrupted. Under balanced SLOs neither shape is optimal on its own, which is an uncomfortable finding and the most useful one in the lesson: there are regimes where the correct answer is a hybrid, or a measurement.

Interview nuance: name the regime, not the technique. Disaggregation loses on small models, short prompts, low concurrency, and any cluster without a fast KV fabric, and saying so unprompted is what separates someone who has run it from someone who has read about it. State the SLO shape first, derive the topology from it, and say which number you would watch to know you chose wrong.

Recap: prefill is compute-bound and decode is memory-bandwidth-bound, so they answer to different SLOs and interfere inside one pool; chunked prefill spreads that interference along a single knob, disaggregation removes it by giving each phase its own fleet at the price of a KV transfer that is cheap over RDMA and ruinous over TCP; size the two pools from measured rates rather than token counts; and remember that the split buys SLO separability, not throughput.

Sources: DistServe: disaggregating prefill and decoding · Splitwise: phase splitting · vLLM disaggregated prefilling · TaiChi: aggregation or disaggregation