GPU Fleet Sizing and the Self-Host Crossover

A different unit, and a bottleneck that moves

Level 4 sizes a fleet from queries per second and a per-core service time, and Level 9 allocates cloud spend once the fleet exists. Neither answers the question an AI team is actually asked, which is how many GPUs, and whether self-hosting beats the API. The arithmetic is different for two reasons. The unit is tokens per second per GPU rather than requests per second per core, and the binding constraint moves between phases: FLOPs during prefill, memory bandwidth during decode, and HBM capacity for the KV cache that both of them depend on.

Every number in this lesson is a ratio you can recompute. Prices and card specifications are stated as of August 2026 and will drift; the ridge point, the KV bytes per token and the dollars per unit of bandwidth will not.

Read the spec sheet, then find the ridge point

H100 SXM, as published (figures as of August 2026)

  BF16 tensor core     1,979 TFLOP/s   quoted WITH structured sparsity
  dense equivalent       989 TFLOP/s   halve it
  HBM3 bandwidth          3.35 TB/s    per GPU
  NVLink                   900 GB/s    GPU to GPU. NOT memory bandwidth

ridge point, dense
  989 TFLOP/s / 3.35 TB/s = ~295 FLOP per byte

  the two ways to get this wrong, and what they cost
    using the sparse figure          ~591 FLOP per byte   out by 2x
    using NVLink as the bandwidth  ~1,100 FLOP per byte   out by nearly 4x

where decode sits on that line
  at batch 1 each weight is read once and used in one multiply-add,
  which is 2 FLOPs for a 2-byte weight, so ~1 FLOP per byte
  1 against a ridge of 295 is about a third of one percent of peak compute

Batching is what walks decode up that line. Two sequences read the same weights once and do twice the math, four sequences do four times, so arithmetic intensity rises roughly with the batch size and throughput is nearly free until the batch approaches the ridge. Past it you are on the compute wall and each additional sequence costs what it looks like it costs. One caveat keeps this honest: attention over the KV cache does not amortize across the batch, because every sequence has its own cache to read, so long contexts push the true crossover to the left of where the weight arithmetic alone would put it.

the per-sequence speed limit

  one sequence advances one token per forward pass
  a forward pass reads the resident model out of HBM

  a 70B model at FP8 is ~70 GB resident
  3.35 TB/s / 70 GB = ~48 forward passes per second

  so ~48 tokens/sec for ONE sequence, on that card, at that precision.
  no batch size, scheduler or autoscaler changes that number.
  batching raises tokens/sec for the FLEET. it never raises it for the user

The attention architecture is a line in the budget

Now the arithmetic that decides cost per token more than any other single choice, and which lives on the model card rather than in your infrastructure.

KV bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_element

a 70B-class model with grouped-query attention
  2 x 80 layers x 8 kv heads x 128 head dim x 2 bytes
    = 327,680 bytes = 320 KiB per token

the same shape with full multi-head attention, 64 kv heads instead of 8
  2 x 80 x 64 x 128 x 2 = 2,621,440 bytes = 2.5 MiB per token   (8x more)

one 8-GPU node, 640 GiB of HBM, weights at FP8 take 70 GiB -> ~570 GiB for KV
  at 8,192 tokens per sequence
    grouped-query   8,192 x 320 KiB = 2.5 GiB  ->  ~228 concurrent sequences
    multi-head      8,192 x 2.5 MiB =  20 GiB  ->   ~28 concurrent sequences

Eight times the concurrency on identical silicon, decided by one integer on a model card. Below the ridge point, throughput rises roughly with concurrency, so 8x the concurrent sequences is close to an 8x cut in cost per token. A model's KV-head count is not an architecture detail you note in passing; it is a budget line, and it belongs in the model-selection conversation next to quality.

Mixture of experts breaks the naive calculation

A dense model has one parameter count and it sets everything. A mixture-of-experts model has two, and they set different things: you provision memory for the total parameter count, because every expert has to be resident somewhere before a router can choose it, and you provision compute for the active parameter count, because only the selected experts run for a given token.

Table

Total-to-active ratios of 1 to 1, roughly 11 to 1, and roughly 31 to 1. Model card figures as of August 2026. A fleet sized on either column alone is wrong in a predictable direction: size on total and you buy compute nobody uses, size on active and the weights do not fit.

Model	Total parameters	Active per token	Experts	Memory sized on	Compute sized on
A dense 70B	70B	70B	none	70B	70B
Qwen3-235B-A22B	235B	22B	128	235B	22B
Kimi K2	1T	32B	384	1T	32B

Expert parallelism is how the memory side is made survivable: experts are spread across GPUs rather than replicated, which needs a fat fabric because routing produces an all-to-all exchange every layer, and which frees per-GPU memory that the KV cache then grows into. So expert placement is a concurrency decision as well as a compute one, and a cluster without the interconnect for it cannot run the model the spreadsheet said it could.

Sizing a fleet end to end

a support-summarization tool, sized in five steps

1  demand       900 requests/minute = 15 requests/sec
2  token rate   15 x 250 output tokens  = ~3,750 decode tokens/sec
                15 x 1,200 prompt tokens = ~18,000 prefill tokens/sec
3  divide by MEASURED per-node throughput at a batch that meets the SLO
                decode   3,750 / 2,500  = 1.5 nodes
                prefill 18,000 / 12,000 = 1.5 nodes
4  peak-to-average from real arrival data, say 3x   ->  9 nodes
5  one spare per failure domain                     -> 10 nodes

step 3 says MEASURED, and that is the step people skip. A throughput derived
from FLOPs and bandwidth is an upper bound nothing reaches, and sizing on it
is how a fleet ends up at half the capacity it needed.

step 4 usually costs more than every quantization project on the roadmap saves

Utilization is the hidden variable, and it decides everything

A reserved GPU bills at one hundred percent whether or not you are using it. An API bills at zero percent when you are idle. That single asymmetry is why the self-host crossover is not a price, it is a duty cycle.

Two facts to attach to that widget. First, identical silicon rents across roughly a three to four times range depending on the vendor and the commitment, so shopping is worth real money before any engineering is. Second, for a decode-heavy workload the ranking metric is not dollars per hour but dollars per terabyte per second of memory bandwidth per hour, and ranking a vendor list that way reorders it, because the cards with the best headline compute are not always the ones with the best bandwidth per dollar.

Then the levers that move the numerator without touching the fleet: a provider's batch tier at roughly half price for anything that tolerates a delay, cached input from the prompt-cache lesson for anything with a stable prefix, spot capacity at a discount that you pay for in eviction handling, and the engineers a self-hosted fleet needs, who are a real line in a real budget and are missing from every comparison that concludes self-hosting is cheaper.

Interview nuance, and the honest conclusion. Published self-host break-even estimates disagree with one another by about two orders of magnitude, and that disagreement is itself the finding: each of them buried a utilization assumption. The senior answer states a threshold rather than a verdict ("above roughly this sustained utilization, for this model, against this API price, self-hosting wins"), names the non-cost reasons that usually decide it anyway (data residency, custom or fine-tuned weights, guaranteed capacity during a provider incident, a latency floor you cannot get from a shared endpoint), and does not claim self-hosting is cheaper.

Recap: halve the sparse headline figure and divide by real HBM bandwidth to get a ridge point; decode at batch 1 sits far below it, batching walks it up, and attention over the KV cache does not amortize; one sequence's token rate is bandwidth over resident bytes and no batch changes it; KV bytes per token comes from layers, KV heads, head dimension and dtype, so a model card's KV-head count is a cost line; an MoE is sized on total parameters for memory and active parameters for compute; size fleets from measured per-node throughput, then peak-to-average, then a spare; and the self-host crossover is a sustained-utilization threshold rather than a price comparison.

Sources: NVIDIA H100 product page · Efficiently scaling transformer inference · Kimi K2 model card · Qwen3-235B-A22B model card