Prompt Cache Economics and Prefix Ordering

One token, three prices

The serving lesson teaches prefix caching as a latency lever and the gateway lesson teaches token metering as a cost control. What falls between them is that on a hosted API the same token costs three different amounts depending on what the provider had to do with it. It can be read fresh, it can be written into a cache, or it can be read back out of one, and those are three separate line items on the bill.

Table

Multipliers as published in August 2026; check the current price page before quoting one. The ratios are what this lesson is about, and the two highlighted columns are where the providers stop agreeing with each other.

Provider and tier	Uncached input	Cache write	Cache read	Ongoing storage
Anthropic, 5-minute TTL	1x	1.25x	0.1x	none
Anthropic, 1-hour TTL	1x	2x	0.1x	none
OpenAI, automatic caching	1x	no separate write tier	0.1x	none
Google Gemini, explicit cache	1x	no separate write tier	discounted	charged per token per hour

A cache write costs more than an uncached read on Anthropic, which is the fact that makes this a decision rather than a switch. A cache read costs a tenth, which is the fact that makes the decision easy. OpenAI applies the same 0.1x discount to cached input with a 1,024-token minimum and no separate write tier, and lets you steer which entry a request lands on with a prompt_cache_key.

one request's input, priced in multiples of the uncached rate

no caching      request 1: 1.00   request 2: 1.00   ->  2.00 for two
5-minute tier   request 1: 1.25   request 2: 0.10   ->  1.35 for two
1-hour tier     request 1: 2.00   request 2: 0.10   ->  2.10 for two

three requests: 3.00 uncached, 1.45 on the 5-minute tier, 2.20 on the 1-hour tier

the 5-minute write has paid for itself by the SECOND request
the 1-hour write has paid for itself by the THIRD

Two requests is a low bar. Any multi-turn conversation, any agent loop, any assistant that a user asks a follow-up question clears it in seconds, which is why prefix caching is close to unconditionally correct on those providers rather than being a tuning option.

The provider whose model is shaped differently

Google's explicit context caching adds a term the other two do not have: storage, charged per token per hour for as long as the cache lives. That is rent rather than a purchase, and rent changes which caches are worth holding.

a 200,000-token cached prefix, held for 8 hours, hit twice an hour

no storage term
  write   200k x the write multiplier, paid once
  reads   16 x 200k x the read multiplier
  idle    free. an unhit hour costs nothing at all

with a per-token-per-hour storage term
  write   200k x the write multiplier, paid once
  reads   16 x 200k x the read multiplier
  rent    200k x 8 hours x the hourly rate, paid whether anyone hits or not

the rent term does not care about your hit rate, so a big cache that is
rarely hit can cost more than never caching it. "cache everything" belongs
to one pricing structure and is wrong under the other

Gemini also runs implicit caching on by default on recent models, so on that provider you may already be receiving a discount you have not accounted for, and a cost model that assumes every input token is billed at full rate will not reconcile.

Why the prefix has to be a prefix

A cache entry is keyed on the exact token sequence starting at position zero. Not a hash of the content, not a set of blocks: an ordered prefix. The match runs forward from the first token and ends at the first token that differs, and everything after that point is uncached work.

layout A   [ "Hello Ana, you are a support agent" ][ 12k policy corpus ][ user turn ]
             ^ differs per user
           first difference at token ~3
           cacheable prefix: 3 tokens. effectively nothing.

layout B   [ "You are a support agent" ][ 12k policy corpus ][ "Ana asks:" + user turn ]
             ^ byte-identical on every request
           first difference at token ~12,030
           cacheable prefix: ~12,030 tokens, read at 0.1x from the second request on

the minimum cacheable prefix is per-model and NOT monotonic across generations.
on Anthropic it ranges from 512 to 4,096 tokens depending on the model, so a
3k-token prompt caches on one model and silently does not on its successor.
the failure reports zero cache-creation tokens. it never raises.

The ordering rule, and the budget on it

The rule falls straight out of that picture: order the prompt from most stable to least stable. System instructions first. Then whichever of the tool definitions and the shared corpus changes least often, and on a team that deploys daily, that is the corpus. Retrieved RAG chunks late, because they are chosen per question and are therefore volatile, with one exception worth naming: if a small hot corpus is attached to nearly every request, it is stable and belongs early. The user's turn goes last, always.

You are not annotating freely. On Anthropic you get at most four explicit cache_control breakpoints per request, so you are choosing four boundaries in the prompt and everything between two boundaries is one cacheable unit. There is also a coupling that catches people: changing tool_choice invalidates cached message blocks, so a routing decision made per request can quietly cost the cache on a prompt whose text never moved.

The clock starts earlier than you think

The cache lifetime is measured from the start of the request that writes or reads the entry, not from the end of the response. Generation time is spent out of the TTL, so a long answer eats its own cache window.

5-minute tier, TTL measured from the start of the request

14:00:00   request 1 arrives, writes the prefix. the 5-minute clock starts HERE
14:00:04   first token
14:04:10   last token of a long streamed answer
14:05:00   the entry expires
14:05:30   the user's follow-up arrives  ->  MISS, and the prefix is rewritten

the user waited 80 seconds. the cache saw a gap of five and a half minutes.

a read refreshes the TTL at no extra charge, so a chatty session stays warm,
while a session with one long generation per turn can miss on every turn

Why an agent needs this rather than benefits from it

The other half of the bill is structural. An agent resends the whole conversation as input on every turn, so turn N carries everything from turns 1 through N. The input token count over a run is therefore proportional to the sum of 1 through N, which is quadratic in the number of turns rather than linear.

The self-hosted mirror image

Nothing above is unique to hosted APIs; it is the same mechanism with the price tag removed. On a self-hosted engine the equivalent is automatic prefix caching over the KV cache. SGLang's RadixAttention keeps the cached prefixes in a radix tree so the shared prefix across requests is discovered rather than declared, with LRU eviction on the tree's leaves. vLLM hashes fixed-size blocks into a global table and caches only complete blocks, and ships it on by default because the measured cost when the hit rate is zero is under one percent of throughput. Real hit rates are not marginal: DeepSeek published a production breakdown in which 56.3 percent of input tokens over twenty-four hours were served from KV cache.

Interview nuance: prefix caching and semantic caching are different levers and candidates blur them. A prefix cache is a discount on work you still perform: the model still runs, it just skips recomputing KV for tokens it has seen. A semantic cache skips the call entirely and returns a stored answer. Say which one you mean. Then answer the isolation question before it is asked: a cross-request prefix cache is shared state, and the concrete control is a per-tenant salt on the cache key, which vLLM exposes directly as cache_salt, so a prefix produced under one tenant can never be reused under another.

Recap: a token has three prices, so prompt assembly is a financial decision; order blocks from most stable to least stable within your four breakpoints; remember that the match ends at the first differing token and the minimum prefix varies per model; the TTL runs from the start of the request rather than the end of the response; an agent's input cost is quadratic in turns, which is what makes caching a prerequisite; and a shared prefix cache is shared state that wants a per-tenant salt.

Sources: Anthropic prompt caching · OpenAI prompt caching · Gemini context caching · vLLM automatic prefix caching