Prompt caching, from the causal mask outward

The bill you are already paying

The Messages API is stateless. There is no session, no conversation ID, no handle to something the server is holding for you. Every request carries the entire conversation, and on turn twelve of an agent loop that looks like this:

await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16_000,
  tools,              //    600 tokens — unchanged since the last deploy
  system: houseRules, //  2,400 tokens — unchanged since the last deploy
  messages: history,  // 11 turns you have now sent eleven times
})

The server reads all of it. For every token it computes a key and a value vector at every layer, runs the forward pass, streams the answer, and throws the whole intermediate state away. On turn thirteen you send the same bytes and it does the same arithmetic again.

This is not a rounding error. Fifty questions about one 20,000-token document, each with a hundred-token question attached, is 1,005,000 input tokens — of which 980,000 are the same document read forty-nine more times. At Opus 5’s $5 per million that is $5.03 to ask fifty questions about a document you sent once. Cached, the same fifty requests bill one write, forty-nine reads and the questions: $0.64. And you pay the uncached version twice over — in money, and in time-to-first-token, because prefill is the latency floor before generation can begin at all.

So why is the API stateless in the first place? Because the alternative is worse. A server-side session pins your conversation to one machine, makes retries stateful, and takes away the thing that makes the Messages API tractable: you own the transcript. You can rewind it, branch it, edit a turn, replay it against a different model. Prompt caching exists to keep that property and stop paying for it twice — the client stays the sole owner of the conversation, and the server keeps a disposable copy of the arithmetic.

Why the unit is a prefix, and can never be a block

Attention is causal. Token i attends to tokens 1 through i and to nothing to its right — that is what the causal mask enforces, and it is the reason a language model can be trained on all positions at once. Two things follow immediately.

First, the work is a pure function of the prefix. The K/V vectors at position i depend on tokens 1..i and on nothing else — not on what comes after, not on the question you eventually ask. A pure function of its inputs is exactly what a cache can stand in for, and purity is what makes the cache correct rather than merely fast.

Second, and this is the load-bearing half: the cacheable unit is a prefix, not a block. A block of text has no position-independent representation. Change one byte anywhere before it and every vector from that point on is different.

But I send the same document every time — why can’t the server recognise it wherever it lands? Because “the same document” is not the same computation. Its vectors are a function of the 12,000 tokens of system prompt sitting in front of it. Caching it independently of that would mean caching a function of an argument you never recorded, and returning it later would be returning a wrong answer quickly. A block-level cache is not a harder engineering problem; it is an incorrect one.

Then why a hash, rather than comparing the text? Because you cannot diff 200,000 tokens on every request for free. The system fingerprints the cumulative prefix and compares fingerprints. That makes the match exact and byte-level: one space, one reordered JSON key, one tool description reworded, and it is a different prefix. There is no fuzzy match, and there cannot be one for the same reason as above.

And why does the order tools → system → messages exist? A prefix requires a total order over the request, and the request is a set of fields. Something has to impose one. That single choice, frozen, is what produces the entire invalidation hierarchy later on: it is why editing a tool costs you the whole conversation, and why appending a message never costs you the tools.

What is actually in the cache

The K/V tensors, plus the hash that identifies them. Held in memory, not written to disk, never used for training, and isolated: per workspace on the Claude API, Claude Platform on AWS and Microsoft Foundry, and per organization on Amazon Bedrock and Vertex AI. Never across organizations.

That isolation is worth knowing before you debug a hit rate. Traffic for one prompt split across two workspaces writes and reads two separate entries and looks, from the usage fields, exactly like a broken prefix. Caches are also model-scoped — a fact that quietly prices every multi-model design, and which we will come back to.

Anatomy of a cached request

The request is assembled in that fixed order, and a cache_control marker is a cut line through the stack. Everything at or above the line is one cache entry, keyed by a hash of all of it together. Everything below is ordinary uncached input.

There are two ways to place one.

Automatic — a single cache_control at the top level of the request body. The system applies the breakpoint to the last cacheable block and moves it forward as the conversation grows, so you never touch markers again:

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16_000,
  cache_control: { type: 'ephemeral' },   // ← top level, not on a block
  system: houseRules,
  messages: history,
})

Explicit — a marker on individual content blocks, up to four per request. Use these when different sections change at different rates:

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16_000,
  tools,                                    // changes on deploy
  system: [
    { type: 'text', text: HOUSE_RULES },
    { type: 'text', text: retrievedDoc,
      cache_control: { type: 'ephemeral' } }, // ← caches tools + both system blocks
  ],
  messages: history,
})

The two compose. The automatic breakpoint consumes one of the four slots, and there are two documented ways to get a 400 out of combining them: all four slots already taken by explicit markers, and an explicit marker on the last block whose TTL differs from the top-level field’s. Declaring breakpoints is otherwise free — you are billed on what is written and read, never on how many markers you placed.

Write here; read backwards

Two rules govern whether you get a hit, and the second is the one that surprises people.

1. A write happens only at your breakpoint. Marking a block creates exactly one entry — the hash of the cumulative prefix ending there. No entries are created for earlier positions.
2. A read walks backward. If the hash at your breakpoint does not match, the system steps back one position at a time, checking each earlier position for an entry a previous request wrote. It is looking for prior writes, not for content that happens to be stable.

Why not just write an entry at every block boundary? Because a write is real memory and real work. The K/V cache for a 20,000-token prefix is large, and writing one at every position would multiply the store by the number of blocks in service of a benefit only one position will ever deliver. The design instead asks you to declare where you expect to resume, and honours exactly that. The four-breakpoint limit is the same trade seen from the other side.

The placement that costs more than no caching at all

Now the failure. Static context in the first few blocks, a per-request block carrying a timestamp at the end, breakpoint on the last block. Every request produces a different hash there. The walk steps back through the earlier blocks and finds nothing — because no request ever wrote at those positions. So you pay a fresh cache write every single time and never once read.

Surely the backward walk saves me here? No, and this is the crux of the whole mechanism. The walk looks for writes. Those earlier blocks are stable, but stability is not the criterion — no request ever placed a breakpoint there, so there is nothing to find. The fix is to move the breakpoint to the last block that is byte-identical across requests, not to hope the fallback covers you.

Automatic caching falls into exactly this trap when the prompt ends in unique per-request content, because it targets the last block. The signature in the usage fields is unmistakable once you know it: cache_creation_input_tokens on every request while cache_read_input_tokens never covers the shared prefix.

Move the breakpoint below and watch the arithmetic. The document here is 18,000 tokens; the timestamp that ruins it is forty.

Two things in there are worth sitting with. The pathological placement is more expensive than no caching — not merely wasteful. And nothing in the response says so: no error, no warning, no field that turns red. Just a bill.

When “the last block” is exactly right

A growing conversation is the case automatic caching was built for. Earlier turns never change, so each request’s lookback finds the entry the previous request wrote a few positions back. The breakpoint rolls forward and the cached region ratchets.

That gives you the signature of a loop that is working, which is worth memorising because it is how you tell a real hit from a coincidence:

• cache_read_input_tokens — the whole prior prefix, growing every turn.
• cache_creation_input_tokens — roughly the last assistant output plus the newly appended input. Small.
• input_tokens — just the tail after the last breakpoint.

The catch is the twenty-position lookback. Each turn in the diagram adds two positions, so turn three easily reaches turn two’s write. An agent loop that appends a dozen positions per iteration — long sequential tool chains, many text or image blocks — will eventually push the breakpoint more than twenty positions past the last write, and then it falls off a cliff: full miss, full rewrite, byte-identical payloads, no warning. Place an intermediate breakpoint roughly every fifteen positions in long turns, so a write has accumulated closer in before you need it.

Four breakpoints, four clocks

The reason for multiple breakpoints is not granularity for its own sake. You place them at change-frequency boundaries, and four is enough because prompts rarely have more than four distinct rates of change:

tools            ← breakpoint 1    changes on deploy
system: rules    ← breakpoint 2    changes on deploy
system: docs     ← breakpoint 3    changes daily
last user turn   ← breakpoint 4    changes every turn

Swap the documents and breakpoints 1 and 2 still hit; only 3 and 4 are rewritten. That is the entire point of the segmentation, and it is why a RAG agent that reindexes nightly should never put its retrieved rows ahead of its house rules.

The robust default for an agent loop is a combination: one explicit breakpoint on the last block of the static system prefix — so the expensive shared part has a guaranteed read point that survives whatever happens later in messages — plus top-level automatic caching for the growing conversation tail.

The cascade, and the doors out of it

Because the prefix is ordered tools → system → messages, a change at one tier invalidates that tier and everything after it. Nothing before it. That single sentence is the whole table, but the table is worth having in front of you, because the consequences are asymmetric in ways that are easy to get backwards.

Three of those rows now have escape hatches, and they are the most useful recent development in this whole area, because each one converts a prefix rewrite into an append.

The one to reach for first: stop editing the top-level system prompt mid-conversation. An operator instruction that arrives on turn nine — a mode switch, injected state, a policy change — used to mean rewriting the block that sits in front of the entire conversation. Send it as a message instead:

await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16_000,
  system: [
    { type: 'text', text: STABLE_CORE,
      cache_control: { type: 'ephemeral' } },   // byte-identical, still hits
  ],
  messages: [
    ...history,
    { role: 'user', content: userMessage },
    { role: 'system',                            // ← after the cached prefix
      content: 'Terse mode enabled — keep responses under 40 words.' },
  ],
})

Available today with no beta header on Claude Opus 5, Opus 4.8, Fable 5 and 5.1, and Mythos 5 and 5.1 — not on Claude Sonnet 5, where it returns a 400 you should catch and fall back from. It has a second property worth as much as the caching one: a role: "system" message is a channel nothing else can write to, whereas operator instructions embedded as text inside a user turn can be forged by anything that reaches user input.

It must follow a user message (or an assistant message ending in server-tool use), must be either last in messages or followed by an assistant turn, and cannot be messages[0] — the initial prompt still belongs in top-level system.

What about the inject-a-reminder-then-delete-it pattern? It is a history edit, and it costs you the cache from that position on every single turn — plus, on Fable 5.1 and Mythos 5.1, it invalidates every later thinking block. Give the message clear_at: "next_user_message" instead (beta mid-conversation-system-clear-at-2026-08-21): it renders for one turn, then stays in the transcript cleared — costing no input tokens, still part of the prefix. Append a fresh copy each turn and never remove the old ones.

The row with no door is the model switch. Caches are model-scoped, so a cost cascade that routes easy turns to a cheaper model forfeits cache reuse across its lanes — each model is its own namespace. Measure the simpler alternative first: the better model at lower effort, on one cache.

One more that catches people building agents: a fork must reuse the parent’s exact prefix. Summarizers, compactors and sub-agents usually spin up a separate call, and if that call rebuilds system, tools or model with any difference at all, it misses the parent’s cache entirely. Copy them verbatim and append the fork-specific content at the end.

The arithmetic

Writes cost more than ordinary input; reads cost far less. Everything else is division.

Break-even is two requests on the 5-minute TTL (1.25 + 0.1 = 1.35, against 2.0 for sending it twice uncached) and three on the 1-hour TTL (2 + 0.2 = 2.2, against 3.0). One hit repays the write. These multipliers stack with other modifiers, including the Batch API discount.

Claude Fable 5.1 is the exception worth knowing: reads there are 0.025× ($0.25/MTok), which moves every break-even below proportionally and changes a TTL decision we will get to in a moment. Whether Mythos 5.1 shares that rate was open at launch.

The clock starts earlier than you think

Default TTL is five minutes, and a read refreshes it for free. The part that bites: the lifetime is measured from the start of the request that writes or reads the entry, not from the end of the response. A generation that streams for four minutes leaves you about one minute to get the next request in.

So choose by the start-to-start gap between requests that share the prefix, not by how long a session feels:

Set it per block, and remember the ordering constraint: 1-hour entries must appear before 5-minute ones in the prompt.

system: [
  { type: 'text', text: HOUSE_RULES,
    cache_control: { type: 'ephemeral', ttl: '1h' } },   // long-lived first
  { type: 'text', text: retrievedDoc,
    cache_control: { type: 'ephemeral' } },              // then 5m
]

Is the 1-hour TTL ever the wrong tool in that 5–60 minute window? On Fable 5.1, usually. Because reads there are 0.025× rather than 0.1×, a read is nearly free and a miss is expensive relative to one — so instead of paying 2× on the write, stay on the 5-minute TTL and, while idle, resend the previous request with max_tokens: 0 shortly before the entry expires. That refreshes the timer and bills only a cheap read. It beats the 1-hour TTL unless pauses regularly approach an hour.

One more thing about reads that matters for capacity, not cost: on the Claude API they do not count toward input-token rate limits on most models (Haiku 3.5 is the documented exception). Keeping entries warm raises effective throughput as well as cutting the bill.

The floor

Below a per-model minimum, caching is skipped silently — no error, no warning, both usage counters at zero.

Note that it is not monotonic across generations. A 3,000-token prompt caches on Opus 5, Opus 4.8 and Sonnet 4.5, and silently will not on Opus 4.6 or Haiku 4.5. Opus 5 halved the Opus 4.8 minimum, so prompts that were previously too short to cache now create entries with no code change on your side.

What the response tells you

Three fields, and one of them is routinely misread:

console.log(response.usage.cache_read_input_tokens)      // served from cache, 0.1×
console.log(response.usage.cache_creation_input_tokens)  // written this request, 1.25×
console.log(response.usage.input_tokens)                 // NOT your total input

input_tokens is only the tokens after your last breakpoint. The total prompt is the sum of all three. With a 200,000-token cached document and a fifty-token question you will see a read of 200,000, a creation of 0, and input_tokens: 50 — and if you are metering credits or attributing cost per user, that three-way decomposition is exactly what you want to bill against, since the three components have different prices.

With 1-hour caching in play, usage.cache_creation breaks down further into ephemeral_5m_input_tokens and ephemeral_1h_input_tokens.

When reads collapse to zero, the usage fields tell you that the prefix broke but not where. Two ways to localise it. Log consecutive request bodies and diff the overlapping region — adjacent payloads in a growing conversation legitimately differ at the end, but the previous request’s prompt must reappear unchanged as a prefix of the next one. Strip the cache_control markers before diffing: the moving marker always differs between adjacent requests and is not the invalidator. Or opt into cache diagnostics (beta cache-diagnosis-2026-04-07), which does the comparison server-side and names the tier that diverged:

const res = await client.beta.messages.create({
  betas: ['cache-diagnosis-2026-04-07'],   // send on EVERY request —
  diagnostics: { previous_message_id: prevId },  // fingerprints are only kept
  ...request,                                     // for requests that carried it
})
console.log(res.diagnostics)   // model | system | tools | message history

A one-shot retrofit fails with previous_message_not_found, because nothing fingerprinted the earlier request.

And verify after every change, not once at setup. The costliest caching failure in production is a regression, not a bad first implementation: it worked when written, then someone added a dynamic field to the system prompt, or a feature that rewrites history, or a tool list that stopped being sorted — and nothing announced it. Requests keep succeeding. The bill is just higher. An integration test asserting that a second identical request shows cache_read_input_tokens > 0 is the cheapest standing check there is.

Warming up, and the fan-out trap

To kill the first-request latency penalty, send a max_tokens: 0 request at startup. The API runs prefill, writes the cache at your breakpoint, and returns immediately with an empty content array and stop_reason: "max_tokens". No output tokens are billed.

await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 0,
  system: [
    { type: 'text', text: SYSTEM_PROMPT,
      cache_control: { type: 'ephemeral' } },  // ← on the SHARED block
  ],
  messages: [{ role: 'user', content: 'warmup' }],  // never on this
})

That comment is the whole trick. Put the breakpoint on the last block shared with your real requests — and not via automatic caching, which would place it after the placeholder and key the entry to a message no real request will ever send. The request is rejected outright with stream: true, thinking.type: "enabled", output_config.format, a forced tool_choice, or inside a Batches request.

Pre-warming is worth it when first-request latency is user-visible, the prefix is large enough that a cold write is noticeably slow, and there is a moment before traffic to fire it. Skip it when traffic is continuous — real requests keep the cache warm on their own, and a separate warm call is a pure extra write.

Can I just fire N parallel requests over the same prefix? They will all miss. An entry becomes readable only once the first response begins streaming, so N simultaneous requests each pay full price and none can read what the others are still writing. Send one, await its first token, then fire the remaining N−1. The same arithmetic shapes multi-agent designs: N workers each assembling a slightly different prompt over the same context write N entries and read none of each other’s. Fewer lanes over a byte-identical shared prefix — or one worker making N sequential passes — turns those writes into reads.

The rest of the sharp edges

• There is no eviction API. You wait out the TTL or change the prefix. Plan system-prompt rollouts around that: a change goes live everywhere at once and every warm prefix in flight dies with it.
• Serialize deterministically. Go and Swift randomize map key order during JSON serialization, which silently changes the hash of tool_use blocks. Sort tools by name. Sort keys. This failure looks like nothing at all.
• Some blocks cannot be cached. Sub-content blocks such as citations (mark the top-level document instead), empty text blocks, and clear_at system messages — a marker on one of those is a 400, so put the breakpoint on the preceding user turn. Thinking blocks cannot carry cache_control directly, though they are cached as part of prior assistant turns.
• Thinking blocks interact with the messages cache, model-specifically. On Fable 5/5.1, Mythos 5/5.1, Opus 4.5 and later and Sonnet 4.6 and later, previous-turn thinking blocks are preserved. On earlier Opus and Sonnet models and every Haiku through 4.5, a plain user message following tool use strips previously-cached thinking blocks and everything after the first stripped block falls out of cache — visible as a cache_creation spike on exactly those turns.
• Unexplained writes are sometimes correct. Server tools such as web search insert their own 5-minute cache write after tool results when the request already uses caching. That is expected behaviour at a position you did not mark, not an invalidator.
• Some prompts should not be cached. If the first thousand tokens differ per request there is no reusable prefix, and adding a marker only buys you the write premium. Leave it off.

Building one

Everything above is a claim about a system you cannot open. So here is the mechanism rebuilt from nothing — small enough that you can hold all of it, and real enough that the failures from earlier sections reproduce in it.

Three pieces, and each one is a sentence from earlier turned into a rule:

• A fingerprint per position, each computed from the one before it. Position i’s hash takes position i−1’s hash and this block’s bytes. That is the causal mask expressed as a loop, and it is what makes a block have no identity independent of its prefix.
• A read that looks for stored entries, not for unchanged blocks. Walking backward, the test at each position is “is there something in the store under this hash” — never “did this block change”. Nothing anywhere records the second, which is exactly why stability alone never yields a hit.
• A write that bills the delta. What is stored is the prefix up to the breakpoint; what is charged is that minus whatever the read already covered. A growing conversation therefore writes the turn it just added, not the conversation — which is where the healthy-loop signature comes from.

Those three rules are the whole thing. The implementation is eighty lines with no dependencies, and the tests are the claims from earlier sections written as assertions: that a change poisons every position after it, that the same document under a different preamble is a different entry, that a breakpoint on a varying block writes forever and reads never, and that a turn longer than the lookback falls off the cliff.

// A prefix cache, reduced to the part that decides whether you get a hit.
//
// Pure and in-memory (code_guidelines.md §2): the store is passed in, nothing
// reads a clock, and the whole thing runs in a unit test. The post displays
// this file's own bytes, so the code a reader sees is the code the tests run.

/** A content block as the API sees it: text, and whether it carries a marker. */
export type Block = { readonly text: string; readonly tokens: number }

/** What a real cache holds here is the KV tensors. The size is all this model
 *  needs, because size is what you are billed for. */
export type Store = Map<string, number>

/** FNV-1a, 32 bits. Any stable hash works — what matters is that it is computed
 *  over bytes and is exact. There is no near-miss. */
function hash(seed: string, text: string): string {
  let h = 0x811c9dc5
  const input = `${seed}${text}`
  for (let i = 0; i < input.length; i += 1) {
    h ^= input.charCodeAt(i)
    h = Math.imul(h, 0x01000193)
  }
  return (h >>> 0).toString(16)
}

/** The lookback: a read checks at most this many positions back, counting the
 *  breakpoint itself as the first. */
export const LOOKBACK = 20

/** The fingerprint at each position, each one covering every block up to and
 *  including it. Cumulative because a block has no identity independent of what
 *  precedes it — which is the whole reason the unit is a prefix. */
export function prefixHashes(blocks: readonly Block[]): readonly string[] {
  const out: string[] = []
  let running = ''
  for (const b of blocks) {
    running = hash(running, b.text)
    out.push(running)
  }
  return out
}

export type Usage = {
  /** Served from the cache, billed at the read rate. */
  readonly read: number
  /** Processed and stored, billed at the write rate. */
  readonly written: number
  /** After the breakpoint. Ordinary input. */
  readonly uncached: number
}

/** One request. Mutates the store exactly the way a write does, and returns the
 *  three numbers the API reports back in `usage`. */
export function run(store: Store, blocks: readonly Block[], breakpoint: number): Usage {
  const hashes = prefixHashes(blocks)
  const upto = (i: number): number =>
    blocks.slice(0, i + 1).reduce((n, b) => n + b.tokens, 0)
  const total = upto(blocks.length - 1)

  if (breakpoint < 0 || breakpoint >= blocks.length) {
    return { read: 0, written: 0, uncached: total }
  }

  // READ. Walk backward looking for an entry an earlier request *wrote*. Not
  // for blocks that happen to be unchanged — nothing records those.
  let hit = -1
  for (let i = breakpoint; i >= 0 && breakpoint - i < LOOKBACK; i -= 1) {
    const h = hashes[i]
    if (h !== undefined && store.has(h)) { hit = i; break }
    }

  // WRITE. One entry, at the breakpoint, billing only the delta past the hit.
  const key = hashes[breakpoint]
  const read = hit >= 0 ? upto(hit) : 0
  let written = 0
  if (key !== undefined && !store.has(key)) {
    written = upto(breakpoint) - read
    store.set(key, upto(breakpoint))
  }
  return { read, written, uncached: total - upto(breakpoint) }
}

Run those rules over the same request three times, once with the breakpoint on the last stable block and once on the block carrying the date. The rows below are produced by running the implementation when this page is built, not typed in:

The right-hand columns never read anything and rewrite 21,040 tokens every time. That is the failure, reproduced from first principles — and reproducing it took no knowledge of Anthropic’s servers at all, because it follows from the first rule alone.

So is a non-prefix cache really impossible, or just not built? Not impossible — and this is the place the post’s main line is weakest, so it is worth being precise. The claim that survives is the exact one: you cannot reuse a block’s K/V at a different position and get bit-identical results, because those vectors are a function of the tokens before it. What you can do is reuse them approximately. Gim et al.’s PromptCache (MLSys 2024) precomputes attention states for reusable text segments, patches their position encodings when a segment lands somewhere new, and accepts the accuracy cost that follows; vLLM’s PagedAttention and SGLang’s RadixAttention take the other road — exact reuse, but organised as a paged or radix-tree-shaped prefix store so that many requests share whatever prefix they genuinely have in common. A production API sells exactness, so it ships the second kind. A research system free to trade a little quality for a lot of reuse can ship the first. “Prefix-only” is a consequence of the exactness requirement, not of the mathematics alone.

The whole thing, on one page

For the second reading, and for the moment six months from now when the hit rate drops and you need the table rather than the argument.

// automatic — one slot, last block
cache_control: { type: "ephemeral" }

// explicit — up to 4, per block
{ type: "text", text: DOC,
  cache_control: {
    type: "ephemeral", ttl: "1h" } }

cache_read_input_tokens      0.1×
cache_creation_input_tokens  1.25×
input_tokens                 1×  ← tail only

total prompt = sum of all three

max_tokens: 0
→ content: [], stop_reason:
  "max_tokens", cache written

Check yourself

Quick reference

Every abbreviation this post used, expanded, plus the neighbouring terms you will meet in the papers below. Ordered for lookup, not for reading.

Serving-side internals are inferred from the published work above; Anthropic does not document its cache implementation. The API-visible contract — every number, every rule — comes from the first reference.