Why the cache is a prefix

Why the unit is a prefix, and can never be a block

Attention is causal. Token i attends to tokens 1 through i and to nothing to its right — that is what the causal mask enforces, and it is the reason a language model can be trained on all positions at once. Two things follow immediately.

First, the work is a pure function of the prefix. The K/V vectors at position i depend on tokens 1..i and on nothing else — not on what comes after, not on the question you eventually ask. A pure function of its inputs is exactly what a cache can stand in for, and purity is what makes the cache correct rather than merely fast.

Second, and this is the load-bearing half: the cacheable unit is a prefix, not a block. A block of text has no position-independent representation. Change one byte anywhere before it and every vector from that point on is different.

But I send the same document every time — why can’t the server recognise it wherever it lands? Because “the same document” is not the same computation. Its vectors are a function of the 12,000 tokens of system prompt sitting in front of it. Caching it independently of that would mean caching a function of an argument you never recorded, and returning it later would be returning a wrong answer quickly. A block-level cache is not a harder engineering problem; it is an incorrect one.

Then why a hash, rather than comparing the text? Because you cannot diff 200,000 tokens on every request for free. The system fingerprints the cumulative prefix and compares fingerprints. That makes the match exact and byte-level: one space, one reordered JSON key, one tool description reworded, and it is a different prefix. There is no fuzzy match, and there cannot be one for the same reason as above.

And why does the order tools → system → messages exist? A prefix requires a total order over the request, and the request is a set of fields. Something has to impose one. That single choice, frozen, is what produces the entire invalidation hierarchy later on: it is why editing a tool costs you the whole conversation, and why appending a message never costs you the tools.

What is actually in the cache

The K/V tensors, plus the hash that identifies them. Held in memory, not written to disk, never used for training, and isolated: per workspace on the Claude API, Claude Platform on AWS and Microsoft Foundry, and per organization on Amazon Bedrock and Vertex AI. Never across organizations.

That isolation is worth knowing before you debug a hit rate. Traffic for one prompt split across two workspaces writes and reads two separate entries and looks, from the usage fields, exactly like a broken prefix. Caches are also model-scoped — a fact that quietly prices every multi-model design, and which we will come back to.

Anatomy of a cached request

The request is assembled in that fixed order, and a cache_control marker is a cut line through the stack. Everything at or above the line is one cache entry, keyed by a hash of all of it together. Everything below is ordinary uncached input.

There are two ways to place one.

Automatic — a single cache_control at the top level of the request body. The system applies the breakpoint to the last cacheable block and moves it forward as the conversation grows, so you never touch markers again:

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16_000,
  cache_control: { type: 'ephemeral' },   // ← top level, not on a block
  system: houseRules,
  messages: history,
})

Explicit — a marker on individual content blocks, up to four per request. Use these when different sections change at different rates:

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16_000,
  tools,                                    // changes on deploy
  system: [
{ type: 'text', text: HOUSE_RULES },
{ type: 'text', text: retrievedDoc,
  cache_control: { type: 'ephemeral' } }, // ← caches tools + both system blocks
  ],
  messages: history,
})

The two compose. The automatic breakpoint consumes one of the four slots, and there are two documented ways to get a 400 out of combining them: all four slots already taken by explicit markers, and an explicit marker on the last block whose TTL differs from the top-level field’s. Declaring breakpoints is otherwise free — you are billed on what is written and read, never on how many markers you placed.

Write here; read backwards

Two rules govern whether you get a hit, and the second is the one that surprises people.

1. A write happens only at your breakpoint. Marking a block creates exactly one entry — the hash of the cumulative prefix ending there. No entries are created for earlier positions.
2. A read walks backward. If the hash at your breakpoint does not match, the system steps back one position at a time, checking each earlier position for an entry a previous request wrote. It is looking for prior writes, not for content that happens to be stable.

Why not just write an entry at every block boundary? Because a write is real memory and real work. The K/V cache for a 20,000-token prefix is large, and writing one at every position would multiply the store by the number of blocks in service of a benefit only one position will ever deliver. The design instead asks you to declare where you expect to resume, and honours exactly that. The four-breakpoint limit is the same trade seen from the other side.