Running it

Warming up, and the fan-out trap

To kill the first-request latency penalty, send a max_tokens: 0 request at startup. The API runs prefill, writes the cache at your breakpoint, and returns immediately with an empty content array and stop_reason: "max_tokens". No output tokens are billed.

await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 0,
  system: [
{ type: 'text', text: SYSTEM_PROMPT,
  cache_control: { type: 'ephemeral' } },  // ← on the SHARED block
  ],
  messages: [{ role: 'user', content: 'warmup' }],  // never on this
})

That comment is the whole trick. Put the breakpoint on the last block shared with your real requests — and not via automatic caching, which would place it after the placeholder and key the entry to a message no real request will ever send. The request is rejected outright with stream: true, thinking.type: "enabled", output_config.format, a forced tool_choice, or inside a Batches request.

Pre-warming is worth it when first-request latency is user-visible, the prefix is large enough that a cold write is noticeably slow, and there is a moment before traffic to fire it. Skip it when traffic is continuous — real requests keep the cache warm on their own, and a separate warm call is a pure extra write.

Can I just fire N parallel requests over the same prefix? They will all miss. An entry becomes readable only once the first response begins streaming, so N simultaneous requests each pay full price and none can read what the others are still writing. Send one, await its first token, then fire the remaining N−1. The same arithmetic shapes multi-agent designs: N workers each assembling a slightly different prompt over the same context write N entries and read none of each other’s. Fewer lanes over a byte-identical shared prefix — or one worker making N sequential passes — turns those writes into reads.

The rest of the sharp edges

• There is no eviction API. You wait out the TTL or change the prefix. Plan system-prompt rollouts around that: a change goes live everywhere at once and every warm prefix in flight dies with it.
• Serialize deterministically. Go and Swift randomize map key order during JSON serialization, which silently changes the hash of tool_use blocks. Sort tools by name. Sort keys. This failure looks like nothing at all.
• Some blocks cannot be cached. Sub-content blocks such as citations (mark the top-level document instead), empty text blocks, and clear_at system messages — a marker on one of those is a 400, so put the breakpoint on the preceding user turn. Thinking blocks cannot carry cache_control directly, though they are cached as part of prior assistant turns.
• Thinking blocks interact with the messages cache, model-specifically. On Fable 5/5.1, Mythos 5/5.1, Opus 4.5 and later and Sonnet 4.6 and later, previous-turn thinking blocks are preserved. On earlier Opus and Sonnet models and every Haiku through 4.5, a plain user message following tool use strips previously-cached thinking blocks and everything after the first stripped block falls out of cache — visible as a cache_creation spike on exactly those turns.
• Unexplained writes are sometimes correct. Server tools such as web search insert their own 5-minute cache write after tool results when the request already uses caching. That is expected behaviour at a position you did not mark, not an invalidator.
• Some prompts should not be cached. If the first thousand tokens differ per request there is no reusable prefix, and adding a marker only buys you the write premium. Leave it off.

Building one

Everything above is a claim about a system you cannot open. So here is the mechanism rebuilt from nothing — small enough that you can hold all of it, and real enough that the failures from earlier sections reproduce in it.

Three pieces, and each one is a sentence from earlier turned into a rule:

• A fingerprint per position, each computed from the one before it. Position i’s hash takes position i−1’s hash and this block’s bytes. That is the causal mask expressed as a loop, and it is what makes a block have no identity independent of its prefix.
• A read that looks for stored entries, not for unchanged blocks. Walking backward, the test at each position is “is there something in the store under this hash” — never “did this block change”. Nothing anywhere records the second, which is exactly why stability alone never yields a hit.
• A write that bills the delta. What is stored is the prefix up to the breakpoint; what is charged is that minus whatever the read already covered. A growing conversation therefore writes the turn it just added, not the conversation — which is where the healthy-loop signature comes from.

Those three rules are the whole thing. The implementation is eighty lines with no dependencies, and the tests are the claims from earlier sections written as assertions: that a change poisons every position after it, that the same document under a different preamble is a different entry, that a breakpoint on a varying block writes forever and reads never, and that a turn longer than the lookback falls off the cliff.

// A prefix cache, reduced to the part that decides whether you get a hit.
//
// Pure and in-memory (code_guidelines.md §2): the store is passed in, nothing
// reads a clock, and the whole thing runs in a unit test. The post displays
// this file's own bytes, so the code a reader sees is the code the tests run.

/** A content block as the API sees it: text, and whether it carries a marker. */
export type Block = { readonly text: string; readonly tokens: number }

/** What a real cache holds here is the KV tensors. The size is all this model
 *  needs, because size is what you are billed for. */
export type Store = Map<string, number>

/** FNV-1a, 32 bits. Any stable hash works — what matters is that it is computed
 *  over bytes and is exact. There is no near-miss. */
function hash(seed: string, text: string): string {
  let h = 0x811c9dc5
  const input = `${seed}${text}`
  for (let i = 0; i < input.length; i += 1) {
    h ^= input.charCodeAt(i)
    h = Math.imul(h, 0x01000193)
  }
  return (h >>> 0).toString(16)
}

/** The lookback: a read checks at most this many positions back, counting the
 *  breakpoint itself as the first. */
export const LOOKBACK = 20

/** The fingerprint at each position, each one covering every block up to and
 *  including it. Cumulative because a block has no identity independent of what
 *  precedes it — which is the whole reason the unit is a prefix. */
export function prefixHashes(blocks: readonly Block[]): readonly string[] {
  const out: string[] = []
  let running = ''
  for (const b of blocks) {
    running = hash(running, b.text)
    out.push(running)
  }
  return out
}

export type Usage = {
  /** Served from the cache, billed at the read rate. */
  readonly read: number
  /** Processed and stored, billed at the write rate. */
  readonly written: number
  /** After the breakpoint. Ordinary input. */
  readonly uncached: number
}

/** One request. Mutates the store exactly the way a write does, and returns the
 *  three numbers the API reports back in `usage`. */
export function run(store: Store, blocks: readonly Block[], breakpoint: number): Usage {
  const hashes = prefixHashes(blocks)
  const upto = (i: number): number =>
    blocks.slice(0, i + 1).reduce((n, b) => n + b.tokens, 0)
  const total = upto(blocks.length - 1)

  if (breakpoint < 0 || breakpoint >= blocks.length) {
    return { read: 0, written: 0, uncached: total }
  }

  // READ. Walk backward looking for an entry an earlier request *wrote*. Not
  // for blocks that happen to be unchanged — nothing records those.
  let hit = -1
  for (let i = breakpoint; i >= 0 && breakpoint - i < LOOKBACK; i -= 1) {
    const h = hashes[i]
    if (h !== undefined && store.has(h)) { hit = i; break }
    }

  // WRITE. One entry, at the breakpoint, billing only the delta past the hit.
  const key = hashes[breakpoint]
  const read = hit >= 0 ? upto(hit) : 0
  let written = 0
  if (key !== undefined && !store.has(key)) {
    written = upto(breakpoint) - read
    store.set(key, upto(breakpoint))
  }
  return { read, written, uncached: total - upto(breakpoint) }
}

Run those rules over the same request three times, once with the breakpoint on the last stable block and once on the block carrying the date. The rows below are produced by running the implementation when this page is built, not typed in:

The right-hand columns never read anything and rewrite 21,040 tokens every time. That is the failure, reproduced from first principles — and reproducing it took no knowledge of Anthropic’s servers at all, because it follows from the first rule alone.

So is a non-prefix cache really impossible, or just not built? Not impossible — and this is the place the post’s main line is weakest, so it is worth being precise. The claim that survives is the exact one: you cannot reuse a block’s K/V at a different position and get bit-identical results, because those vectors are a function of the tokens before it. What you can do is reuse them approximately. Gim et al.’s PromptCache (MLSys 2024) precomputes attention states for reusable text segments, patches their position encodings when a segment lands somewhere new, and accepts the accuracy cost that follows; vLLM’s PagedAttention and SGLang’s RadixAttention take the other road — exact reuse, but organised as a paged or radix-tree-shaped prefix store so that many requests share whatever prefix they genuinely have in common. A production API sells exactness, so it ships the second kind. A research system free to trade a little quality for a lot of reuse can ship the first. “Prefix-only” is a consequence of the exactness requirement, not of the mathematics alone.