inqizitblog

Theme

Linen — The house face. Warm paper, rust ink, and nothing shouting.

ai · llm ·

Prompt caching, from the causal mask outwardAfter the parts

Recap

Every number, rule and trap from the post on one surface, laid out to be recalled at a glance rather than read.

IndexFull pageRecap

ContentsAfter the parts
  1. Start hereIntroduction
  2. Part 1Why the cache is a prefix
  3. Part 2Where the breakpoint goes
  4. Part 3What it costs, and for how long
  5. Part 4Running it
  6. After the partsRecap
    1. 18The whole thing, on one page
  7. After the partsPractice
  8. AppendixGlossary
  9. AppendixReferences

IndexFull page

The whole thing, on one page

For the second reading, and for the moment six months from now when the hit rate drops and you need the table rather than the argument.

Prompt caching on one page

Every rule below is a consequence of the first panel. Figures are Claude Opus 5 unless noted.

The invariant

Caching is a prefix match. Attention is causal, so a token’s K/V vectors are a function of everything before it and nothing after. Any byte change anywhere in the prefix invalidates everything from that point on. There is no fuzzy match and there cannot be one.

Render order: tools, then system, then messagestoolssystemmessagesposition 0volatile content belongs here →

Syntax

// automatic — one slot, last block
cache_control: { type: "ephemeral" }

// explicit — up to 4, per block
{ type: "text", text: DOC,
  cache_control: {
    type: "ephemeral", ttl: "1h" } }
  • Max 4 breakpoints; automatic uses one.
  • 1h entries must precede 5m entries.
  • Declaring breakpoints is free — you are billed on writes and reads.

Resolution

  • Write happens only at your breakpoint: one entry, hashing the whole prefix above it. Nothing is written at earlier positions.
  • Read walks backward from the breakpoint looking for entries prior requests wrote — not for content that happens to be stable.
  • ≤ 20 positions. A run of consecutive tool_use blocks counts as one, and so does a run of tool_result blocks.
  • An entry is readable only once the first response begins streaming.

Price (Opus 5, $5/MTok in)

base input1×$5.00
write, 5m ttl1.25×$6.25
write, 1h ttl2×$10.00
read0.1×$0.50
  • Break-even: 2 requests at 5m (1.25 + 0.1 < 2), 3 at 1h (2 + 0.2 < 3).
  • Reads are 0.025× on Claude Fable 5.1 ($0.25/MTok); unconfirmed for Mythos 5.1.
  • Stacks with the Batch API discount.

Minimum cacheable prefix

Opus 5, Fable 5 / 5.1, Mythos 5 / 5.1512
Opus 4.8, Sonnet 5 / 4.6 / 4.5, Opus 4.1 / 4, Sonnet 41,024
Opus 4.7, Mythos Preview, Haiku 3.52,048
Opus 4.6, Opus 4.5, Haiku 4.54,096

Below the minimum, caching is skipped with no error. Not monotonic across generations: a 3K prompt caches on Opus 5 and silently will not on Opus 4.6.

TTL and the clock

start-to-start gapttl
under 5 min5m — every request refreshes it free
5–60 min1h — the only window the 2× write repays
over an hourneither; re-warm or accept the miss

The clock starts when the request starts, not when the response ends: a 4-minute generation leaves about 1 minute of a 5-minute entry. Reads do not count toward input-token rate limits on most models (Haiku 3.5 excepted).

Invalidation — what survives

changetoolssystemmessages
message content editedkeptkeptlost
tool_choice, imageskeptkeptlost
system prompt contentkeptlostlost
speed, web search, citationskeptlostlost
thinking / effortmodel-specificmodel-specificlost
tool definitions add/remove/reorderlostlostlost
model switchlostlostlost

Escape hatches

  • System prompt change → a {role:"system"} message in messages[]. Opus 5, Opus 4.8, Fable 5/5.1, Mythos 5/5.1. No beta header. Not Sonnet 5.
  • Tool add/remove → tool_addition / tool_removal blocks. Opus 5 onward, beta mid-conversation-tool-changes-2026-07-01.
  • Effort change → system message with content: [] and output_config. Fable 5.1, Mythos 5.1, Opus 5, beta mid-conversation-output-config-2026-07-01.
  • Per-turn reminder → clear_at: "next_user_message", left in the transcript forever.
  • Model switch → none.

Reading usage

cache_read_input_tokens      0.1×
cache_creation_input_tokens  1.25×
input_tokens                 1×  ← tail only

total prompt = sum of all three

Healthy loop: read grows turn over turn, creation ≈ the last turn’s delta, input_tokens is just the tail. Creation near full conversation size every request means the prefix is being rewritten upstream — or the turn exceeded the 20-position lookback.

usage.cache_creation splits by TTL into ephemeral_5m_input_tokens / ephemeral_1h_input_tokens.

Pre-warming

max_tokens: 0
→ content: [], stop_reason:
  "max_tokens", cache written
  • Breakpoint goes on the shared block, never the placeholder message — and never via automatic caching, which would key the entry to the placeholder.
  • Rejected with stream: true, thinking.type:"enabled", output_config.format, forced tool_choice, or inside Batches.
  • Skip it when traffic is continuous, the prefix is small, or the prefix varies per user.

Silent invalidators

  • Date.now() / uuid() anywhere in the prefix.
  • JSON serialized without sorted keys — Go and Swift randomize map order, which silently changes tool_use hashes.
  • Conditional system sections: every flag combination is its own prefix.
  • Per-user tool sets or session IDs in the system prompt: no cross-user sharing.
  • A fork (summarizer, sub-agent) that rebuilds system/tools instead of copying the parent’s verbatim.
  • Workspace split: caches are per workspace on the Claude API, Claude Platform on AWS and Foundry; per organization on Bedrock and Vertex.

Limits and non-goals

  • No manual eviction. Wait out the TTL or change the prefix — plan system-prompt rollouts around it.
  • Sub-content blocks (citations) are not cacheable; mark the top-level document instead.
  • Empty text blocks are not cacheable. Neither is a clear_at system message (a marker on one is a 400).
  • Thinking blocks cannot carry cache_control, but are cached as part of prior assistant turns.
  • N parallel requests with the same prefix all miss. Send one, await its first token, then fan out.
  • Server tools such as web search insert their own 5-minute write after tool results. Expected, not a bug.

When it breaks, in order

  • Check cache_read_input_tokens across two identical requests. Zero means nothing hit.
  • Check the prefix clears the model’s minimum.
  • Log consecutive request bodies and diff the overlap. Strip cache_control first — the moving marker always differs and is not the invalidator.
  • Or use cache diagnostics (beta cache-diagnosis-2026-04-07), which names the divergence server-side. Send the header on every request: a one-shot retrofit fails with previous_message_not_found.
  • Assert it in CI. The expensive failure is a regression six months after it worked.
cached / readwritteninvalidated / traprecomputed at full price