inqizitblog

Theme

Linen — The house face. Warm paper, rust ink, and nothing shouting.

ai · llm ·

Prompt caching, from the causal mask outward

Almost everything written about prompt caching is a list of rules. There is only one rule. A transformer’s work on a token depends on everything before it and nothing after it — and every breakpoint, every price, every trap in this post is deduction from that sentence.

Written against the Claude API. Figures are Claude Opus 5 unless stated.

  • prompt-caching
  • kv-cache
  • inference
  • claude-api
  • cost
  • latency
  • agents

IndexFull pageRecap

Contents9 pages
  1. Start hereIntroduction
  2. Part 1Why the cache is a prefix
  3. Part 2Where the breakpoint goes
  4. Part 3What it costs, and for how long
  5. Part 4Running it
  6. After the partsRecap
  7. After the partsPractice
  8. AppendixGlossary
  9. AppendixReferences

IndexFull page

Start reading · IntroductionRead it all on one pageShort on time? The recap

  1. Start here

    Introduction

    What an agent loop pays to resend an unchanged prefix every turn, and the one fact about attention this post derives the fix from.

    1. 01The bill you are already paying
    2. 02What this post does

  2. Part 1 of 4

    Why the cache is a prefix

    Why the causal mask makes a prefix the only cacheable unit, what the cache holds, where a breakpoint cuts, and why a miss walks backward.

    1. 03Why the unit is a prefix, and can never be a block
    2. 04What is actually in the cache
    3. 05Anatomy of a cached request
    4. 06Write here; read backwards

  3. Part 2 of 4

    Where the breakpoint goes

    The placement that bills more than no caching, why the last block suits a growing loop, and how four breakpoints and the cascade divide a prompt.

    1. 07The placement that costs more than no caching at all
    2. 08When “the last block” is exactly right
    3. 09Four breakpoints, four clocks
    4. 10The cascade, and the doors out of it

  4. Part 3 of 4

    What it costs, and for how long

    Break-even in two requests, a TTL that starts with the request, a per-model floor below which nothing caches, and the usage fields that show it.

    1. 11The arithmetic
    2. 12The clock starts earlier than you think
    3. 13The floor
    4. 14What the response tells you

  5. Part 4 of 4

    Running it

    Warming a cache without the fan-out trap, the sharp edges that break it silently, and the whole mechanism rebuilt from scratch in eighty lines.

    1. 15Warming up, and the fan-out trap
    2. 16The rest of the sharp edges
    3. 17Building one

  6. After the parts

    Recap

    Every number, rule and trap from the post on one surface, laid out to be recalled at a glance rather than read.

    1. 18The whole thing, on one page

  7. After the parts

    Practice

    Cards that walk the argument one step at a time: pick the answer from first principles, and a miss names the section to go back to.

    1. 19Check yourself

  8. Appendix

    Glossary

    Every abbreviation and outside term the post uses, expanded in a line and ordered for lookup.

  9. Appendix

    References

    The primary sources behind every number and claim, each with what you will find there, including the paper that complicates the argument.