Prompt caching, from the causal mask outwardAppendix
References
The primary sources behind every number and claim, each with what you will find there, including the paper that complicates the argument.
ContentsAppendix
References
Primary sources first, each with what you will find there — including at least one that complicates the argument above.
References6 sources
- Prompt caching — Claude Platform Docs — the primary source for every API number here: multipliers, per-model minimums, TTL semantics, the usage fields, the breakpoint and lookback rules. Read it before trusting any third-party summary, this one included; the minimums in particular have moved between model generations.
- Vaswani et al., Attention Is All You Need (2017) — where the causal mask the whole mechanism rests on is defined. Section 3.2.3 is the half-page that makes the first section of this post something you can derive rather than something you remember.
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — how the KV cache is actually stored on the serving side, and why paging it is what makes sharing a prefix between requests cheap. The best public answer to “what is on the other end of that hash”.
- Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention: the prefix store as a radix tree with LRU eviction. Read it for the shape of the data structure the backward walk is walking over.
- Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference (MLSys 2024) — the paper that complicates this post. It reuses attention states for text segments at positions they were not computed at, which is precisely what the prefix rule says you cannot do. The resolution is in the cross-question above: it buys reuse by giving up exactness, and a billed API will not make that trade for you.
- Anthropic Cookbook — runnable notebooks for the request shapes described here, when you would rather read a working script than a specification.
Serving-side internals are inferred from the published work above; Anthropic does not document its cache implementation. The API-visible contract — every number, every rule — comes from the first reference.