Glossary

Every abbreviation and outside term the post uses, expanded in a line and ordered for lookup.

ContentsAppendix

Quick reference

Every abbreviation this post used, expanded, plus the neighbouring terms you will meet in the papers below. Ordered for lookup, not for reading.

Glossary20 terms, A–Z

API: Application Programming Interface — here, the HTTP endpoint you send a prompt to.
block: One content item in a request: a text span, an image, a tool definition, a tool result.
breakpoint: A cache_control marker. The cut line that says "cache everything above here".
causal mask: The rule that a token may only attend to tokens at or before its own position.
CoT: Chain of thought — the model’s intermediate reasoning tokens.
eviction: Removing an entry from a cache. There is no API for it here; entries expire.
fingerprint / hash: A short value derived from bytes, used to test exact equality cheaply.
K/V, KV cache: The key and value vectors attention computes per token per layer. What is actually cached.
lookback: How many positions back a read searches for an entry. 20 here.
MTok: One million tokens. The unit API pricing is quoted in.
PagedAttention: vLLM’s technique for storing the KV cache in fixed-size pages, so prefixes can be shared.
prefill: Processing the input prompt, before any output token is generated. The latency floor.
prefix: Every token from position zero up to some point. The unit of caching.
RadixAttention: SGLang’s prefix cache, organised as a radix tree so many requests share common prefixes.
RAG: Retrieval-Augmented Generation — putting fetched documents into the prompt.
TTFT: Time to first token. What prefill dominates.
TTL: Time to live. How long a cache entry survives without being touched.
token: The unit a model reads and bills in; roughly a word-piece.
tool_use / tool_result: The blocks carrying a model’s call to a tool and the value you returned.
write / read (cache): Storing a prefix (1.25× or 2× base input) and serving one back (0.1×).