KV cache

Attention & memory

The idea

Generation would recompute every key and value for every new token - so models cache them. Ask once, remember twice. The price: the cache grows with every token of context.

A warning that guards this whole page: the network itself is stateless - a fixed function from context to next-token, with nothing carried across time. Karpathy reaches for a biological analogy and immediately bounds it for exactly this reason (ch. 0:24-0:25). The cache stores keys and values already computed; it does not summarise, condense, or remember anything. Nothing is merged.

The story, animated

Picks up where Queries, keys, values & heads leaves off: attention has answered its question - the cache is about what to keep. One slide per idea, playing as you scroll.

Watch

"Ask Once, Remember Twice" · 68s, from the film series.

"Nothing Moved" - the full explanation (prefill/decode, the O(n) vs O(n^2) shape) · 302s, from the film series.

Try it

Where it sits

Worth having first: Queries, keys, values & heads

Unlocks: The memory wall · Quantization · The serving layer

Episode 1 · The Memory Problem

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).