Queries, keys, values & heads
Attention & memory
The idea
Section titled “The idea”Each token makes three things of itself: a query (what am I looking for?), a key (what do I offer?), a value (what do I hand over if picked?). Heads run many of these matchmakings in parallel, each in its own small subspace.
The story, animated
Section titled “The story, animated”The whole attention story, told with a pronoun and a library - one slide per idea, playing as you scroll. What happens to the keys and values afterwards continues on KV cache.
Where it sits
Section titled “Where it sits”Worth having first: Attention