Attention by shapes

The word-token pages teach what attention is for. This track teaches the shapes: every matrix labeled with live dimensions, so the picture is the arithmetic. Spec from Prof. Tom Yeh’s AI by Hand Attention library (byhand.ai) — redrawn in this site’s palette; dog metaphors left behind.

Worth having first: Queries, keys, values & heads.

1. QKV Projection — Three learned maps turn one embedding matrix into questions, labels, and payloads.
2. Attention Computation — Scores, shares, and the blended output — no new weights.
3. Self Attention — Everyone asks, everyone can be found — S is N×N.
4. Cross Attention — Queries from one sequence; keys and values from another.
5. Self vs Cross — Same X asks both times; only where K and V come from changes the shape.
6. Shared KV — Only Wq — Q is reused as the key.
7. Multi-Head Attention — Several full heads, concat, then Wo.
8. Fused QKV — One Wqkv, sliced into per-head Q/K/V.
9. Single vs Multi-Head — Same width: H× more dots, same V×A multiplies.
10. MQA (shapes) — One full KV drawer; the other heads only bring a query.
11. GQA (shapes) — Groups of heads share a K/V set — the dial between MHA and MQA.