Self Attention

Attention by shapes · 3 of 11

The idea

Pull back from one querier. Every token in the sequence asks; every token offers a key and a value. The same N people play both roles, so the score matrix is square: N × N.

The shapes

Left: QKV from X. Right: the score block. Copy edges show Q and V feeding the compute. Grow N — S grows as N×N. This is what GPT-style decoder layers run (with a causal mask the shape diagram omits).

Live diagram

Next: Cross Attention