Attention Computation

Attention by shapes · 2 of 11

The idea

Once you have Q, K and V, the rest is arithmetic. S = Kt × Q puts a score on every (key, query) pair. Softmax (the fn on the diagram) turns each column into shares that sum to 1. F = V × A blends the values — not a pick of one neighbour, a weighted mix.

The shapes

Grow the sequence length: S stays square. There are no new learned weights in this step. Softmax scale and the causal mask live on the Attention scrollytelling page; here the lesson is the matrices.

Live diagram

Next: Self Attention