Multi-Head Attention

Attention by shapes · 7 of 11

The idea

One head watches one kind of pattern. Run several in parallel — each with its own Wq, Wk, Wv — then concatenate their outputs and project with Wo.

PaperAttention Is All You NeedVaswani et al. · NeurIPS · 2017arXiv:1706.03762 →

Multi-head attention, specifically, is this paper’s named contribution — one of two mechanisms (with scaled dot-product attention) the abstract calls out by name.

The shapes

Three full heads on the left; concat stacks the F’s; Wo maps back to the output dim. The Queries, keys, values page teaches why heads; here you watch the matrices.

Live diagram

Next: Fused QKV