Shared KV

Attention by shapes · 6 of 11

The idea

Before heads start sharing drawers, there is a simpler move: project once with Wq and reuse Q as K (Kt = Qᵀ). Values still carry the payload.

Why it is here

This is the single-head seed of the idea MQA will push across heads: stop paying for a fresh K (and later V) every time. The MQA page tells the memory-bill story; this page shows the shape.

Live diagram

Next: Multi-Head Attention