Linear attention
The efficiency lineage · 2020
The idea
Section titled “The idea”Reassociate the same computation - (QK)V becomes Q(KV) - and the past compresses into a state that never grows. Scrub the tokens below: each one folds into S as a tile, and S never gets bigger.
Verdict
Section titled “Verdict”What it fixedReplace the growing cache with one fixed-size state S - memory flat at any context.
What it still costsAssociations interfere and exact retrieval is gone. For years this made it a losing trade.
The story, animated
Section titled “The story, animated”Same three symbols, two ways to bracket them - watch the wall and the box arrive at the exact same answer.
Try it
Section titled “Try it”Where it sits
Section titled “Where it sits”Worth having first: The memory wall
Unlocks: Delta rule · Sliding window with sinks
Episode 2 · The Linear Bet