Superposition

Inside the blocks

The idea

At exactly 90 degrees a space holds only as many ideas as dimensions. Relax to nearly perpendicular and capacity explodes - the price is a small overlap that never reads exactly zero. Try the angle slider: the floor moves.

A frame for what all this capacity stores: the weights are a lossy zip of the training text - reconstructive, gist-first, emphatically not an archive (ch. 0:50). The exception proves it: documents seen many times (Wikipedia is deliberately oversampled, roughly ten exposures) can come back near-verbatim, memorisation breaking through the compression (ch. 0:53).

Try it

Where it sits

Worth having first: Directions & dot product

Chapter: how facts are stored (parts bin D9)

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).