Context moves vectors
Foundations
The idea
Section titled “The idea”A word’s vector is not fixed: context pushes it around in space, one nudge per clue. The word is moved, never replaced - meaning accumulates.
The window has a size for economic reasons, not architectural ones: attending over more context simply costs more compute per token, so the window is cropped where the bill says stop (ch. 0:20).
Two capabilities live entirely inside that window. Content in context is exact working memory - directly readable on every pass, unlike the lossy recollection baked into weights (ch. 1:32; the closing page What it can’t remember turns on this). And a model can pick up a brand-new pattern purely from examples in the prompt and continue it - in-context learning, no weight changed (ch. 0:56-0:57).
This is GPT-3’s paper. Its argument: push a model to 175B parameters and few-shot in-context learning - examples in the prompt, zero gradient updates - starts rivalling fine-tuned task-specific models. In-context learning isn’t a trick this site invented a name for; it’s the specific, scale-dependent capability this paper discovered.
The story, animated
Section titled “The story, animated”Each earlier word gives “it” a small nudge. The word is moved, never replaced — then the window is cropped where the bill says stop.
Where it sits
Section titled “Where it sits”Worth having first: Directions & dot product
Unlocks: Attention
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).