Quantization
Running the model & the world
The idea
Section titled “The idea”The same numbers, stored smaller.
The practical payoff is local inference: pair a distilled (smaller) model with weights stored below native precision and frontier-adjacent capability fits on an ordinary laptop GPU - the how-do-I-run-this-at-home path (ch. 3:19-3:21).
Verdict
Section titled “Verdict”What it fixedWeights and cache in fewer bits - memory and bandwidth fall by the bit-width ratio, often the difference between fitting and not fitting.
What it still costsPrecision at the margins; push too far and quality pays.
The story, animated
Section titled “The story, animated”The same numbers, stored narrower - watch the same register shrink, and the difference between fitting and not.
Where it sits
Section titled “Where it sits”Worth having first: KV cache
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).