Skip to content
AI, Explained

Speculative decoding

Running the model & the world · 2023

The idea

Section titled “The idea”

Make the expensive model a checker, not a typist.

PaperFast Inference from Transformers via Speculative DecodingLeviathan, Kalman, Matias · Google, ICML · 2022arXiv:2211.17192 →

The story, animated

Section titled “The story, animated”

A small model drafts. The big one checks in one pass. Several tokens per expensive step — until the drafter starts missing.

Watch

Section titled “Watch”
Speculative decoding — a small model drafts, the big one checks · 20s, companion clip.

Verdict

Section titled “Verdict”
What it fixedA small model drafts several tokens; the big model verifies them in one pass - several tokens per big-model step, distribution exactly preserved.
What it still costsNeeds a drafter that agrees often - when acceptance drops, the speedup evaporates.

Where it sits

Section titled “Where it sits”

Worth having first: Sampling