The next-token objective

Foundations

The idea

The entire model trains on one task: given the text so far, predict the next token. Every capability you have seen falls out of doing this one thing extremely well at scale.

One update, worked small: before training touches an example, the correct next token might carry ~3% probability; a single nudge of the weights lifts it toward ~4% while wrong candidates give ground. Training is that nudge repeated across trillions of tokens (ch. 0:17-0:19).

And the nudges are not sequential: every position in every batch is updated in parallel, all at once - the wrong mental image is a model reading left to right; the right one is a whole page graded simultaneously (ch. 0:19-0:20).

The story, animated

Cover the next chip — a distribution, not a word. One nudge lifts 3% toward 4%. Then every blank on the page is graded at once.

Watch

The next-token objective — a distribution, then a 3% to 4% nudge, graded at once · 18s, companion clip.

Where it sits

Worth having first: Tokens · Softmax & temperature

Unlocks: Backpropagation · Pretraining · Sampling

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).