Embeddings

Foundations

The idea

Each token becomes a long list of numbers - 12,288 of them in GPT-3. The list of numbers and the arrow in space are the same object seen two ways.

This table of columns is the literal junction in the architecture: token ids stop being labels here and become the vectors every later block reads and rewrites (ch. 0:23-0:24). Everything downstream - attention, the MLPs, the whole map - operates on what leaves this lookup.

The story, animated

A word becomes a column of numbers, and that column is a point in space. Two views of one object.

Watch

"Where The Word Ends Up", part A - the position · 125s, from the film series.

Try it

Where it sits

Worth having first: Tokens

Unlocks: Directions & dot product · Softmax & temperature

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).