Directions & dot product

Foundations

The idea

In that space, directions can carry meaning. The dot product is the meter: how much of this idea is in this vector. Drag E below and watch the readouts - they are the shadow’s length, not the arrow’s.

One honest limit, secondhand from interpretability work Karpathy relays: a direction can encode something orthogonal to what a prompt is about. A famous wrong comparison (9.11 vs 9.9) may trip a feature associated with chapter-and-verse numbering, activated by nothing but the formatting (ch. 2:06, flagged as unverified in the talk itself). Directions carry meaning - not always the meaning you assume.

The story, animated

The dot product is a shadow’s length, not a hug. Watch the shadow form, then shrink when the arrows turn sideways.

Watch

"Where The Word Ends Up", part B - the directions · 138s, from the film series.

Try it

Try: drag the white E onto a coloured direction, then orbit around it - the readouts move only when E does.

loading the 3D space…

squeeze the angle - watch the overlap floor rise

84°

reset

E stays where you leave it; aligned, its readout reads 1.00 while the other two pin at the floor. The dashed drop is E's shadow on Michael's axis - the readout IS that shadow's length, not the arrow's.

Where it sits

Worth having first: Embeddings

Unlocks: Context moves vectors · Rows ask questions · Superposition

Chapter: embedding space (parts bin D3/D6)

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).