Inside the blocks

Attention moves information between tokens; this track is what happens to a token on its own, block after block. Every block reads from and writes back into one running vector per token - the residual stream - and the MLP is where most of the parameters, and most of what looks like “knowledge,” live.

The core chain: the residual stream carries state across layers, LayerNorm keeps hundreds of additions from blowing up, then the MLP does its work in two halves - rows of the up-projection ask a question by dot product, columns of the down-projection write the answer back as a direction in embedding space. Mixture of Experts is what happens when you want many more of those MLPs without paying for all of them on every token. Two side notes matter too: superposition is why a small number of dimensions can hold so many concepts at once, and tokens-to-think is why forcing an answer into one token can break arithmetic a model would otherwise get right.

1. The residual stream - the model’s working memory across layers; every block adds, nothing overwrites.
2. LayerNorm - a plumbing detail with outsized importance: rescale before every read.
3. Rows ask questions - each row of the up-projection is a question asked by dot product.
4. Columns write back - each column of the down-projection is a direction added to the stream.
5. Mixture of Experts - most parameters live in the MLPs, so keep many and route.
6. Superposition - a space can hold more ideas than dimensions if they’re only nearly perpendicular.
7. Tokens to think - compute per token is fixed; reasoning needs room to spread across tokens.

Next: Training, where these blocks first get their weights.