World models

A language model is trained on one bet: given the text so far, what token comes next? That bet took this site from tokens to serving. A world model makes a different bet: given what I just saw (and maybe what I just did), what happens to the world next?

Why that matters: if you can predict the next state, you can try an action in your head before you try it in the world. Planning needs a model of consequences, not a model of plausible English.

It is tempting to think “world model” just means “an LLM that also sees images.” That misses the objective. A multimodal LLM still predicts tokens. A world model predicts future states - often in a learned representation, not in pixels - so a planner can search.

By 2026 the same phrase names three programs: latent dynamics for control (JEPA, Dreamer), generative simulators (Genie, Cosmos), and language/digital world models. This track starts with that fork, walks the JEPA / LeCun spine, then the Dreamer imagination spine. Genie, Cosmos, and language WMs come in later waves.

1. What is a world model - the transition function, not the demo reel.
2. Three programs - latent vs generative vs language; where claims usually slide.
3. JEPA - predict in representation space (I-JEPA).
4. V-JEPA 2 - video encoder at internet scale; still not a robot.
5. V-JEPA 2-AC - the action-conditioned planner on a frozen encoder.
6. LeJEPA - SIGReg replaces teacher-student scaffolding.
7. LeVJEPA - LeJEPA’s recipe on video; frozen probing, not planning.
8. LeWorldModel - end-to-end JEPA world model that plans from pixels.
9. DreamerV3 - train the policy in imagination; Minecraft diamonds from scratch (online).
10. Dreamer 4 - diamonds from offline data only; real-time single-GPU world model.

Then a lab of our own: Lab: Two Rooms - a tiny JEPA learns a room and a door from pixels, and we test which fixes survive moving the door. Three interactive explainers hang off it.

Worth having first: The next-token objective - the bet this track is not making.