LeWorldModel

World models · 2026

The idea

V-JEPA 2-AC freezes a giant video encoder, then trains a separate action-conditioned predictor. Can you skip the foundation model and learn a latent world model end-to-end from pixels without the collapse heuristics?

LeWorldModel (LeWM) (Maes, Le Lidec, Scieur, LeCun, Balestriero, 2026, arXiv:2603.19312) is that bet. An encoder maps each frame to a latent; a predictor autoregressively forecasts the next latent given the current latent and the action. The loss has two terms only:

1. next-embedding prediction (MSE)
2. λ · SIGReg on the latents (anti-collapse)

No stop-gradient, no EMA teacher, no required pretrained encoder. About 15M parameters, trainable on a single GPU in a few hours. On reported 2D/3D control tasks, planning completes in under a second and is up to 48× faster than foundation-model-based world-model baselines, while staying competitive on success rate (and beating PLDM on harder tasks like PushT).

LeWM is what LeJEPA looks like when the objective is control, not frozen probing. Contrast LeVJEPA: same SIGReg family, video encoder, no planner.

The misconception to name

It is tempting to read “world model” here as “replaces V-JEPA 2 on robots.” Different regime. LeWM is a compact end-to-end learner on curated control datasets. V-JEPA 2-AC is a large frozen encoder plus a small interaction stage. Both plan in latent space; they are not the same artifact.

The paper is also honest about a residual: in very simple, low-diversity environments, matching a high-dimensional Gaussian prior can hurt, and planning horizons stay short.

Verdict

What it fixedEnd-to-end JEPA world model from raw pixels: next-embedding prediction plus SIGReg, ~15M parameters, single-GPU training, planning up to 48x faster than foundation-model world models on reported control tasks.

What it still costsShort-horizon control in curated 2D/3D suites, not internet-scale video robots. Low-diversity environments can fight the Gaussian prior. Still needs action-labeled offline data.

The story, animated

A tiny end-to-end model beside V-JEPA 2-AC’s giant-plus-bolt-on pipeline - different scale, different regime.

Where it sits

Worth having first: LeJEPA · V-JEPA 2-AC

Next on this track: DreamerV3 - the other latent-control column, where the policy trains inside the dream.

Source-checked against Maes et al., LeWorldModel (arXiv:2603.19312).