JEPA

World models · 2022-23

The idea

Why spend a model’s capacity painting the exact texture of every leaf in the next frame, if a planner only needs to know that the leaf is still on the tree?

JEPA - Joint-Embedding Predictive Architecture - is LeCun’s answer: encode two views of the world into a latent space, and train a predictor so one latent forecasts the other. Prediction happens in representation space. Unpredictable pixel noise is never a target.

I-JEPA (Assran et al., CVPR 2023) is the image instantiation. Mask large regions of an image; predict the missing regions’ embeddings from a context encoder, against targets from a target encoder. No hand-designed augmentations as the sole signal; the mask is the pretext. Training a ViT-H/14 on ImageNet in under 72 hours on 16 A100s was the efficiency headline of that paper.

The architecture sketch that matters for later pages:

• a context encoder sees what is given
• a target encoder (often an EMA copy) produces the thing to predict
• a predictor maps context latents toward target latents

That teacher-student scaffolding exists for one reason: without it, joint-embedding training collapses - every input maps to the same vector, the loss goes to zero, and nothing was learned. The whole engineering story from V-JEPA through LeJEPA is what to do about that collapse.

Verdict

What it fixedPredict missing pieces in representation space, not in pixels - skip the leaf flutter you cannot and need not reconstruct.

What it still costsWithout extra scaffolding, joint-embedding training collapses to a constant. The whole JEPA engineering story is how to stop that.

The story, animated

Five distinct latents collapse to one without scaffolding - then the same five, run through it, stay five.

Where it sits

Worth having first: Three programs · Attention

Unlocks: V-JEPA 2 · LeJEPA

Source-checked against Assran et al., I-JEPA (CVPR 2023) and the JEPA framing in later V-JEPA / LeJEPA papers.