LeJEPA

World models · 2025

The idea

What if the teacher, the stop-gradient, and the predictor existed only to stop collapse - and a statistical regularizer could do that job instead?

LeJEPA (Balestriero & LeCun, 2025, arXiv:2511.08544) starts from a claim: under mild assumptions, the embedding distribution that minimizes worst-case downstream probing risk is an isotropic Gaussian. Then it supplies a tool to enforce that distribution: SIGReg, the Sketched Isotropic Gaussian Regularizer.

SIGReg samples many random 1-D directions, projects the batch of embeddings onto each, and penalizes deviation from a standard Gaussian with a univariate normality test (Epps-Pulley). By the Cramér-Wold theorem, matching every 1-D projection matches the full distribution. A collapsed solution - zero variance on some axis - fails those tests.

Combine a JEPA-style predictive (or invariance) loss with λ · SIGReg and the scaffolding can leave: no EMA teacher, no stop-gradient, a single trade-off weight. LeJEPA reports scaling across ResNets, ViTs, and ConvNets; one cited ImageNet linear-probe result is 79% with a ViT-H/14.

The misconception to name

It is tempting to treat SIGReg as “just another VICReg.” The difference LeJEPA insists on is which distribution is optimal (isotropic Gaussian) and a linear-cost sketch that targets that distribution, not only second-order statistics. The “provable” language lives inside the paper’s assumptions - state them, do not mystify them.

Verdict

What it fixedCollapse prevention becomes a distributional regularizer (SIGReg): one shared network, one loss weight, no teacher EMA, no stop-gradient, no predictor required for stability.

What it still costsTheory and image-scale evidence first; video and control are the pages that follow. The isotropic-Gaussian claim is an argument under stated assumptions, not a free lunch.

The story, animated

Check every 1-D slice against a Gaussian - then three pieces of scaffolding collapse into one loss weight.

Where it sits

Worth having first: JEPA

Unlocks: LeVJEPA · LeWorldModel

Source-checked against Balestriero & LeCun, LeJEPA (arXiv:2511.08544).