V-JEPA: Latent Video Prediction for Visual Representation Learning

The problem

I-JEPA showed that predicting masked image regions in representation space learns strong features without augmentations or pixel reconstruction — but a static image has no time axis. It cannot learn that the same object persists across frames, that motion is smooth, or that a falling cup will be lower a moment later. Everything it knows is spatial. Video supplies exactly the missing dimension, and the question V-JEPA answers is whether the JEPA recipe — abstract, latent-space prediction — extracts good dynamics representations from raw video, where the dominant prior alternative (video masked autoencoders) reconstructs pixels and the contrastive lineage relies on heavy augmentation.

The method

The machinery is the image setup lifted into three dimensions. A spatiotemporal ViT tokenizes a clip into 3D patches that span height, width, and a short stretch of time. The context encoder processes only the visible tokens to produce context embeddings; a target encoder — again an EMA copy whose weights receive no gradient — encodes the full clip to produce the targets; and the predictor takes the context embeddings plus positional mask tokens for each masked spatiotemporal location and outputs a predicted representation there. Training is the same stop-gradient mean-squared-error objective that kept I-JEPA from collapsing — the masked region may sit later in the clip than the context, so the model is no longer only reasoning about what lies outside the frame but about what comes next.

How the masking decides what is learned

How the clip is masked decides what the model is forced to learn, and two schemes dominate. Tube masking removes the same spatial region across every time step, so the hole is a column drilled straight through the clip — nothing visible at any frame sits where the answer should be, and the only way to fill it is to infer how the scene evolves from the surrounding frames. Random block masking removes 3D blocks scattered through space and time, which usually leaves a visible version of the masked region at some nearby frame, so the model can solve much of the task by copying across a short temporal gap rather than predicting genuine dynamics. Tube masking is the harder of the two and the one that pushes the encoder toward temporal reasoning.

Nothing in the objective mentions objects, motion, or physics, yet a model that does it well has to behave as though it understood all three: to place a sensible representation on a tube it cannot see, it must carry an object's identity across the frames where it is hidden, respect that motion is smooth rather than teleporting, and infer the contents of an occluded region from what the visible frames imply. None of that is labeled — it is what a purely predictive loss extracts from raw video once the masking makes copying impossible.

Results

V-JEPA is trained on large unlabeled video collections (Kinetics-400 and Something-Something v2 among the sources) and evaluated frozen, with only a lightweight attentive probe on top. The contrast with pixel-level pretraining is where the design earns its keep. A video masked autoencoder such as VideoMAE reconstructs the missing pixels in extraordinary detail — but that detail is the problem, because the model burns capacity modeling fabric textures, static backgrounds, and incidental motion no downstream task cares about. V-JEPA predicts representations instead, discards that low-level detail, and as a result transfers better to action recognition while training at lower compute than the pixel-reconstruction baselines.

The sharpest evidence is Something-Something v2, whose labels are not "dog" or "kitchen" but fine-grained physical interactions — pushing something so it rolls, pretending to pick something up, putting one thing behind another. You cannot tell those classes apart from appearance alone, because the same objects and scenes recur across opposite labels; you have to read the dynamics. V-JEPA's margin over appearance-driven models on exactly this dataset is the signal that temporal prediction in representation space captures something closer to physical intuition than to texture statistics.

Why it matters

V-JEPA closes the loop on the JEPA program: it demonstrates that the same latent-prediction principle that learns good spatial features from images learns good temporal features from video, without reconstruction and without contrastive augmentation. Beyond representation learning, it is the most direct stepping stone toward world models in the LeCun sense. Strip the action out of a world model's definition and V-JEPA's predictor — mapping context representations plus a target position to the representation that should appear there — is a world model for the actionless case, forecasting future latent state from current context. Adding an action-conditioning input to that predictor is the difference between forecasting how a scene evolves on its own and how it evolves because of what an agent did — the natural extension toward planning agents that the JEPA line points at, and that follow-up work (V-JEPA 2 and its action-conditioned variants) began to cash out.