Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

The problem

Self-supervised vision in 2022 split into two camps, each paying a different tax. Invariance-based methods — the SimCLR/DINO contrastive and self-distillation lineage — learn by forcing two augmented views of an image to the same representation, which works well but bakes the invariances of a hand-tuned augmentation recipe (random crops, color jitter, grayscale, blur) directly into the learned features. Generative methods like masked autoencoders (MAE) avoid augmentations by reconstructing masked pixels, but spend capacity modeling low-level detail — exact textures, lighting — that carries little semantic meaning and tends to yield representations that need fine-tuning to shine. I-JEPA aims for the best of both: no augmentations, and prediction in a representation space that ignores irrelevant pixel detail.

The method

I-JEPA takes one image, splits it into patches as a Vision Transformer would, and carves the patch grid into two roles. A single large context block — a contiguous region covering roughly half the image — and a handful of target blocks — four non-overlapping squares, each around 15% of the image, sampled from patches the context does not contain. The model never sees a second view, a crop, or a color-jittered copy; it sees one image, holds back most of it, and predicts the held-back parts from what is left.

Three networks carry that prediction. The context encoder is a standard ViT that ingests only the visible context patches and produces context embeddings $s_x$. The target encoder is an exponential-moving-average (EMA) copy of the context encoder — its weights are never touched by gradient descent, only by a slow blend toward the context encoder's weights — and it processes the full image to produce the target embeddings $s_{y_i}$. Between them sits the predictor, a narrow transformer that takes the context embeddings plus a set of learnable positional mask tokens, one per patch of each target block, and outputs a predicted embedding for every target patch. Training minimizes the squared error between predicted and actual target embeddings.

$$\mathcal{L} = \frac{1}{M}\sum_{i=1}^{M}\bigl\lVert\, p_\theta(s_x, \text{pos}_i) - \operatorname{sg}(s_{y_i}) \,\bigr\rVert^2$$

The stop-gradient $\operatorname{sg}(\cdot)$ on the target side is load-bearing: without it the fastest way to drive the loss to zero is for both encoders to map everything to the same constant vector, and the stop-gradient plus the EMA update are exactly what prevent that representational collapse.

Why the masking shape matters

What makes the task hard enough to learn from is the shape of the masking, not its quantity. The context is one large square and each target is a contiguous square, never a scatter of random patches. This is deliberate. An MAE-style scheme that hides individual patches at random leaves so many visible neighbors around each hole that prediction collapses into local texture interpolation — copy the patch next door and you are usually close. Forcing the targets to be coherent blocks, sampled away from the context region, removes that crutch: to place a plausible representation on a 15% chunk of image it has never seen, the model has to reason about what the whole scene contains and where its parts sit relative to each other. Predicting in representation rather than pixel space is the second half of the design — the target encoder has already abstracted away the unpredictable low-level detail, so the predictor is never penalized for failing to hallucinate exact textures.

Results

The numbers say the trade is worth making. With a ViT-H/14 backbone, I-JEPA matches or beats masked autoencoders and data2vec on ImageNet linear probing while spending on the order of 10× less compute than an equivalent MAE — predicting a few hundred target embeddings is far cheaper than reconstructing every pixel, and the representations are strong enough that a linear probe (no fine-tuning) reads them off directly. The gap widens on low-level tasks: on object counting and monocular depth estimation — where you need to know where things are, not just what the image is of — I-JEPA outperforms contrastive models, which spend their invariances discarding exactly the spatial detail those tasks depend on.

Why it matters

I-JEPA is the image instantiation of Yann LeCun's broader argument that prediction in an abstract representation space, rather than reconstruction of raw inputs, is the right objective for self-supervised learning. Its practical contribution is a clean recipe — EMA target encoder, block-structured masking, latent-space prediction — that learns semantically strong features without the augmentation engineering contrastive methods depend on or the pixel-reconstruction overhead generative methods pay. Its limitation is set by its input: a static image has no axis along which anything moves, no notion that the same object persists from one frame to the next. Removing that limitation — keeping the architecture almost unchanged and adding the dimension of time — is exactly what V-JEPA does for video.