← Deep dives

Deep dives · 2026-11-16 · Basic · By bombong

JEPA: Pretraining that predicts in latent space instead of reconstructing pixels

JEPA does not redraw masked parts in pixels. Instead, it has the model guess 'what' those parts are in embedding space, and in doing so it learns meaningful representations. It started with images and has expanded to audio, sensors, satellite imagery and driving video. More and more research now connects it to action planning and language models.

Contents
Key points
  • JEPA uses the visible part (the context) to predict the embedding of the masked part (the target), not its original pixels. As a result, it spends learning capacity on structure that can be predicted rather than on details that cannot.
  • Because the model also produces the target embeddings, 'collapse' can occur, where every input is mapped to the same vector. A mechanism that anchors the representations, such as an auxiliary task, is needed.
  • What gets masked and what is given as a condition determine performance. This has led to variants such as prediction ordering, caption conditioning and uncertainty modeling.
  • Recently, V-JEPA-family representations have been connected to driving plan selection and video language models. However, even when the information is present in the encoder, the training that 'reads it out' has to be designed separately.
  • Latent-space prediction alone does not guarantee robustness. In a medical imaging study, representations shifted substantially under noise and attacks.

Why it was needed: the problem with pretraining that redraws pixels

In pretraining, where a model is trained in advance on large amounts of unlabeled data, a common method is 'mask and predict'. If you erase part of an image or a sound clip and ask the model to restore it, the data itself becomes both the problem and the answer, with no human labeling required. This is called self-supervised learning.

But if the model must restore the original exactly as pixels or waveforms, it tries to match details that cannot be predicted in the first place, such as the texture of leaves, camera noise or background noise. That leaves less capacity for learning what actually matters later: meaning, such as 'what this is and how it is arranged'.

JEPA (Joint-Embedding Predictive Architecture) changed the prediction target to avoid this problem. Audio-JEPA paper describes JEPA as 'a self-supervised learning framework that predicts latent representations of masked regions in a high-level feature space', and instead of reconstructing raw audio, it predicts latent representations of spectrogram patches. CHARM paper, which applies JEPA to sensor time series, also explains that predicting in latent space induces representations that are robust to sensor noise.

Core idea: 'say what it is' instead of 'redraw it'

Imagine masking part of a photo and giving two tests. The first test asks you to paint the masked part exactly with a brush. You only score if every strand of fur and every shadow matches. The second test asks you only to state the gist, such as 'a dog's head facing forward should be here'. Doing well on the second test requires understanding the scene but no painting skill. JEPA gives the second test.

The 'joint embedding' in the name means that the visible and masked parts are mapped into the same embedding space (a space that represents meaning as numeric vectors). 'Predictive' means that, within that space, one vector is used to predict the other. Because the answer is a vector rather than pixels, unpredictable details can be filtered out at the stage where the vector is produced.

There is a cost. Since the output is not pixels, it cannot be used directly to generate images. And because the model produces the target vectors itself, there is a risk that training collapses in an unintended direction. Compared with another alternative, contrastive learning (pulling two views of the same object closer and different objects apart), JEPA learns from prediction alone without needing to select 'different object' examples. Text-Conditional JEPA paper reported that vision-language pretraining based solely on feature prediction outperforms contrastive approaches on many tasks, especially those requiring fine-grained visual understanding and reasoning.

Pixel reconstruction vs. JEPA

Pixel reconstructionRedraw the masked part

  • Answer is original pixels / waveforms
  • Tries to match even unpredictable details
  • Leads directly to generative uses

JEPAPredict the embedding of the masked part

  • Answer is the target encoder's embedding
  • Focuses on predictable structure
  • Needs a mechanism to prevent collapse

공통 A pretraining method that learns from the data itself, without labels

Both are mask-and-predict self-supervised learning. The difference is whether the answer is pixels or embeddings.

How it works: two encoders and one predictor

Here is how it works step by step. First, the input is split into small pieces (patches). For images, these are square patches; for audio, as in Audio-JEPA paper, they are patches of a mel spectrogram (sound turned into a time × frequency picture). Some of these are designated as the 'target' and masked, and the rest are kept as the 'context'.

Second, the context encoder turns the visible patches into embeddings. Third, the target encoder looks at the original masked patches and produces the 'target embeddings'. Fourth, the predictor takes the context embeddings plus positional information about 'where to predict' and predicts the target embeddings. Fifth, the model is trained to reduce the difference between the predicted vectors and the target vectors. After training, usually only the encoder is taken and used for other tasks such as classification, retrieval or planning. As with CHARM paper, representation quality is often measured with a linear probe, where the encoder is frozen and only a single linear classifier is added on top.

This structure has one weakness. Because the model also produces the answers, if it maps every input to the same vector, the prediction is always correct and the loss becomes 0. This is called representation collapse. The auxiliary task study paper analyzed a simple JEPA variant that trains an auxiliary regression head alongside latent state transition prediction. It proved a 'No Unhealthy Representation Collapse' theorem: in a deterministic environment, when both losses go to 0, observations with different transitions or different auxiliary values must map to different representations. In other words, the auxiliary task acts as an anchor that holds on to 'what needs to be distinguished'.

One JEPA training stepInput → Visible part; Input → Masked part; Visible part → Context encoder; Masked part → Target encoder; Context encoder → Predictor; Predictor → Compare embeddings(Prediction); Target encoder → Compare embeddings(Target); Auxiliary task → Compare embeddingsInputImage / spectrogrampatchesVisible partContextMasked partTargetContext encoderTarget encoderProduces targetembeddingsPredictorAlso receivespositionalinformationCompare embeddingsTrain to reduce thedifferenceAuxiliary taskAnchor againstcollapsePredictionTarget
One JEPA training step The top row is the path that predicts from the visible part; the bottom row is the path that produces target embeddings from the masked part. The two results are compared in embedding space, not in pixels.

How it developed: from images to audio, sensors and driving video

On the image side, work refined 'where to predict and in what order'. DSeq-JEPA paper pointed out that I-JEPA predicts all masked regions at once in parallel, so it cannot impose an order. It therefore first finds key regions using an importance map built from attention, then predicts the rest in order of importance. This curriculum-like order imitates how people look at the most important clues first. It reported better representations across many tasks, including classification, fine-grained classification, detection and segmentation, and CLEVR reasoning. Text-Conditional JEPA paper addressed the problem that prediction is hard because masked regions are inherently uncertain by using image captions. Adjusting predicted features with sparse cross-attention over text tokens improved both downstream task performance and training stability.

The approach also moved quickly to other data. Audio-JEPA paper simply transferred the original model to audio almost unchanged, yet performed comparably to wav2vec 2.0 and data2vec. It used less than one-fifth of their training data and did no hyperparameter tuning. CHARM paper fed the model the descriptive text attached to each sensor channel and performed well on anomaly detection, classification and forecasting using only linear probes. Its analysis found that performance came mainly from the JEPA objective and the conditioning structure, while the descriptive text served as an identifier that let the model recognize channels across different datasets. CAPE-JEPA paper tackled retrieval between physically different sensors, such as SAR (radar) and multispectral imagery. It noted that the earlier X-JEPA assumed every patch is equally predictable, and used a probabilistic predictor that outputs a mean and variance for each patch. As a result, it outperformed X-JEPA by up to 13.74 points on BEN-14K and up to 15.59 points on fMoW.

V-JEPA, which extended the approach to video, has moved into driving. Drive-JEPA paper pretrained a ViT encoder on large-scale driving video using the V-JEPA approach. Attaching only a simple transformer decoder to these representations scored 3 PDMS higher than prior methods in NAVSIM's perception-free setting, and the full framework recorded 93.3 PDMS on v1 and 87.8 EPDMS on v2.

Where it is heading now: connecting prediction to action and language

The recent trend is to use the representations JEPA learns about 'how things will change' for actual decisions. ActWorld-JEPA paper starts from the point that consistently predicting the future in latent space does not by itself tell you which candidate driving plan is better. So it predicts future latent states conditioned on a candidate Trajectory (planned path), then revises plan scores and re-ranks them based on how well the plan and the predicted future agree or conflict. It reported 94.1 PDMS on NAVSIM v1 navtest, but stated that the final selector was calibrated using aggregate NAVTEST feedback. This condition should be kept in mind when reading the number. A structure like this, which predicts the future given actions, is close to a world model (a model that internally simulates how the world will change).

V-JEPA-LM paper addressed the language side. Video encoders such as V-JEPA learn dynamics such as direction, speed, order and state changes. But when they are attached to a language model via captions and question answering, that information goes unused. On the same questions, a linear probe on frozen V-JEPA features identified speed correctly 87% of the time, while the VLM answered at chance level. The researchers used pixel transformations to create counterfactual training data that 'differs only in the change', so the language model would read that information. As a result, a 4B model scored an average of 72.1 across 11 video benchmarks. The best open 4B model of the same size scored 61.1, and the best open 8B model scored 64.8.

The two studies point to the same conclusion. The encoder determines 'what can be read', and the training signal on top determines 'what is actually read'. The value of JEPA representations only shows when the side that extracts and uses them is designed as well.

Average score across 11 video benchmarks

V-JEPA-LM 4B
Best open 4B
Best open 8B
Average scores reported in the V-JEPA-LM paper. The 4B model scored higher than even the best open 8B model.

Limitations, open issues and practical criteria

The first issue is robustness. A chest X-ray study paper put four JEPA encoders trained on 78,468 images from NIH ChestX-ray14 through six tests, including attacks, noise, and brightness and contrast changes. The cosine similarity of encoder embeddings dropped as low as 0.33, and in the worst case (I-JEPA, Boundary attack) AUROC fell by 0.12. Adversarially fine-tuning the predictor with both clean and perturbed images raised embedding consistency above 0.97 in almost all cases. By contrast, a vector quantization codebook trained only on clean images actually got worse. This means latent-space prediction can 'induce' noise-robust representations but does not guarantee them; robustness requires training directly on perturbed data.

The second is uncertainty. What belongs in a masked region inherently has multiple possible answers. This is why Text-Conditional JEPA paper added caption conditioning and CAPE-JEPA paper added per-patch variance prediction. The third is collapse. As the auxiliary task study paper shows, without a signal indicating what needs to be distinguished, representations can lose necessary distinctions.

In practical terms: JEPA fits well when labels are scarce but unlabeled video or sensor data is plentiful, and when the end use is 'semantic judgment' such as classification, retrieval or planning. For generative tasks that must output pixels or waveforms, a separate decoder is needed. There are three common mistakes: watching only the loss go down and missing collapse; applying uniform image-style masking as-is to domains with different structure, such as medical or satellite imagery; and deploying based only on performance on clean validation data. Especially in fields like medicine that affect patients, research benchmark numbers do not mean real clinical performance. Final responsibility for decisions lies with people.

Checklist before adopting JEPA

Fit

  • Is there enough unlabeled data?
  • Is the end use semantic judgment, such as classification, retrieval or planning?
  • If pixel output is needed, is there a separate decoder?

Training

  • Has collapse been checked with a linear probe?
  • Do masking positions match the domain's structure?
  • Are there conditions or mechanisms to handle prediction uncertainty?

Deployment

  • Has robustness been tested with noisy and perturbed data?
  • Has the training on the side that reads out representations been designed separately?
  • Is there a process for people to make the final decision?
The limitations and common mistakes from the text, turned into check items.

Further study

This list does not include the original I-JEPA or V-JEPA papers. So it is best to use the explanations in each paper's introduction as a starting point. For first learning the structure, Audio-JEPA paper is a good fit. Because it simply transfers the original design to audio, it shows what the encoder, predictor and masking are without extra complexity.

Next, reading DSeq-JEPA paper and Text-Conditional JEPA paper, which cover improvements on the image side, establishes the design axes of 'where to predict' and 'what to condition on'. If you want the theory of why collapse happens and how to prevent it, read the auxiliary task study paper.

If you are interested in the world model side, such as robotics and driving, read Drive-JEPA paper followed by ActWorld-JEPA paper, and for the connection to language models, see V-JEPA-LM paper. To assess risks before real deployment, we strongly recommend also reading the chest X-ray robustness study paper.

Related terms

World model Embedding Pre-training Unsupervised learning VLM Multimodal Transformer Medical imaging AI

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.