Deep dives · 2026-11-16 · Basic · By bombong
JEPA: Pretraining that predicts in latent space instead of reconstructing pixels
JEPA does not redraw masked parts in pixels. Instead, it has the model guess 'what' those parts are in embedding space, and in doing so it learns meaningful representations. It started with images and has expanded to audio, sensors, satellite imagery and driving video. More and more research now connects it to action planning and language models.
Contents
- Why it was needed: the problem with pretraining that redraws pixels
- Core idea: 'say what it is' instead of 'redraw it'
- How it works: two encoders and one predictor
- How it developed: from images to audio, sensors and driving video
- Where it is heading now: connecting prediction to action and language
- Limitations, open issues and practical criteria
- Further study
- JEPA uses the visible part (the context) to predict the embedding of the masked part (the target), not its original pixels. As a result, it spends learning capacity on structure that can be predicted rather than on details that cannot.
- Because the model also produces the target embeddings, 'collapse' can occur, where every input is mapped to the same vector. A mechanism that anchors the representations, such as an auxiliary task, is needed.
- What gets masked and what is given as a condition determine performance. This has led to variants such as prediction ordering, caption conditioning and uncertainty modeling.
- Recently, V-JEPA-family representations have been connected to driving plan selection and video language models. However, even when the information is present in the encoder, the training that 'reads it out' has to be designed separately.
- Latent-space prediction alone does not guarantee robustness. In a medical imaging study, representations shifted substantially under noise and attacks.
Why it was needed: the problem with pretraining that redraws pixels
In pretraining, where a model is trained in advance on large amounts of unlabeled data, a common method is 'mask and predict'. If you erase part of an image or a sound clip and ask the model to restore it, the data itself becomes both the problem and the answer, with no human labeling required. This is called self-supervised learning.
But if the model must restore the original exactly as pixels or waveforms, it tries to match details that cannot be predicted in the first place, such as the texture of leaves, camera noise or background noise. That leaves less capacity for learning what actually matters later: meaning, such as 'what this is and how it is arranged'.
JEPA (Joint-Embedding Predictive Architecture) changed the prediction target to avoid this problem. Audio-JEPA paper describes JEPA as 'a self-supervised learning framework that predicts latent representations of masked regions in a high-level feature space', and instead of reconstructing raw audio, it predicts latent representations of spectrogram patches. CHARM paper, which applies JEPA to sensor time series, also explains that predicting in latent space induces representations that are robust to sensor noise.
Core idea: 'say what it is' instead of 'redraw it'
Imagine masking part of a photo and giving two tests. The first test asks you to paint the masked part exactly with a brush. You only score if every strand of fur and every shadow matches. The second test asks you only to state the gist, such as 'a dog's head facing forward should be here'. Doing well on the second test requires understanding the scene but no painting skill. JEPA gives the second test.
The 'joint embedding' in the name means that the visible and masked parts are mapped into the same embedding space (a space that represents meaning as numeric vectors). 'Predictive' means that, within that space, one vector is used to predict the other. Because the answer is a vector rather than pixels, unpredictable details can be filtered out at the stage where the vector is produced.
There is a cost. Since the output is not pixels, it cannot be used directly to generate images. And because the model produces the target vectors itself, there is a risk that training collapses in an unintended direction. Compared with another alternative, contrastive learning (pulling two views of the same object closer and different objects apart), JEPA learns from prediction alone without needing to select 'different object' examples. Text-Conditional JEPA paper reported that vision-language pretraining based solely on feature prediction outperforms contrastive approaches on many tasks, especially those requiring fine-grained visual understanding and reasoning.
Pixel reconstruction vs. JEPA
Pixel reconstructionRedraw the masked part
- Answer is original pixels / waveforms
- Tries to match even unpredictable details
- Leads directly to generative uses
JEPAPredict the embedding of the masked part
- Answer is the target encoder's embedding
- Focuses on predictable structure
- Needs a mechanism to prevent collapse
공통 A pretraining method that learns from the data itself, without labels
How it works: two encoders and one predictor
Here is how it works step by step. First, the input is split into small pieces (patches). For images, these are square patches; for audio, as in Audio-JEPA paper, they are patches of a mel spectrogram (sound turned into a time × frequency picture). Some of these are designated as the 'target' and masked, and the rest are kept as the 'context'.
Second, the context encoder turns the visible patches into embeddings. Third, the target encoder looks at the original masked patches and produces the 'target embeddings'. Fourth, the predictor takes the context embeddings plus positional information about 'where to predict' and predicts the target embeddings. Fifth, the model is trained to reduce the difference between the predicted vectors and the target vectors. After training, usually only the encoder is taken and used for other tasks such as classification, retrieval or planning. As with CHARM paper, representation quality is often measured with a linear probe, where the encoder is frozen and only a single linear classifier is added on top.
This structure has one weakness. Because the model also produces the answers, if it maps every input to the same vector, the prediction is always correct and the loss becomes 0. This is called representation collapse. The auxiliary task study paper analyzed a simple JEPA variant that trains an auxiliary regression head alongside latent state transition prediction. It proved a 'No Unhealthy Representation Collapse' theorem: in a deterministic environment, when both losses go to 0, observations with different transitions or different auxiliary values must map to different representations. In other words, the auxiliary task acts as an anchor that holds on to 'what needs to be distinguished'.
How it developed: from images to audio, sensors and driving video
On the image side, work refined 'where to predict and in what order'. DSeq-JEPA paper pointed out that I-JEPA predicts all masked regions at once in parallel, so it cannot impose an order. It therefore first finds key regions using an importance map built from attention, then predicts the rest in order of importance. This curriculum-like order imitates how people look at the most important clues first. It reported better representations across many tasks, including classification, fine-grained classification, detection and segmentation, and CLEVR reasoning. Text-Conditional JEPA paper addressed the problem that prediction is hard because masked regions are inherently uncertain by using image captions. Adjusting predicted features with sparse cross-attention over text tokens improved both downstream task performance and training stability.
The approach also moved quickly to other data. Audio-JEPA paper simply transferred the original model to audio almost unchanged, yet performed comparably to wav2vec 2.0 and data2vec. It used less than one-fifth of their training data and did no hyperparameter tuning. CHARM paper fed the model the descriptive text attached to each sensor channel and performed well on anomaly detection, classification and forecasting using only linear probes. Its analysis found that performance came mainly from the JEPA objective and the conditioning structure, while the descriptive text served as an identifier that let the model recognize channels across different datasets. CAPE-JEPA paper tackled retrieval between physically different sensors, such as SAR (radar) and multispectral imagery. It noted that the earlier X-JEPA assumed every patch is equally predictable, and used a probabilistic predictor that outputs a mean and variance for each patch. As a result, it outperformed X-JEPA by up to 13.74 points on BEN-14K and up to 15.59 points on fMoW.
V-JEPA, which extended the approach to video, has moved into driving. Drive-JEPA paper pretrained a ViT encoder on large-scale driving video using the V-JEPA approach. Attaching only a simple transformer decoder to these representations scored 3 PDMS higher than prior methods in NAVSIM's perception-free setting, and the full framework recorded 93.3 PDMS on v1 and 87.8 EPDMS on v2.
Where it is heading now: connecting prediction to action and language
The recent trend is to use the representations JEPA learns about 'how things will change' for actual decisions. ActWorld-JEPA paper starts from the point that consistently predicting the future in latent space does not by itself tell you which candidate driving plan is better. So it predicts future latent states conditioned on a candidate Trajectory (planned path), then revises plan scores and re-ranks them based on how well the plan and the predicted future agree or conflict. It reported 94.1 PDMS on NAVSIM v1 navtest, but stated that the final selector was calibrated using aggregate NAVTEST feedback. This condition should be kept in mind when reading the number. A structure like this, which predicts the future given actions, is close to a world model (a model that internally simulates how the world will change).
V-JEPA-LM paper addressed the language side. Video encoders such as V-JEPA learn dynamics such as direction, speed, order and state changes. But when they are attached to a language model via captions and question answering, that information goes unused. On the same questions, a linear probe on frozen V-JEPA features identified speed correctly 87% of the time, while the VLM answered at chance level. The researchers used pixel transformations to create counterfactual training data that 'differs only in the change', so the language model would read that information. As a result, a 4B model scored an average of 72.1 across 11 video benchmarks. The best open 4B model of the same size scored 61.1, and the best open 8B model scored 64.8.
The two studies point to the same conclusion. The encoder determines 'what can be read', and the training signal on top determines 'what is actually read'. The value of JEPA representations only shows when the side that extracts and uses them is designed as well.
Average score across 11 video benchmarks
Limitations, open issues and practical criteria
The first issue is robustness. A chest X-ray study paper put four JEPA encoders trained on 78,468 images from NIH ChestX-ray14 through six tests, including attacks, noise, and brightness and contrast changes. The cosine similarity of encoder embeddings dropped as low as 0.33, and in the worst case (I-JEPA, Boundary attack) AUROC fell by 0.12. Adversarially fine-tuning the predictor with both clean and perturbed images raised embedding consistency above 0.97 in almost all cases. By contrast, a vector quantization codebook trained only on clean images actually got worse. This means latent-space prediction can 'induce' noise-robust representations but does not guarantee them; robustness requires training directly on perturbed data.
The second is uncertainty. What belongs in a masked region inherently has multiple possible answers. This is why Text-Conditional JEPA paper added caption conditioning and CAPE-JEPA paper added per-patch variance prediction. The third is collapse. As the auxiliary task study paper shows, without a signal indicating what needs to be distinguished, representations can lose necessary distinctions.
In practical terms: JEPA fits well when labels are scarce but unlabeled video or sensor data is plentiful, and when the end use is 'semantic judgment' such as classification, retrieval or planning. For generative tasks that must output pixels or waveforms, a separate decoder is needed. There are three common mistakes: watching only the loss go down and missing collapse; applying uniform image-style masking as-is to domains with different structure, such as medical or satellite imagery; and deploying based only on performance on clean validation data. Especially in fields like medicine that affect patients, research benchmark numbers do not mean real clinical performance. Final responsibility for decisions lies with people.
Checklist before adopting JEPA
Fit
- Is there enough unlabeled data?
- Is the end use semantic judgment, such as classification, retrieval or planning?
- If pixel output is needed, is there a separate decoder?
Training
- Has collapse been checked with a linear probe?
- Do masking positions match the domain's structure?
- Are there conditions or mechanisms to handle prediction uncertainty?
Deployment
- Has robustness been tested with noisy and perturbed data?
- Has the training on the side that reads out representations been designed separately?
- Is there a process for people to make the final decision?
Further study
This list does not include the original I-JEPA or V-JEPA papers. So it is best to use the explanations in each paper's introduction as a starting point. For first learning the structure, Audio-JEPA paper is a good fit. Because it simply transfers the original design to audio, it shows what the encoder, predictor and masking are without extra complexity.
Next, reading DSeq-JEPA paper and Text-Conditional JEPA paper, which cover improvements on the image side, establishes the design axes of 'where to predict' and 'what to condition on'. If you want the theory of why collapse happens and how to prevent it, read the auxiliary task study paper.
If you are interested in the world model side, such as robotics and driving, read Drive-JEPA paper followed by ActWorld-JEPA paper, and for the connection to language models, see V-JEPA-LM paper. To assess risks before real deployment, we strongly recommend also reading the chest X-ray robustness study paper.
Related terms
World model Embedding Pre-training Unsupervised learning VLM Multimodal Transformer Medical imaging AI
References
Foundational papers
- Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning (2025)
- DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture (2025)
- Text-Conditional JEPA for Learning Semantically Rich Visual Representations (2026)
- Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings (2026)
- CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA (2026)
- Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving (2026)
- Why and How Auxiliary Tasks Improve JEPA Representations (2025)
Recent research
- ActWorld-JEPA: Action-Conditioned JEPA World Models Meet Latent Planning for End-to-End Autonomous Driving (2026)
- V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations (2026)
- Cracks in Embeddings: Robustness and Stability of JEPA-Based Chest X-Ray Representations (2026)
Written by AI from the listed references and reviewed. Please check the original sources for details.