Learning JEPA

How I-JEPA works, explained simply, and what happened when I tested it on CIFAR-100

Cover part of a photo, and you can still guess roughly what's behind your hand. You don't picture every pixel. You just know "that's probably more wing". JEPA teaches a network to do the same, without any labels.

In one sentence JEPA hides part of the data and learns to predict a description of the hidden part, not its exact pixels.

The main architecture

I-JEPA is the version of JEPA for images. It has three networks that work together.

I-JEPA training: an image is split into visible context patches and hidden target blocks. The context encoder and predictor guess the hidden features, the target encoder gives the answers, and MSE compares them. Gradients update the context encoder and predictor, and EMA slowly updates the target encoder.
Our small I-JEPA. Select the picture to open it full size.
  1. Hide: choose some blocks of the image to hide.
  2. Look: the context encoder sees only the visible patches.
  3. Guess: the predictor guesses the description of the hidden patches.
  4. Check: the target encoder sees the whole image and gives the answer to compare with.
  5. Learn: the error trains the context encoder and predictor, and the target encoder slowly follows.
Where do the "answers" come from, with no labels? From the image itself. The target encoder's description of the hidden patches is the answer. No person labels anything during this training.
Six CIFAR-10 images. The top row shows the visible context patches. The bottom row outlines the hidden target patches in orange.
Real masks. Top: what the context encoder sees. Bottom: the hidden targets, in orange.

The three parts, one at a time

1. Context encoder: looks at what's visible

Think of it as a student looking at a photo with parts covered up.
Context encoder: input with hidden patches, patch embedding, positions, removal of hidden tokens, Transformer layers and output.

It cuts the image into patches, removes the hidden ones and describes the rest. Hidden patches are removed before attention, so it can't peek at them. It learns by gradients.

2. Target encoder: gives the answers

Think of it as a slowly changing answer key that sees the whole photo.
Target encoder: the whole image, patch embedding, positions, Transformer layers, LayerNorm, then only the target tokens are kept.

It has the same structure, but it isn't trained by gradients. After every step, it moves slightly towards the context encoder. This is called EMA, and nothing is copied across only at the end:

target weight = 0.996 × target weight + 0.004 × context weight

The 0.996 rises towards 1 during training, so the target changes more and more slowly. After training, this is the encoder that's kept.

3. Predictor: makes the guess

Think of it as the student's guess: "given what I see, and where the hidden patch is, here's its description."
Predictor: context features and mask tokens with target positions are joined, passed through Transformer layers, and the mask tokens become predictions.

It's a smaller network. It receives the visible features, plus a "blank" mask token for each hidden patch with that patch's position. After training, it isn't needed for image features.

Inside the encoder: the Vision Transformer

Both encoders are Vision Transformers (ViT). A ViT treats an image a bit like a sentence, with patches as its words. Our small one works like this:

StepWhat happensShape
1. ImageA small colour photo3 × 32 × 32
2. PatchesCut into small squares of 4 × 4 pixels64 patches
3. TokensEach patch becomes a list of 192 numbers64 × 192
4. PositionsAdd a learned vector saying where each patch was64 × 192
5. Transformer layers6 layers, each with attention and a small MLP64 × 192
6. Image featureThe average of the tokens192 numbers
A frog image, the same image cut into 64 patches, and a table of 64 tokens with 192 numbers each.
An image becomes patches, then tokens.

Attention, in plain words

Attention lets every patch look at every other patch and borrow useful information. Each token makes three things:

PartPlain meaning
Query"What am I looking for?"
Key"What do I offer?"
Value"Here's my information."

Each query is compared with every key. Good matches get larger weights, and each token becomes a weighted mix of the values. This runs in 4 "heads" at once, so different heads can look for different things.

Why positions matter

Attention compares content, so it has no idea where a patch came from. I tested this with a supervised ViT, trained with labels, as a check on the encoder alone:

CIFAR-10, 20 epochs, 3 seedsNormal imagesPatches shuffled
With positions81.05% ± 0.2757.00% ± 0.58
No positions73.90% ± 0.3273.90% ± 0.32

Without positions, shuffling changes nothing at all: the network really can't see layout. This test used labels, so it isn't JEPA.

An 8 by 8 grid of small maps. Each shows that a position vector is most similar to itself and nearby positions.
The network learned the grid by itself: each position is most similar to nearby ones.
Show where attention looks
Four images with the middle patch marked, and the attention maps from layers 1 and 6.
Where the middle patch looks. These maps are hard to interpret, so they aren't explanations of a decision.

The experiment on CIFAR-100

Can a robot-style memory learn new classes using I-JEPA features? I replaced ResNet in CBCL and CBCL-PR, from Dr Ali Ayub's work, with I-JEPA. I call this side experiment Jepa-CBCL-PR.

Jepa-CBCL-PR: new class images pass through a frozen I-JEPA encoder, then Agg-Var clustering into class memory. CBCL votes with centroids, while CBCL-PR samples fake features and trains a linear classifier.
Important: no encoder was trained on CIFAR-100. All encoders were frozen, so the 100 classes were new to them. Only CBCL's memory and CBCL-PR's small classifier learned anything here.

Time and epochs

How each encoder was trained

EncoderTrained onLabels?EpochsTraining timeEpochs on CIFAR-100
Meta I-JEPA, ViT-H/14ImageNet-1K, by MetaNo300Not known to me0
ResNet-34ImageNet, pretrained in torchvisionYesNot checkedNot known to me0
Our small I-JEPA50,000 CIFAR-10 images, by meNo60About 26 minutes0
Untrained small encoderNothing—0—0

What ran for this test

On an RTX 4060 laptop GPU, measured from the run log.

StepDetailsTime
Features from Meta's model60,000 images, enlarged to 224 × 22421 min
Comparison, Meta I-JEPAAll methods and settings, 2 or 3 runs each21 min
Comparison, small I-JEPASame10 min
Comparison, untrained encoderSame9 min
Comparison, ResNet-34Same, using features saved earlier9 min
All comparisonsAfter the features were ready49 min

Epochs inside the test

PartWhat it trainsEpochs
CBCLNothing. It builds centroids and votes0
CBCL-PRA linear classifier, retrained at each step: SGD, learning rate 0.01, momentum 0.9, batch 6425 to 70, rising by 5 per step
ReferenceThe same classifier, all at once70

Results

Method
Images per class
Show these values as a table

Final accuracy on all 100 classes

Percent, mean ± standard deviation. Chance is 1%. There are 3 runs with 10 images, 2 runs with all images, and 1 reference run with all images.

What this shows

  1. A strong encoder matters most. With CBCL-PR, Meta's I-JEPA beat ResNet-34 by about 12 points in both settings: 64.06% against 51.66%, and 74.23% against 62.59%.
  2. That isn't proof that JEPA beats CNNs. Meta's model has about 30 times the weights of ResNet-34, and I didn't test a large supervised model.
  3. Fake features have a cost. With all images and limited memory, CBCL-PR scored 7 to 11 points below learning everything at once, for our three trained or pretrained encoders.
  4. Our small I-JEPA fell well behind on new classes, at 26.67%. It still beat the untrained encoder's 10.53%, so it learned something that transfers.
  5. Memory can be small. Meta's 1,019 centroids need about 10 MB, against about 256 MB for all its real features, calculated from their sizes.

Limits

Credit