Continuous Learning

CBCL

Centroid-Based Concept Learning learns new classes a few at a time without forgetting old ones. It never retrains a network. It keeps each class as a small set of points, called centroids, and classifies a new image by a vote of the nearest ones.

Training architecture

The parts

ResNet

A network pretrained on ImageNet turns each image into 512 numbers. It stays frozen, so learning new classes can't damage old knowledge. Its final 1,000-class layer is removed.

ResNet-34 for CIFAR-100, ResNet-18 for the other two datasets. See every ResNet-34 layer.

Frozen: never trained Convolution stages Averagepool 512numbers 1,000-class layerremoved

Agg-Var clustering

Each class is learned on its own, one image at a time. Find the nearest centroid of the same class:

  • Closer than distance D: the image joins it, and the centroid moves towards it.
  • Otherwise: the image starts a new centroid.
c ← (w·c + x) / (w + 1), w ← w + 1

c is the centroid, w the images it holds, and x the new image's features.

Close: it joins D Centroid moves towards it Far: a new centroid More than D Its own centroid, w = 1

Long-term class memory

For each class, memory keeps its centroids, how many images each holds, and the class total N. Raw images aren't kept, and old classes stay unchanged.

If memory passes a budget K, old classes shrink in proportion to their size: k-means merges each one's centroids into fewer.

chairN = 20 cupN = 50 butterflyN = 30 Over budget K: k-means The newest classes aren't shrunk.

Weighted voting

Nothing is trained for this. At test time, the n nearest centroids from all classes vote. A closer centroid counts more, and each class's total is divided by its image count, so classes with many images don't win just for being large.

Pred(y) = (1 / N_y) × Σ 1 / dist(x, c_j)

The sum covers those of the n nearest centroids c_j that belong to class y. The class with the highest score wins. This is the thesis's formula, and the official code's differs (see Results).

Closer counts more 1 / distance Smaller classes get a boost N = 50 N = 20 1 / N: votes count more at N = 20

Testing architecture

Vote of the nearest neighbours

Change n to see how the answer changes. These are made-up 2-D points with made-up image counts, to show the idea. Real features have 512 numbers.

Nearest centroids that vote, n:

CBCL-PR

CBCL with Pseudo-Rehearsal builds the same memory as CBCL, but also remembers how spread out each centroid is. It uses this to generate fake features of old classes, then trains a small classifier on them. Voting isn't needed, and old images still aren't kept.

Training architecture

The parts

ResNet

The same frozen network as CBCL, with the same 512 numbers per image. Only the classifier at the end is ever trained.

ResNet-34 for CIFAR-100, ResNet-18 for the other two datasets.

FrozenResNet 512numbers

Agg-Var clustering, with spread

The join-or-start rule with distance D is the same as in CBCL. The difference is that each centroid also records how its images are spread around it, as a covariance.

A centroid holding one image has no spread, so all its fake features match it exactly.

CBCL keeps The point and its count CBCL-PR also keeps The spread: a covariance

Long-term class memory

Each centroid is stored as a mean, a covariance and a count. A covariance takes far more room than a point, and the thesis gives CBCL-PR a smaller budget K: for example, 1,600 centroids instead of 7,500 on CIFAR-100.

Old classes shrink by k-means when the budget is passed, as in CBCL. Raw images still aren't kept.

chair cup butterfly

Pseudo-exemplars

These are fake features, not images. Each is drawn at random from a centroid's stored mean and covariance:

x̃ ~ N(c_i, Σ_i)
  • Few images per class: about 40 fake features for every class, old and new.
  • All images: fake features for old classes, real ones for the new classes.
Illustration only, in 2-D.

Linear layer + softmax

One layer maps the 512 numbers to one output per class so far. It restarts from random weights at each step and trains on the features above, so old classes keep being rehearsed without their images.

SGD, momentum 0.9, learning rate 0.01, batch size 64, 25 epochs plus 5 more per step.

Old classes: fake New classes: real (all-images setting) 512 inputsOne per class chaircupbutterfly

Testing architecture

The softmax decision

This replaces the vote. The layer gives each class a score, and softmax turns the scores into probabilities that add up to 100%. The numbers below are made up, to show the idea.

chair
71% cup
20% butterfly
9%
Pred = argmax softmax(W x)

x is the 512 numbers and W the trained layer. Prediction is one matrix multiplication, so its time doesn't depend on how many centroids are stored.

Comparison

Both methods extract features and cluster them in the same way. They differ in what they store and how they make a decision.

Learning phase

LearningCBCLCBCL-PR
Feature extractorFrozen ResNetFrozen ResNetSame
ClusteringAgg-Var with DAgg-Var with DSame
Raw images keptNoNoSame
Stored per centroidMean, countMean, count, covarianceDifferent
Fake featuresNoneFrom each centroid's mean and covarianceDifferent
What gets trainedNothing after clusteringA linear softmax layerDifferent
Trained on—Few images: about 40 fake features per class. All images: real new, fake oldDifferent
Memory budget K, in the thesisCIFAR-100 7,500
Caltech-101 1,100
CUBS-200-2011 800
CIFAR-100 1,600
Caltech-101 1,000
CUBS-200-2011 600
Different

Testing phase

TestingCBCLCBCL-PR
FeaturesSame frozen ResNetSame frozen ResNetSame
Tested onAll classes seen so farAll classes seen so farSame
DecisionVote of n nearest centroidsHighest softmax probabilityDifferent
Needed at test timeAll centroids and class countsOnly the layer's weightsDifferent
Prediction timeGrows with the centroids storedGrows only with the number of classesDifferent
Settings tunedD and nD reused from voting. n isn't usedDifferent
Accuracy in this testLower in all 7 settings where both ranHigher in all 7. See ResultsDifferent

Both together

The full architecture in one picture. Select it to open it full size.

CBCL and CBCL-PR: new class images pass through a frozen ResNet and Agg-Var clustering into class memory. CBCL votes among centroids, and CBCL-PR samples pseudo-exemplars and trains a linear layer.

Datasets

The three datasets used in the thesis, from easy objects to very similar birds. Each picture shows the classes with sample images. Select one to open it full size.

Overview of CIFAR-100 classes

CIFAR-100

Everyday objects, animals, people and scenes, in very small images.

Classes used100
Images60,000, 32×32 pixels
Split500 training and 100 test images per class
FeaturesResNet-34
Overview of Caltech-101 classes

Caltech-101

Objects in larger, cleaner images, with classes of very different sizes.

Classes used100 of 101
Images8,677, of varying size
Split80% training, 20% test, per class
FeaturesResNet-18

The thesis doesn't say which category it left out. Here, Faces_easy is dropped, as it's a tighter crop of the images in Faces.

Overview of CUBS-200-2011 classes

CUBS-200-2011

Bird species that look very alike, so it's the hardest of the three to separate.

Classes used100 of 200
Images11,788 in the full set, of varying size
Split80% training, 20% test, per class
FeaturesResNet-18

The thesis doesn't say which 100 species it used. Here, it's the first 100 in the dataset's order. Usually written CUB-200-2011.

Experiments

The aim is to see how closely an independent rebuild of CBCL and CBCL-PR matches the numbers in Ali Ayub's thesis (Tables 3.2 and 6.1 for few images).

How each run works

ShufflePut the classes in a random order, with a different seed for each run.
LimitKeep 5 or 10 random images per class, or all of them.
TuneFor each new group of classes, choose D and n.
LearnAdd the 10 new classes to memory, and train the layer if the method has one.
TestTest on all classes seen so far, then repeat with the next 10.

Each box is 10 classes. Score: average incremental accuracy is the mean of the 10 test results. Each result is a mean of per-class accuracies, as in the official code.

What was run

The number of runs for each setting. A dash means it wasn't run.

The thesis uses 10 runs and also other step sizes. Only 10 classes per step was tested here. FLB is the thesis's baseline: a linear layer trained on all real features seen so far, using SGD, learning rate 0.001, batch size 8 and 25 epochs.

How parameters were chosen

ParameterWhat it controlsHow it was chosenValues
DHow close an image must be to join a centroidTuned at each step, as a multiple of the typical distance between two images of the same class0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 1.2, 1.5, 2 or 3 × that distance
nHow many centroids voteTuned at each step, together with D1, 3, 5, 7, 10, 15 or 25
KMemory budget, in centroidsFrom the thesis. Applied in the all-images runs. A D that would overflow K is ruled outSee Comparison
Fake featuresTraining samples per class in CBCL-PRFrom the thesis, for few images40 per class
CovarianceShape of each centroid's spreadFull for 5 and 10 images. Diagonal for all imagesFull runs sample from a centroid's own features during the run, to avoid forming a 512×512 matrix
Linear layerCBCL-PR's classifier trainingFrom the thesisSGD, momentum 0.9, learning rate 0.01, batch 64, 25 epochs + 5 per step

How D and n are tuned

What tuning chose

CBCL with the thesis's voting formula. Each run tunes 10 times, so a setting has one value per step per run.

DatasetImages per classTypical same-class distanceMedian DD rangeMost common n
CIFAR-100522.6727.26.8–34.07 (22 of 100)
Caltech-101518.6127.95.6–37.23 (17 of 30)
Caltech-1011018.6122.35.6–27.93 (17 of 30)
Caltech-101All18.6127.918.6–55.81 (30 of 30)
CUBS-200-2011517.8326.75.3–35.75 (10 of 30)
CUBS-200-20111017.8321.412.5–35.73 (12 of 30)
CUBS-200-2011All17.8326.717.8–53.53 (11 of 30)

Where this differs from the thesis

Results

Preliminary, not a finished reproduction.
  • Only one schedule was tested: 10 new classes at each step.
  • Most results use 3 runs. The thesis uses 10.
  • The CIFAR-100 CBCL and CBCL-PR results predate a later change to how D is tuned, and haven't been rerun.
Dataset

Accuracy as classes are added

Images per class
Spread

Move over the chart to read values. Select a method in the legend to hide or show it.

Show these values as a table

This test against the thesis

Average incremental accuracy for the dataset chosen above, using the thesis's voting formula.

Show all 19 comparisons as a table

The all-images CBCL-PR runs use diagonal covariance. The others use full covariance. Standard deviations are over the runs shown, and with 3 runs they understate the true variation.

What the comparison shows

The method order mostly agrees with the thesis. In 4 of the 5 few-image settings, the order was the same: CBCL-PR, then CBCL, then the FLB baseline. In Caltech-101 with 10 images, FLB beat CBCL here, but not in the thesis. CBCL-PR beat CBCL in all 7 settings where both ran, as it did in the thesis.

Accuracy is usually close, but not everywhere. Of 19 comparisons, 6 were within 2 points of the thesis, 8 more were within 5 points, and 5 were further away. 16 of the 19 were below it. The largest gap was CBCL on Caltech-101 with 10 images, at 11.52 points below.

The voting formula can matter

CBCL with 5 images per class. The two formulas gave different results, particularly on Caltech-101 and CUBS-200-2011. This could explain part of the gaps above, but I haven't tested that, and each official-formula result rests on only 3 runs.

Show these values as a table

More

The online dashboard shows these results with more detail. It's read-only, because teaching new objects by camera or upload needs a Python backend running on a local computer.