Protoanchor: gaussian prototype memory with distributional anchoring for catastrophic forgetting mitigation in continual learning

Abstract

A major consequence of catastrophic forgetting is that previously learned knowledge can be drastically lost even when a model is trained on new tasks, which is one of the biggest challenges for the successful deployment of a neural network in the continual learning setting. In this paper, we propose ProtoAnchor, a continual learning approach where Gaussian prototypes {μk, Σk} that represent the embedding distribution of each class are stored as a compact and memory-efficient approach to exemplar replay that allows for a reasonable amount of privacy. ProtoAnchor introduces a novel training framework based on combining three complementary mechanisms: (i) Proto-noise generation using prototype sampling in Gaussian statistics, (ii) a distributional loss function penalizing feature space drift from the stored prototypes using KL divergence, and (iii) parameter-space regularization using Elastic Weight Consolidation (EWC). Experiments conducted on Split-CIFAR-100, Split-TinyImageNet and CORe50 reveal average values of forgetting of 1.4%, 1.8% and 1.6% respectively, which are 3–6× smaller than the DER++ method, the best performer up to now, while achieving average accuracy of 83.7%, 80.1%, and 85.3% respectively. Memory analysis to validate orders-of-magnitude storage reduction vs. raw replay was performed.