Continual learning with feature replay through sample reconstruction

Abstract

Continual learning is an online method of learning that requires multi-task solving skills to an endless stream of data. The major challenge is catastrophic forgetting, which is the learner forgets the previous knowledge when is trained on new information. To address this problem, memory-based methods are widely used and have shown high performance. However, models with raw data can lead to low storage issues and also cannot be applied to the sensitive data that cannot be stored. In this paper, we propose Feature Replay through Sample Reconstruction (FR-SR) to address the described problem. FR-SR stores the feature instead of raw data and replays through the variant of Variational Auto-Encoder (VAE). We suggest a novel architecture for feature replay methods and show our model more suitable than the original VAE. Furthermore, the performance of our model comes up to the memory-based model. As a result, FR-SR achieved the highest accuracy of any non-memory methods in the Split MNIST and Split FMNIST datasets.