Continual learning with linearized deep neural networks

Abstract

We propose a continual learning algorithm that effectively mitigates catastrophic forgetting that occurs when a deep neural network is trained on multiple tasks sequentially. Our method takes advantage of the pre-training of neural networks for effective continual learning. Based on the observation that quadratic parameter regularization is able to achieve the optimal continual learning policy with linear models, our algorithm $\textit{linearizes}$ the neural network and applies quadratic penalty to parameters by estimating the Fisher information matrix. We show that the proposed method can prevent forgetting while achieving high performance on image classification tasks. Our method can be used in data incremental and task incremental learning problems.