Continual learning of deep neural networks under strict constraints

Abstract

In this dissertation, we discuss continual learning methods that can be practically used not only in common situations, but also under several strict constraints. Specifically, (1) source task information should not be stored in external memory, (2) the network size should remain constant, and (3) past and future task data should not be used for hyperparameter selection. First, we introduce a residual continual learning method that satisfies the first and second conditions. The residual continual learning method avoids unnecessary weight changes using residual reparameterization and a decay loss, but past data has to be used for hyperparameter selection. To solve the hyperparameter problem, we construct a quadratic penalty method that satisfies the second and third conditions. The quadratic penalty method approximates the original loss function by the extended Kronecker-factored approximate curvature. Finally, by reconstructing data from a trained network through statistics matching and regularizing the output and weight together, we propose a standalone continual learning method that satisfies all of the three conditions.