Selective forgetting of classes and tasks for deep neural networks

Abstract

Pruning units such as filters and neurons have been used to address the inefficiency of memory and computational resources in deep neural networks. Existing work mainly focuses on a single task with a static number of classes, but in practice, classes and tasks become obsolete. Using information about obsolescence, I propose a `Selective Forgetting' method that prunes units that are unnecessary to classes or tasks of interests. Furthermore, I suggest a novel neural architecture that is advantageous to selective forgetting by disentangling contributions of units across classes and tasks. I validate my approach in experiments about class forgetting on a single task model and task forgetting on continual learning models. Experimental results demonstrate that 1) my pruning method outperforms baselines in terms of mean and minimum performance of classes/tasks to be preserved and 2) my architecture makes units be disentangled and gains more benefit from selective forgetting than the entangled network.