Deep neural network pruning for self-supervised transfer learning

Abstract

Recent advancements in self-supervised learning framework show promising results on various downstream tasks, where it can produce on-par or better visual representations than supervised ones with extensive pretraining costs and high complexity. One possible way to solve these problems is to sparsify the network before learning via Pruning-at-Initialization (PaI), aimed to downsize neural networks without significant loss of accuracy compared to dense, overparameterized models. However, our understanding of Pruning-at-Initialization (PaI) methods is limited to supervised learning, where models learn from massive amounts of carefully labeled data. In this work, we first investigate how sparse networks obtained from the criterion of different PaI methods would perform on the Self-Supervised Learning pretraining framework and how comparable they are to the supervised learning setup. Furthermore, we find that sparse networks trained on self-supervised frameworks perform better quantitatively and qualitatively than their supervised counterpart on various downstream tasks, especially in transfer learning tasks.