Automated augmentation for knowledge distillation

Abstract

In the aspect of knowledge distillation, data augmentation techniques serve to augment input data, the medium through which knowledge is distilled from teacher to student. In this case, teacher network and augmentation for teacher pre-training affect the performance of augmentation. Thus, in this paper, we propose novel data augmentation search method with consideration of teacher network and augmentation for teacher. Based on automated augmentation, we demonstrate how to use KD loss to consider teacher network. Moreover, we propose $\textit{policy distance}$ to measure the difference between two augmentation policies. Policy distance is used to maximize the distance from teacher augmentation, in our objectives. We demonstrate the effect of our proposed method by analyzing data distribution changes by augmentations. Through the analysis of these various aspects, we show that our proposed method search an improved data augmentation policy for knowledge distillation.