Feature based knowledge distillation for image recognition

Abstract

The performance of models related to visual recognition, such as image classification and object detection, has recently improved with the development of deep learning. However, in order to train a deep learning-based model, a model with many parameters and a large amount of data are required. This thesis examines the knowledge distillation method. First, we proposed an attention-based meta-network which models relationships between the teacher model layers and the student model layers to distill knowledge effectively. We empirically show through experiments that the proposed methodology is more effective than using a heuristic to designate the layer links between the teacher model layers and the student model layers. Second, when using the method of knowledge distillation by the student model itself without a teacher model (self-knowledge distillation), we proposed the novel method which utilizes spatial information. To this end, we introduced an auxiliary network for training by altering the model used in the existing object detection model.