Balancing knowledge distillation via reliability of knowledge

Abstract

Knowledge distillation is a popular network compressing method which improves the performance of a small network (student) by employing output logits of a pre-trained large network (teacher). However, previous studies undoubtedly trust that the teacher network would always give beneficial knowledge in the logits. In this study, we specify the problem that distilling unreliable knowledge from the prediction of teachers would cause degradation of students. To tackle this problem, we propose the balancing knowledge distillation method which regulates the degree of knowledge distillation by utilizing the prior data distribution from the trained teacher. The proposed method can reflect various data distributions that contain the reliability of knowledge. Our results show that the balancing method based on the prior data distribution improves knowledge distillation regardless of datasets.