Contrastive learning for knowledge distillation-based anomaly detection

Abstract

In this work, we analyzed the two recent knowledge distillation-based anomaly detection methods and proposed a solution for their problem. Recently, in the field of anomaly detection, knowledge distillation-based methods are attracting attention for their excellent anomaly detection capabilities. Multiresolution Knowledge Distillation for Anomaly Detection (MKD) and Anomaly Detection via Reverse Distillation from One-Class Embedding (RD4AD) are representative of knowledge distillation-based anomaly detection techniques. We found that these two state-of-the-art techniques have difficulties in generalizing input data with a large diversity. We proposed residual contrastive learning (RCL) for this problem. RCL is a contrastive learning technique that can be used when anomaly data is available for training data. RCL uses the residuals between feature vectors of the teacher model and the student model used in the knowledge distillation-based anomaly detection model. We show that RCL enhance knowledge distillation-based anomaly detection methods for various datasets. Our method also successfully outperform previous state-of-the-art supervised anomaly detection methods.