Deep anomaly detection for unlabeled dataset

Abstract

Anomaly detection techniques constitute a fundamental resource in various applications to identify observations that deviate considerably from what is considered normal. In recent years, in response to increasingly complex data at a large scale, deep learning-based anomaly detection has been actively researched and has shown high capabilities. Meanwhile, since deep learning is based on a representation learned from a given dataset, most deep anomaly detection models aim to learn normality, assuming that a dataset consisting of only normal samples is available. Thereby, one-class classification-based approaches are one of the representative approaches of deep anomaly detection. However, in practice, it is expensive to prepare a training dataset consisting of normal samples as it requires per-sample inspection. This dissertation deals with deep anomaly detection based on robust normality learning in deep one-class classification for the unlabeled dataset, which is a mixed set of normal and abnormal data. To this end, we propose 1) an unsupervised one-class classification based on an adaptive threshold and pseudo-labeling without human intervention and 2) an active learning-based one-class classification searching for an adaptive threshold by leveraging human annotation feedback. For both methods, we obviate the use of a hyper-parameter for the anomaly ratio of a dataset that is often unknown prior knowledge in practice but critical for one-class classification. Through multiple experiments and analyses with multivariate point datasets, we demonstrate that the proposed methods improve the robustness of anomaly detection under different levels of anomaly ratio, which is unknown in practice. The proposed method is also applied to microscopy cell images contaminated with artifacts, adding its practicality in the real world. This work contributes to increasing the utilization of a vast amount of collected data that remains unlabeled in the real world and the efficiency of labeling workers.