Re-initialization mitigates noisy labels

Abstract

Deep neural networks have an outstanding capability to memorize data, yet this memorization ability sometimes imposes networks to be vulnerable to mislabeled data. Most existing methods focus on utilizing noisy dataset while suppressing the memorization capacity on noise instances. However, these studies designed to estimate clean and noisy samples inherit fundamental limitations. If the network estimates some of the noise samples as clean instances at the early stage of the training, the misinterpreted samples are drifted into clean data distribution. This may distort the feature space severely, leading the network prone to perceive other noisy samples as clean samples. To mitigate this issue, we propose a novel training approach, named Unbiasing Feature Alignment via Re-Initialization, called U-Init. We observe that the classifier is rapidly converged and fitted to noisy data in early epochs, which largely misdirects the feature extractor and corrupt the feature distribution during the rest of the training. Our main assumption is that the fc layer converged in the early epoch is responsible for disturbing the feature extractor to train with noisy labels. Therefore, it is crucial to adjust the fc layer to update the feature extractor in the other direction. To this end, we periodically re-initialize the fc layer during the training to prevent the rest part of the network from being self-biased. Extensive experiments show that our method significantly improves the performance compared to existing methods by preventing the feature space from the distortion.