Robust training with ensemble consensus

Abstract

Deep neural networks optimized with gradient-based method exhibit two different behaviors when trained on poorly-annotated datasets: generalization in the early stage and memorization in the later stage. We analyze these two behaviors by measuring the similarity of the learned patterns between an ensemble of networks. From the analysis, we find that some correctly-annotated examples incur small training losses on all networks in the ensemble during generalization, while wrongly-annotated examples do not. Based on the finding, we propose a robust training method, termed learning with ensemble consensus (LEC) where an ensemble of networks is trained using the examples incurring small training losses on all the networks in the ensemble. The proposed method removes effectively noisy examples from training batches, resulting in robustness on the highly corrupted datasets.