Orthogonal feature regularization

Abstract

Many deep neural networks (DNN) are easily fooled with adversarial examples designed to cause incorrect classification. Many researchers have tried to find a training recipe for defending against adversaries. However, most of these recipes, such as adversarial training, have the drawback that they are only robust for specific adversaries. Recent works have proposed many training algorithms with regularization, such as weight orthogonalization, penalizing the $l_2$ norm of the input gradient, and controlling the Lipschitz constants of each layer, however, they all have some limitations in terms of computational costs and efficacy. To address this problem, we propose a new approach with “resemble regularization”, which can be different from the concept that we generally believe. Our key idea is to encourage each layer’s outputs from different classes to resemble each other. The advantages of this method are that it helps the model to become robust against $l_\infty$ adversarial perturbations while requiring little computational cost, and it can be used with other robust regularization methods simultaneously, resulting in higher robustness. Our method is verified on MNIST and CIFAR-10. On CIFAR-10, we achieve state-of-the- art performance, which substantially improves the accuracy from 38.58% to 65.92% when tested on the adversaries with $l_\infty$ perturbations of $\epsilon$ = 0.1. In addition, robust adversarial errors against most of the adversaries are improved with a large margin of more than 20%. Through analysis of our method, we expect that this approach can reveal the fundamental reasons for the vulnerability of adversarial examples.