Scalable iterative algorithm for robust subspace clustering

Abstract

Subspace Clustering (SC), a generalized task of Principle Component Analysis (PCA), has been used extensively for dimensionality reduction of high-dimensional data. Recently, several methods have been proposed to enhance the robustness of PCA and SC, but most of them are computationally expensive, especially for high-dimensional large-scale data. In this paper, we develop a much faster algorithm for optimizing the NP-hard SC objective using a sum of the $\alpha$ -th power of $\ell_2$ -norm with $0<\alpha \leq 2$, where $\alpha$ =2 is the standard choice and $\alpha$ <2 enhances the robustness of SC. The known implementations achieving a local optimum of the objective would be costly due to the alternation of two separate tasks: optimal cluster-membership assignment and optimal subspace selection, while the substitution of one process to a faster surrogate can cause failure in convergence. Furthermore, such an alternating method has been often criticized due to the sensitivity of initialization. To address the issues, our proposed algorithm has the following key features: (a) release nested robust PCA loops for subspace update, (b) use a simplified singular value decomposition that only requires a few matrix-vector multiplications instead of solving an expensive eigenvector problem and (c) initialize carefully to avoid poor clustering. We prove that it monotonically converges to a local minimum of the SC objective for any $0<\alpha \leq 2$ and finds the true clustering under a statistical assumption on data. In our experiments, it is shown to converge at an order of magnitude faster than the standard implementation optimizing the same objective, and outperforms known SC methods in the literature for MNIST handwritten digit dataset.