NeuKron: constant-space lossy compression of sparse reorderable matrices

Abstract

Many real-world data are naturally represented as a sparse reorderable matrix, whose rows and columns can be reordered without information loss. Storing a sparse matrix in conventional ways requires an amount of space linear in the number of non-zeros, and lossy compression of sparse matrices (e.g., Truncated SVD) typically requires an amount of space sublinear in the number of non-zeros but still linear in the number of rows and columns. In this work, we propose \method for compressing a sparse reorderable matrix, regardless of its size, into a fixed amount of space. NeuKron updates the parameters so that a given matrix is approximated by the product, and NeuKron also reorders the rows and columns of the matrix to facilitate the approximation. Given an $n$-by-$m$ matrix with $p$ non-zeros, where $n \leq m$ without loss of generality, the above update steps, which take $O(m+p\cdot \log m)$ time, are repeated alternatively, and from the trained model, the approximate value of each entry is retrieved in $O(\log m)$ time. Through experiments on six real-world datasets, we demonstrate that NeuKron is (a) Compact: requiring five orders of magnitude less space than its best competitor with similar approximation errors, (b) Accurate: giving up to $10.1\times$ smaller approximation errors than its best competitor with similar space requirements, and (c) Scalable: compressing a matrix with about $230$ million non-zeros.