Approximate inference for large-scale matrix functions

Abstract

We study a specific family of matrix functions (1) spectral functions that rely on only eigenvalues of a matrix and (2) elementwise function that apply scalar function to each element in a matrix. They have been played an important role in many data sciences including quantum chromodynamics, network analysis, computational biology and machine learning including Gaussian graphical model, kernel learning and deep learning. Computation of such functions can yield scalability issues when they come with a huge size of matrices. We propose fast and efficient algorithms for both approximating and optimizing spectral functions and elementwise matrix functions. We utilize randomized trace estimator and polynomial approximation for spectral functions and linear sketching methods for elementwise functions. All algorithms can run in linear-time with respect to the input size, which is tens of magnitude faster in practice than the previously known methods. We provide rigorous approximation error analysis and experimentally verify the effectiveness of all the techniques that we develop, testing them on applications such as classification, interpolation, recommendation and deep network compression.