Infinitely wide neural networks with heavy tails and inter-node dependence

Abstract

Machine learning models are typically initialized by independent Gaussian weights. However, there are evidences which reveal that the weights of some pre-trained models exhibit heavy-tailed distributions and dependence between themselves. This hints that, in terms of Bayesian inference, our prior belief does not properly describe the true behavior of the network function. To alleviate such limitations, we consider a network model where the weights are initialized with possibly dependent heavy-tailed distributions. We prove that, as the network width tends to infinity, the outputs of such a network converge in distribution to stable processes or, in general, mixtures of Gaussian processes, where the limiting stochastic processes are determined according to the distributions the weights are initialized. We also prove that some weights do not converge to zero as the width tends to infinity, and the corresponding node may represent hidden features. Additionally, we investigate the pruning error under the infinite-width limit. Finally, we analyze the optimization of our network via gradient flow, and prove that the gradient flow converges to the global minimum while learning features, unlike the previous Neural Tangent Kernel (NTK) model.