Accelerating Randomly Projected Gradient with Variance Reduction

Abstract

Parallel training methods of stochastic gradient descent (SGD) for deep learning have attracted a huge attention because of excellent scalability properties. In the parallel training methods, workers and a server exchange their gradient vectors (or parameter vectors) each other. However, the communications between workers and server take a lot of time since the dimension of the gradient vectors is extremely high. Transmission needs to cater for network bandwidth, so message compression is necessary. Although message compression algorithms have been proposed for the parallel training methods, there are concerns about the stability and the performance of the compressed training algorithms. hi general, the compressed training algorithms generate random message vectors, and the variance hinders the training. In this paper, we propose a novel compressed training algorithm that uses random projection with a variance reduction trick. Here, we introduce an average gradient vector that reduces the variance. We test the proposed compression method with AlexNet, ResNet20 models with CIFAR100 data. The proposed algorithm shows almost the same performance as the original SGD while uses 16 times compressed message.