4-bit quantized training of deep neural networks

Abstract

Deep Neural Network(DNN) has been widely used to build A.I. in various fields such as computer vision and natural language process and outperformed other classical techniques. However, its massive computation and memory requirement hinder the implementation of DNNs on edge devices that have low computing power and smaller memory. Quantization of DNN is a network compression technique that reduces the precision of data involved only in the inference, or in both the inference and training. The quantized inference has been studied by lots of researchers, but the quantized training has had little focus and nothing has dealt with the utilization of only 4 or fewer-bit for all data objects in the training. Here I propose a novel quantized training algorithm GSNet (Gradient Stabilizer Network), that utilizes only 4 or fewer-bit data using a 4-bit additional object `stabilizer'. The stabilizer securely accumulates 2-bit gradients which are ternarized using a novel gradient scaling method. GSNet was evaluated in the image classification task and showed competitive performance compared with other quantization techniques that utilize even higher bit-precision.