Quantized neural network training and inference in practical scenarios

Abstract

Neural Network Quantization aims to reduce the size and computational complexity of a neural network for more efficient training and inference of neural networks. However, existing methods often render themselves impractical in real-world scenarios, such as On-device Federated Learning, and with compact models such as MobileNet. In this paper, We show that applying Neural Network quantization in these scenarios are difficult. In On-device Federated Learning scenarios, many diverse devices with different hardware constraints can participate in the same Federated Learning, which leads to degenerate performance in the high-performance devices. With compact models with less redundancies in the weights, it is much more difficult to find quantized weights that do not incur a drop in the model's accuracy. We tackle these challenges by introducing two novel methods for practical neural network quantization: Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantizer, and Neural Network Binarization with Task-dependent Aggregated Transform. We show that these methods are effective at applying Neural Network quantization in the aforementioned practical scenarios.