(A) batch orchestration algorithm for straggler mitigation of synchronous SGD in heterogeneous GPU cluster

Abstract

so various researches have been conducted on accelerating the training speed through distributed processing. Data parallelism is one of the widely used distributed training schemes, and various algorithms for the data parallelism have been studied. However, since most of the studies assumed homogeneous computing environment, there is a problem that they do not consider a heterogeneous performance graphics processing unit (GPU) cluster environment induced by rapid performance changes of GPU types. Heterogeneous performance GPU clusters have performance differences between workers. It leads to differences in computation time between GPU workers in synchronous data parallelism, in which the total global mini-batch size is usually divided equally among several workers. Due to the time difference of the computation time of one iteration, the straggler problem that fast workers wait for the slowest worker makes the training speed low. In this thesis, we proposed a batch-orchestrating algorithm (BOA), reducing training time by improving hardware efficiency in heterogeneous performance GPU clusters. The proposed algorithm coordinates local mini-batch sizes for all workers to reduce one training iteration time. Additionally, we conducted performance tuning by searching better GPU worker set. We confirmed that the proposed algorithm improved the performance by 23% over the synchronous SGD with one back-up worker of training ResNet-194 with 8 GPUs of three different types: GTX 1080, GTX1060 and QuadroM2000. The proposed BOA solved the problem caused by the performance difference between GPU workers, and it accelerated the convergence speed of training.

Training deep learning model is time consuming