(A) software framework for estimating training time of trillion-parameter scale distributed machine learning

Abstract

As the size of deep neural network (DNN) models is rapidly increasing to improve performance, the demand for compute resources required for DNN training is exponentially increasing. Such large-scale training is performed on distributed systems with various parallelism techniques, and the training performance in distributed systems varies drastically depending on DNN model architecture, the network topology, and the combination of parallelism techniques. However, finding the optimal training configuration incurs immense expenses, leading to the inability to effectively use compute resources in large-scale training. To address this, this thesis proposes a simulation framework to predict the training iteration time of distributed training. The proposed framework accurately predicts the training iteration time for various configurations with a mean absolute error of 12.80%, facilitating efficient exploration for the optimal training configuration.