In-network architecture for multi-GPU

Abstract

Deep learning technology requires tremendous computing power and memory, and multi-GPU systems are developing accordingly. However, in order to perform deep learning training in a multi-GPU system, data parallelism must be used. Data parallelism utilizes All-Reduce, a cluster communication library, during data parallelism. All-Reduce averages the slopes scattered across each GPU. The process of having to communicate with a different GPU each deep learning training session acts as a tremendous burden. Recently, deep learning frameworks such as Pytorch and TensorFlow are trying to reduce communication overhead by overlapping backpropagation and All-Reduce. While this increased throughput, it was observed that the backpropagation time increased due to the overlapping. We present an in-network architecture to improve the slow backpropagation caused by the aforementioned overlapping. We verified it using the latest multi-GPU simulator, MGPUSim, and will improve the slow backpropagation phenomenon, and achieve performance improvement in the limited All-Reduce size.