Communication optimization for deep learning in distributed processing environments

Abstract

Machine learning has recently been in the spotlight as a solution to problems that were difficult to solve. Distributed processing techniques using graphic processing devices are widely used to deal with the vast amount of data needed to learn neural networks in deep learning, the most widely used type of machine learning, and thus collective communication within distributed systems exists as the main performance bottleneck and impairs the scalability of the system. In this dissertation, we would like to propose techniques and computer architectures that address communication bottlenecks and improve learning performance, considering the characteristics of the intermediate values of collective communication, the multilayer properties of deep learning, and the structure of distributed systems that perform actual computations.