Managing interference and scheduling deep learning tasks on consolidated GPU computing environment

Abstract

Deep-learning applications have gained popularity among service providers who wish to provide cognitive services to consumers. As deep learning gained popularity, ware-house scale servers have been developed and researched for guaranteeing Quality-of-Service (QoS) and throughput of deep learning tasks. Deep-learning tasks such as training and inference are compute-intensive tasks and accelerated by exploiting the parallelism that exists within. Therefore deep-learning applications are executed on ware-house scale server with GPUs, as means of acceleration. While deep-learning applications and related frameworks have been developed to make use of clustered resources, the key challenge of maximizing resource utilization, with limited amount of resource remains as a challenge. When there are not enough number of tasks to fully utilize the resources, servers are underutilized and the problem still exists when the running tasks are underutilizing GPU. In order to fully utilize resources on server, deep-learning tasks must be executed on a consolidated computing environment where multiple tasks can execute on multiple hardware and flexibly using available resources. However, the main problem on consolidated environments arises when latency-sensitive (LS) tasks receive interference from co-located tasks. Performance degradation is severe especially for compute intensive deep learning tasks. In order to prevent performance degradation, we propose an adaptive controlling method for adjusting the duration of co-located tasks with LS tasks in this dissertation. Since the main source of interference is queuing time, adjusting the time of halting co-located tasks is effective. The ratio of halting time to total execution time is determined adaptively to the performance degradation of the last LS task on the GPU. Additionally, imbalanced workload also hurts performance. In order to ensure balanced workload among GPUs, we propose greedy task scheduling and task migration. For evaluation, we have prepared a machine with multiple GPUs and Caffe implementation of Convolutional Neural Networks (CNN) were used as benchmarks throughout this study. The experimental results show that the proposed control method can prevent performance degradation and provide bandwidth for batch tasks by effectively sharing resources.