Workload characterization for gpu partitioning-based machine learning inference server

Abstract

Maximizing the utilization of graphics processing unit (GPU) resources becomes a crucial factor in directly reducing a data center’s Total cost of ownership (TCO). To address this, GPU partitioning technology has been developed, enabling the simultaneous execution of multiple workloads by dividing a single GPU. However, research on the analysis of characteristics when GPU partitioning technology is applied in real-world machine learning inference systems has not been actively conducted. This thesis analyzes workloads when utilizing GPU partitioning technology to enhance the efficiency of machine learning inference systems. Specifically, we focus on aspects such as resource utilization, throughput, and latency in the context of GPU partitioning. Based on the characterization results, we propose an efficient batching system for GPU partitioning-based machine learning inference to optimize the performance of the overall system further.