(An) FPGA-based preprocessing system for GPU-partitioned machine learning inference server

Abstract

In machine learning inference servers, unlike training servers, inference requests are irregularly allocated and must be completed within a limited time. Consequently, operations are performed with small batch sizes, leading to inefficient utilization of GPU resources. Recent advancements in GPUs provide partitioning technology, allowing the efficient use of GPU resources by dividing a single hardware resource into independent hardware of suitable sizes for users. As this technology is implemented in inference servers, there is an increase in the processing capacity and resource utilization of GPUs. However, this leads to a bottleneck in the preprocessing stage on the CPU associated with inference requests. In this thesis research, an analysis of the bottleneck points of preprocessing stage in GPU-partitioned machine learning inference server is conducted, and proposes FPGA-based hardware design to offload the data preprocessing to increase the overall processing throughput of the ML inference server.