Hardware and software systems for accelerating large-scale deep learning recommendation models

Abstract

Deep learning-based recommendation models (DLRMs) are widely used for conducting personalized recommendations, which employ learnable vector parameters, known as embeddings, representing individualized characteristics of users and recommended items such as media contents, products, and ads. A unique characteristic of DLRMs is that due to the embedding layer, the size of the recommender model scales proportional to the size of the online service. Consequently, the size of the DLRMs reaches terabyte-scale for massive-scale online services like Facebook, far exceeding the capacity of bandwidth-optimized accelerator memory. The memory capacity and bandwidth demand from these enlarged embedding layers bring new system-level challenges in training and deploying large-scale recommendation models. This dissertation addresses the bottlenecks of the large-scale deep learning recommendation models by proposing novel hardware and software systems.

This dissertation first identifies that enlarged embedding layers cause major performance challenges in DLRMs. The study clarifies the computational characteristics of such layers and proposes a near-memory processing (NMP) based accelerator hardware that efficiently stores and processes these embeddings. The proposed vertically integrated hardware/software co-design encompasses the required microarchitecture, instruction set architecture (ISA), system architecture, software stack, and a workload parallelization algorithm. Furthermore, to expand the research scope of the NMP-based embedding acceleration to the training context, this dissertation presents an algorithm-architecture co-design, which establishes a theoretical foundation for hardware accelerator design for the embedding layer.

Since such specialized hardware-based acceleration systems can fundamentally address the challenges posed by large embedding layers, developing and maintaining these systems require non-trivial costs. As a cost-effective solution, this dissertation also presents software optimization techniques. By utilizing the highly sparse and skewed access patterns of the embedding layers, this dissertation presents a software-managed caching system using high-bandwidth GPU memory to cache frequently accessed embedding entries. The proposed software system leverages a unique characteristic of the recommendation model training to perfectly prefetches soon-to-be-accessed embedding entries in advance to boost training speed. Lastly, the study analyzes challenges in developing software systems for utilizing the locality of the embedding layer during inference and proposes a new type of caching technique for the embedding layer. The proposed caching mechanism leverages massively parallelized address translation hardware in the accelerator to eliminate bottlenecks in the software-managed embedding cache, which is highly effective for recommendation inference acceleration.