Let it reuse

Abstract

Transformer-based models are rapidly emerging in various fields of DNNs. Therefore, accelerators for the self-attention mechanism, a bottleneck of the transformer, are actively studied today. However, for real-world accelerators, not only high performance but also generality and flexibility are necessary. First, because the required precision and datatype of each task are different, the accelerators should generally support multi-precision. Second, because the required accuracy, energy, and latency change depend on the scenarios, the accelerators should flexibly support the multi-mode without severe HW underutilization. Real-world accelerators need to deliver high performance even under the aforementioned functionalities. This paper shows that the prior design framework has reached its limit in terms of computational savings. This paper presents an interpretable design framework called "Let It Reuse." To effectively utilize this framework and satisfy real-world constraints, it takes a co-optimization approach, including an algorithm, architecture and microarchitecture. In detail, this paper proposes a multi-mode aware pipeline with a unified multi-precision datapath and explores reusability according to the datatype. As an experiment of the Question & Answering task, the Let It Reuse Accelerator improves the geomean speedup by 24 times and 4 times, respectively, compared to a GPU, an up-to-date Nvidia ampere architecture, and Sanger, a state-of-the-art attention accelerator.