A 8.81 TFLOPS/W Deep-Reinforcement-Learning Accelerator With Delta-Based Weight Sharing and Block-Mantissa Reconfigurable PE Array

Abstract

TD3 is one of the most high-performing Deep Reinforcement Learning (DRL) algorithms, providing high training stability and rewards. However, it suffers from low energy efficiency due to high External Memory Access (EMA) and floating point operations. To mitigate this issue and achieve higher throughput and energy efficiency, we propose the DRL accelerator with 3 features: 1) Delta-based Weight Sharing (DWS) represents weights by referencing corresponding network and exploits data locality, leading to EMA reduction of up to 64.3% in feed-forward stage and 39.7% in gradient generation and weight update stage. 2) Block-Mantissa Reconfigurable PE Array (BMRPA) supports variable operations in blocks and mantissa to provide optimal precision for each layer, resulting in up to a 4x increase in throughput. 3) Multi-mode Data Fetcher (MDF) supports bit width adaptive data fetching, achieving twice the bandwidth with an average read overhead of 5.3%. When combined with BMRPA, it attains an energy efficiency of 8.81 TFLOPS/W.