Proximal policy optimization (PPO) based reinforcement learning model for scalable 3D X-point array structure design considering signal integrity issues

Abstract

In this paper, we, for the first time, propose a reinforcement-learning model to design an optimal 3D X-Point array structure considering signal integrity issues. The interconnection design problem is modeled to the MDP. The proposed reinforcement-learning model designs the 3D X-Point array structure based on three reward factors: the number of bits, the crosstalk, and the IR drop. We applied multi-layer perceptron and long shot-term memory to parameterize the policy. Proximal policy optimization is used to optimize the parameters to train the policy. The reward of the proposed reinforcement-learning model is well converged with variations of the array structure size and hyperparameters of the reward factors. We verified the scalability and sensitivity of the proposed reinforcement-learning model. With the optimal 3D X-Point array structure design, we analyzed the reward factor and signal integrity issues. The optimal design of the 3D X-Point array structure shows 17 % to 26.5 % better signal integrity performance than the conventional design in finer process technology. In addition, we suggest a range of possible directions for improvement of the proposed model with variations of MDP tuples, reward factors, and learning algorithms, among other factors. Using the proposed model, we can easily design an optimal 3D X-Point array structure with a certain size, performance capabilities and specifications based on reward factors and hyperparameters.