Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

Abstract

Despite the complexity and dynamic nature of the environment, animals find a way to maximize the amount of future reward without any supervision or a complete knowledge of the environment. Accumulating evidence shows that this behavior can be explained by reinforcement learning (RL). According to the RL theory, animals learn to predict future reward through trial and error. As the learning process is constrained by the limitations of time and resources, a biological agent should deal with the tradeoff between task performance and resource consumption. This study investigated whether the performance–efficiency tradeoff is reflected in the activity of dopamine neurons, the neural substrate that is deeply involved in the RL process. The main contributions of this study are as follows. First, we found that RL with dynamic resource allocation accounts for the ramping and phasic activity of dopamine neurons. Second, we showed that dopamine activity further explains how animals resolve the bias–variance tradeoff.