Development of flexible reinforcement learning algorithms using model-based fMRI analyses

Abstract

Reinforcement learning (RL) has been successfully used to model the value-based learning of human decision-making processes. Humans employ two distinctive RL strategies: model-based and model-free learning. Model-based learning shows excellent performance through fast adaptation to environmental context changes, but it involves large cognitive loads. In contrast, model-free learning is very efficient and involves much smaller cognitive loads

however, it is prone to failure due to dynamic changes in the environmental context. Because the human brain has limited cognitive resources, an appropriate arbitration control between RL strategies is required for humans to make efficient and high-performance decisions. In this work, we investigated how humans perform the arbitration control due to the two essential factors, which are never dealt. The factors that make adaptive behaviors much difficult due to environmental context changes include: 1) changes in task context, which directly increase cognitive loads, and 2) uncertain RL model’s baseline performance due to the environmental context changes, which makes the arbitration control even harder because we cannot precisely estimate the expected performance of the RL model. To do so, a computational model of human RL with arbitration control for context changes was developed. Consequently, fMRI signals were analyzed to understand how human RL functions. First, a computational model for arbitration control incorporating task complexity, which directly affects cognitive load, was developed. By analyzing fMRI signals with the model, it was found that humans employ model-based learning to resolve task complexity. In addition, behavioral analysis showed that the RL strategies have different levels of performance expectation and distributions of prediction error due to changes in task context, and a computational model with adaptive updating of the prediction error baseline was developed. Model-based fMRI analysis showed how the human brain has arbitration control that is adaptive to context changes. Based on the aforementioned findings, it was possible to develop a human RL algorithm that has flexible arbitration control responsive to changes in task context. For large-scale simulations, the computational model for human RL was found to outperform state-of-the-art RL algorithms presented in the artificial intelligence (AI) literature. Finally, using the computational model for human RL, a deep neural network was constructed, classifying the two learning strategies based on electroencephalography (EEG). Surprisingly, reading out the learning strategy is meaningful for classifying decisions through shared informative features in the neural network. In this dissertation, we provide not only full descriptions of flexible human RL algorithm development but also neural evidence for them. Moreover, the findings could possibly be applied to brain-inspired AI and brain–computer interfaces (BCI).